Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Authors: Qi Ju
Thomas Tellier
Meng Sun
Zhemei Fang
YunFeng Luo
Venue: ICML 2025 poster (2025)
Keywords: Nash Equilibrium
CFVFP
customized AI
Game
Primary area: theory->game_theory
TL;DR: This paper presents a novel algorithm, Preference-based Counterfactual Regret Minimization (PrefCFR), which enables convergence to diverse equilibria in imperfect-information games.

Abstract: Artificial intelligence (AI) has surpassed top human players in a variety of games. In imperfect information games, these achievements have primarily been driven by Counterfactual Regret Minimization (CFR) and its variants for computing Nash equilibrium. However, most existing research has focused on maximizing payoff, while largely neglecting the importance of strategic diversity and the need for varied play styles, thereby limiting AI’s adaptability to different user preferences.

To address this gap, we propose Preference-CFR (Pref-CFR), a novel method that incorporates two key parameters: preference degree and vulnerability degree. These parameters enable the AI to adjust its strategic distribution within an acceptable performance loss threshold, thereby enhancing its adaptability to a wider range of strategic demands. In our experiments with Texas Hold’em, Pref-CFR successfully trained Aggressive and Loose Passive styles that not only match original CFR-based strategies in performance but also display clearly distinct behavioral patterns. Notably, for certain hand scenarios, Pref-CFR produces strategies that diverge significantly from both conventional expert heuristics and original CFR outputs, potentially offering novel insights for professional players.

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Qi Ju Thomas Tellier? Meng Sun
Zhemei Fang 12 Yunfeng Luo 12
Abstract
Go (Silver al., 2016; 2018), StarCraft (Vinyals al., 2019), and Texas Hold' em (Moravčík et al., 2017: Bowling et al., 2015; Brown & Sandholm, 2019b) have significantly influenced both academic research and public interest.
Artificial intelligence (AI) has surpassed top hu- man players in a variety of games. In imperfect information games, these achievements have pri- marily been driven by Counterfactual Regret Min- imization (CFR) and its variants for computing Nash equilibrium. However, most existing re- search has focused on maximizing payoff, while largely neglecting the importance of strategic di- versity and the need for varied play styles, thereby limiting Al's adaptability different user prefer- ences.
Traditionally, research has focused primarily on identifying the Nash Equilibrium (NE), as it guarantees that no player can improve their expected payoff by unilaterally deviat- ing from their strategy. From the perspective of expected payoff's, this guarantee makes the NE considered the opti- mal solution in game. Consequently, many studies treat a game problem as resolved once its NE is identified. How- ever, maximizing expected payoffs is not the sole criterion for evaluating strategy quality.
To address this gap, we propose Preference-CFR (Pref-CFR), novel method that incorporates two key parameters: preference degree and vulnerabil- ity degree. These parameters enable the AI to ad- just its strategic distribution within an acceptable performance loss threshold, thereby enhancing its adaptability to wider range of strategic demands. In our experiments with Texas Hold' em, Pref- CFR successfully trained Aggressive and Loose Passive styles that not only match original CFR- based strategies in performance but also display clearly distinct behavioral patterns. Notably, for certain hand scenarios, Pref-CH CFR produces strate- gies that diverge significantly from both conven- tional expert heuristics and original CFR outputs, potentially offering novel insights for professional players.
First, many practical problems feature multiple NEs. In eco- nomics, understanding the diversity of NE often holds more value than simply identifying one. For example, Schelling's s work on predicting specific NE, which earned him the 2005 Nobel Prize in Economics, exemplifies this significance.
Second, state-of-the-as game- comn--plying AIs (c.g.. AlphaGo and Pluribus) have already surpassed top human players. However, strategies with maximizing expected payoff's as the sole objective are typically overly rational, lack diver- sity, and are difficult for humans to master. For instance, Lee Sedol, top human Go expert who competed against AlphaGo, chose to retire and said, "I can no longer en- joy the game." (Wakabayashi & Young, 2024) This high- lights that competition not the sole purpose of gameplay; games should also provide entertainment and spiritual ful- fillment. Additionally, strategies that incur minor payoff losses against top Als may still perform well against hu- man opponents. For example, in chess, professional players are highly familiar with common openings recommended by AI, often resulting in drawn games. To overcome this, top players deliberately choose unconventional strategies to lead the game into unexplored scenarios, leveraging their superior skills to gain an advantage.
1. Introduction
In machine learning, complex gaming problems are impor- tant benchmarks for assessing artificial intelligence (AT). Prominent games such as Chess (Campbell al., 2002).
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology "Nationa Key Laboratory of Science and Technology on Multispectral Information Process- ing 'GTOKing. Correspondence to: Qi Ju <juqi@ ceii@@husteuuuu eddu.en>. Thomas Tellier <thomas@ Aahooas@gtooiig..cmm com?, Zhemei Fang <zm- fang2018 huu.t.cdu.nn"..
To the best of our knowledge, algorithms for incomplete- information games have not considered the importance of identifying diverse NEs, nor have they explored how strat- egy ranges change when some payoff is sacrificed. To ad- dress these limitations, we propose novel algorithm called Preference Counterfactual Regret Minimization (Pref-CFR). This algorithm introduces two parameters for strategy selec-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
tion from the perspectives of style and diversity: the degree of preference 6, representing player's inclination toward specific strategy (the style of strategies), and the degree of vulnerability 5, denoting the maximum exploitability player is willing to tolerate (the diversity of strategies). In two-player zero-sum games, setting å ensures convergence to NE aligned with the specified preferences. the game has unique NE, setting å alone (3 0) does not affect the strategy's is convergence. Incorporating 3 further enables convergence to e-NE (e S 3), significantly expanding the selection of preferred actions. The combination of ō and & can derive a strategy that best suits the user's 's style at a given tolerable loss. In Texas Hold' em experiment, we success- fully obtained strategies exhibiting Aggressive and Loose Passive play-styles. Results indicate these strategies exhibit significant differences from original CFR-trained strategies while maintaining comparable performance in head-to-head matches. Notably, Pref-CFR uncovers novel strategies. For example, in the Aggressive style, it sometimes raises with weak hands like 820-a a move previously missed by both human experts and Game Theory Optimal (GTO) solvers, offering insights for professional players. Our code can be found at GitHub.
However, a common limitation persists: these methods pri- marily target NE computation, overlooking the applicability and diversity of equilibria discussed earlier.
While equilibrium-solving algorithms have advanced, game theory research has also diversified into non-NE equilib- rium concepts. The 2005 Nobel Prize in Economics rec- ognized Aumann and Schelling for refining equilibrium theory: Aumann demonstrated that correlated equilibria yield fairer and more efficient outcomes than NE (Aumann, 1974), while Schelling analyzed equilibrium selection in multi multi-equilibrium scenarios (Schelling, 1980). Fudenberg et al. contended that stable states reached during the learning (or evolutionary) process can also be regarded as equilib- rium (Fudenberg & Levine, 1998). Recently, Ganzfried introduced novel concept called safe equilibrium, which takes into account the irrationality of opponents and enables more flexible responses to various adversaries (Ganzfried, 2023). Despite these innovations, a critical void remains: existing studies rarely address the computational methods for solving these novel equilibria. This methodological gap motivates our synthesis of algorithmic and conceptual research streams.
This paper integrates algorithmic rigor in equilibrium solv- ing with conceptual advances in non-NE research to bridge theory and computation. We aim to define novel stylized equilibrium strategies and demonstrate how learning the- ory enables their precise computation, uniting theoretical innovation with algorithmic practice.
Related Work
The related work in this paper is structured into two in- terrelated categories. First, we survey the foundational algorithms for solving game equilibria, focusing on their convergence properties and limitations. Second, we ex- plore research that transcends traditional NE and Coarse Correlated Equilibrium (CCE), examining efforts to define "better" strategies.
2. Notion and Preliminaries
2.1. Game Theory
In the realm of normal-form games, the preeminent algo- rithm is Regret Minimization (RM), along with its vari- ant, Counterfactual Regret Minimization (CFR) (Zinkevich et al., 2007), which applied in extensive-form games. RM/CFR can converge to the NE in two-player zero-sum games and to CCE in multi-player general-sum games (Han- nan, 1957). Noteworthy variants include CFR+ (Tammelin, 2014). Monte-Carlo CFR (MCCFR) (Lanctot et al., 2009), and Discounted CFR (DCFR) (Brown& & Sandholm, 2019a). In particular, CFR+ and MCCFR have played a pivotal role in significant Al advancements over the past decade (Brown & Sandholm, 2019b; Bowling et al., 2015; Brown & Sand- holm, 2017). Besides the RM algorithm, the Fictitious Play (FP) algorithm is another commonly-employed approach for solving games. FP was initially introduced in Brown's 1951 paper (Brown, 1951), and the treatise The Theory of Learn- ing in Games (Fudenberg & Levine, 1998) further solidified previous research, establishing a standardized framework for FP. In two-player zero-sum games, FP is proven to converge to the NE. Recently, Qi et al. integrated CFR with the FP algorithm to propose the CFVFP algorithm (Ju et al., 2024).
2.1.1. NORMAL-FORM GAME
The normal-form game is the fundamental model in game theory. Let N = {1, 2, i.... denote the set of players, where player i has a finite action set A'' The strategy of of player i is defined as a | probability distribution over A (where | represents the number of elements in the set), with (aa)) indicating the probability of player i choosing action a'. Strategies can be categorized into pure strategies and mixed strate- gies: a pure strategy involves taking specific action with 100% probability, while all strategies other than pure strate- gies are considered mixed strategies. A strategy profile a = x of is a collection of strategies for all players, and iEN = -a'.....o'-1, _0,,__11,, ..... refers to all strategies in except for player i. The set of all strategy profiles is de- noted as Σ- = x D' We define the finite payoff function iEN
w' : Σ + R, where a'(o'o"") represents the payoff re- ceived by player when player i selects strategy o¹ and all other players follow the strategy profile -i, Finally, we
2

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
define L EW u' ('())- min (o) as the payoff' interval of the game.
2.2. Counterfactual Regret Minimization
In normal-form games, let oj be the strategy used by player at iteration t, and the regret of player for not choosing action E AA' is defined as:
2.1.2. EXTENSIVE-FORM GAMES
In extensive-form games, which are commonly depicted as game trees, the set of players is represented as W (1.2....). J. The nodes s within the game tree signify pos- sible states, collectively forming the state set E S, while leaf nodes: z Z denote terminal states, For each state S. the successor edges delineate the action set A(s) accessible to either player chance events. The player function P: : S - NU {c} specifies which entity acts at particular state, where e represents chance.
(a) - 1 -
Σ u (c) (ot);
(4)
the new strategy is generated as follows:
(a) Eacsi By (a)
if iG (a')
(a') #
o1+1(a)=
(5)
otherwise,
where (a) max (R) 0) and 0 denotes the zero vector. Since the probability of taking action o (a) is pro- portional to the regret value N2 (a), this strategy is called the regret matching strategy. Define o.. = Ei= of as the average strategy of player i. When T + oc, 0% con- verges to NE in two-player zero-sum games with a conver- gence rate of O (T-1/2).
Information sets E I' comprise collections of states that player cannot distinguish from one another. The payoff function R Z |F assigns payoff vector for the players based on the terminal states. The behavioral strategy (1()) E RIA(1)) is defined as a probability distribution over each information set I for all I E I'. We also define To (I) as the probability of encountering information set when all players select actions according to the strategy profile a.
In extensive-fore games, we define the counterfactual value u( a)) the expected value conditioned on reaching the information set I while all players adopt the strategy o, with the exception that player i plays in way that allows reaching /. For every action a E (T),,, we denote a I- as the strategy profile that is identical to a except that player always selects action a when in information set I. The average counterfactual regret defined as:
2.1.3. NASH EQUILIBRIUM
The best response (BR) strategy for player i in relation to the strategy profile o is defined as:
(1) The BR strategy can either be a pure strategy or a mixed strategy; yet, identifying pure BR strategy is generally more straightforward. For the purposes of our analysis, we will assume that b(o) is pure strategy. In this context, arg max denotes the action that produces the highest pay- off within a given set. If there are multiple actions that yield this maximum payoff, the strategy that appears first in lexicographic order will be selected.
(( arg a tt
T
T.. (T) (u' (1,0rle-sa) - (1,o1)). R1 0) T
1
Σ
(6)
where (1) is the probability of information set / oc- curring given that all players (including chance, except for player i) choose actions according to o.. We define R4 (I, a) max( max(Pi (T, a), 0). The strategy for player i at time T + is given by:
In two-player zero-sum game, the deviation incentive of player i for strategy profile a is defined as:
R
if if (1) #
(T,
Eecai R (L.))
é = ut (b' " (o))
(2)
otherwise.
[A(Z)
while the overall exploitability across all players is calcu- lated as:
The average strategy (T) () for information set 7 after T iterations is defined as:
I Σ
(3)
TEN
Th th (T)o(1)
(8)
if = O, the strategy profile a is NE; otherwise, it termed an e-NE, If an algorithm ensures that the exploitability of the game meets the condition e < CT after T iterations (where C is constant), then the convergence rate of this algorithm is O(T-1). This formula can still serve basis for strategy convergence in multiplayer games. However, since more than one player may deviate from the strategy simultaneously, it is no longer termed exploitability but NashConv.
EtITT (I)
Ultimately, T + 00, ōT will converge to NE.
3. Motivation
3.1. Style and Diversity of Strategies in the Game
This paper re-examines strategy selection through the lenses of "diversity" and "style", an aspect that has been relatively
3

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
unexplored in existing research. To formalize this perspec- tive, first define these two metrics in the context of game theory. Specifically, diversity refers to the size of the acceptable strategy space Exce, while style quantifies the similarity between a given strategy distribution and a preferred strategy o².
different initial strategies, it typically converges unique equilibrium point. Formally, we propose the following con- jecture:
Conjecture 3.1. In a two-player zero-sun game, if the set of Nash equilibrium ENE a convex polyhedron, then for any initial strategy otmo, the RM iteration converges to a unique fixed point оRM E ΣNt-
The definition of Exce is relatively straightforward. If player is professional competitor aiming solely max- imize their probability of winning, the acceptable strat- egy space should coincide with the NE strategy set, i.e., Lace = ENE. In contrast, if the game is played casually among friends, where entertainment takes precedence over optimal strategy, then the acceptable strategy space is the set of all possible strategies, E-xx= = Σ.
This conjecture suggests that modifying the initialization of RM alone is insufficient to steer convergence towards different equilibria. Instead, altering the final strategy or requires intervention during the iterative process. While RM lacks prior research in this direction, insights can be drawn from the Generalized Weakened Fictitious Play (GWFP) algorithm, whose update rule is:
In comparison, defining "style" is more intricate. In real- world settings, styles are often measured by macroscopic statistical indicators. Roughly speaking, in football, pos- session percentage is commonly used to characterize Tiki- taka style-wy -wehn possession exceeds a certain thresh- old, the playing style is classified as such. However, such statistical indicators are difficult to integrate directly into game training. As game training is inherently sparse re- ward pwwplem-oobeeeeeeeeee payoffs are only obtained at terminal nodes incorrorncni macro-style indicators (e.g., the entry rate in Texas Hold" em) would require numerous games to generate meaningful style signals. This effectively exacer- bates the sparsity issue, making learning more challenging. To address this, we map style-related measurements from the macroscopic level the decision-makeen formation sets. We define style the distance Dis(ōr, o*) between the final learned strategy or and the preferred strat- egy o². For instance, if a ball possession rate exceeding 70% is defined as characteristic of the Tiki-taka style, and the corresponding strategy set is Emsi- with centroid strat- egy Tikittkka then the degree to which or adheres to the Tiki-taka style can be measured by Dis(ōr, Tik-taaa)
Tet1 (1- -1-00++1000 + aeti + M411,, (10)
where {Mc)r21 is a sequence of perturbation terms, and (o)) is a sub-BR strategy with a gap of & from the BR strategy, that is:
u' (((())))) -u' u'
c
(11)
GWFP converges to the NE when the following three condi- tions are met. First, - 0 as + co. Second, a₁ - 0 as
- oc and Σ121 at = co. Finally:
level within in-
k-1
lim sup 8-+00 k
Σ a4 M++1
Σ
ast1 S T
0.
(12)
We first establish that RM can be interpreted as a special case of GWFP (see Appendix D). Based on this, we pro- pose three methods to influence RM's convergence behav- ior: (i) Adjusting the learning rate ay; (ii) Modifying the exploitability parameter in be, (ōt); (iii) Introducing per- turbation sequence Mc.
Given these definitions, we seek final strategy ōy that min- imizes the distance to the preferred style while remaining within the acceptable strategy space:
oT min Dis(o,a") o Excc
(9)
Among these. the most effective approach is modifying 6,, as we have already defined Dis(a, as the style metric. Moreover, 4 directly corresponds to the vulnerability pa- rameter 33. Therefore, our subsequent improvements will all be centered around b., (o).
The choice of Eace can be guided by different criteria. In this paper. we constrain it based on the worst-case performance loss relative to NE strategies, which we term the vulner- ability degree. Specifically, we introduce a vulnerability parameter 3 during training, ensuring that the final learned strategy satisfies ōT E Σe-NE where e < 3.
4. Method
4.1. Preference CFR
3.2. Controlling the Convergence of RM Iteration
We now introduce the Pref-CFR algorithm. For each infor- mation set, we define a preference degree 5(T) E RIA(), where 8(I, 2 1. Note that in our setting, Va E A(T) 8(I,a) = 1 not allowed. The strategy for the next itera-
Extensive experiments (Section 5.1 provides an example) on two-player zero-sum games reveal that even when multi- ple equílibria exist and the RM/CFR algorithm starts from

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
tion T 1 is calculated as:
a strategy that deviates significantly from the NE, mak- ing it susceptible to astute opponents.
((
if R (T) +
(I,)
aay ((T,)) (T.a) 8(1.0)-1 Eueaien 8(Tra)-1
otherwise.
While can adjust 6(1, a) and ((T) at the micro level, translating macro-level style characteristics rec- ognized by the public into parameter settings for each information set is challenging.
(13)
Alternatively, we can also adopt the BR strategy for the next iteration:
(T) = arg max (1) a). (14)
To address the first problem, setting appropriate parameters for 8(I, a) and (T) can help the final strategy converge to a reasonable interval. For the parameter 8(1, a), experiments indicate that when 8(1, a) S 5, the convergence speed re- mains acceptable, significantly increasing the likelihood of the preferred action being adopted in the final output strat- egy. Regarding 3(7), it can be determined based on the specifics of different games and the experience of human experts. For instance, an error margin of 25 mbb/h in Texas Hold' hold'eemii is typically inconspicuous.
In this paper, we denote Pref-CFR(RM) indicate that the next strategy follows the regret minimization approach, as shown in Equation 13, while Pref- (BeFCCFR(BR) signifies that the next strategy is based on the BR approach, as presented in Equation 14.
We prove in the Appendix B.2 that the convergence of the Pref-CFR algorithm consistent with the original CFR. Specifically, it can converge to the NE in two-player zero- sum games and to the CCE in multi-player general-sum games. The action preference degree 5(a) is related to the preferred strategy o (a); a larger proportion of a' in o (a') warrants a higher 8(a'. How to define a suitable 5(a) will be discussed in the next section.
The second problem is more complex. Fortunately, for common games like Texas Hold'em em, there is ample expert knowledge to draw upon. If a player has a narrow range of hands when entering the pot, their style is classified as "tight"; conversely, a wider range is termed "loose." " For definitions of these Texas Hold' em terms, please refer to Appendix A. If player exhibits a relatively high proportion of 3-bets after entering the pot, they are labeled as "aggres- sive." These styles exhibit strong consistency. Thus, we can apply the same set of ó values across all information sets. For example, to make the Al more aggressive, we could increase 51, Raise) for all raising actions across all information sets.
Introducing 8 drives the strategy toward convergence to different-style equilibria. However, with d alone (B = 0), the acceptable strategy set is Ence = ENE- In many cases, the adjustments are small or have almost no macroscopic dif- ference from non-ō strategies. As highlighted in Section 1, competition is not the sole dimension of game. To address this, we introduce the vulnerability degree B(I). Define:
aI) = (I, ((T),
(15)
the strategy for time T ++ 1 is determined as follows:
However, for more extensive-form games that lack extensive expert analysis, the design of 8(I, a) and (()) remains an area requiring further research.
arg (L,) atA: 5(1.0)-1
if B (1,
(T,)(
otherwise,
Encrion 8(Ta3-I
5. Experiments
(16)
where (1, max (1, a), 0}. In Appendix B.3, we prove that in normal-fore games, the vulnerability de- gree 3 ensures the final strategy is an e-NE strategy with c S 3. By integrating preference and vulnerability, we devise strategy that not only aligns with desired stylistic requirements but also minimizes potential losses.
Our experiments are conducted using Kuhn poker (Kuhn, 1950), Ledue poker (Shi & Littman, 2001) as well as two- player and three-player Texas Hold'en ei poker. A brief introduction to the rules of these games can be found in Appendix A. The experimental codes for Kuhn poker and Leduc poker have been made publicly available on GitHub.
In our Kuhn poker experiments, we used the vanilla Pref- CFR while Texas Hold' ao experiments utilized a variant of the multi-valued state technique (Brown et al., 2018). Solutions were computed in under 10 minutes with 24- core CPU; subgames included 35k states and the full game used for leaf estimates had 25M states. In Hcads-Up play, this setup achieved exploitability below 4 mBB/h in our Texas Hold' em experiments. These settings are comparable to previous experiments (Brown et al., 2018).
4.2. Analysis of Pref-CFR Algorithm
Pre-CFR offers a method for converging to different strate- gies. Nevertheless, during actual training, the settings for 8(T) and 3 still require careful configuration by experts.
1. Setting larger values for 5(I, a) and (T) can enhance the distinctiveness of the final strategy's 's style. How- ever, this may result in slower convergence speeds and
5

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
5.1. Pref-CFR Converges to Different Nash Equilibria
gesting Ledue poker may have single NE. As analyzed in Appendix B.2, setting 8 0 implies Exx ΣeE, so different ò values lead to convergence to the unique NE.
Before analyzing the Pref-CFR algorithm, crucial to first elucidate the scenarios where CFR fails to converge to distinct NEs. We will use Kuhn poker as example. In this game, player 1 faces multiple equilibria. The equilibrium strategy for player in Kuhn poker is detailed in Table 1.
Users then face a trade-off: sacrifice some utility (com- pared to the NE strategy) to diversify the strategy. Fig. 4 reveals that a larger 3 increases the Raise probability in the final strategy, making it more aggressive, but at the expense of higher exploitability. It is also noteworthy that in the Ledue poker experiments, the difference between setting 8( (raise) = 10 and (raiss)) = 5 is negligible. Therefore, we set ō(a) = 5 in subsequent experiments.
Infoset The probability of bet
J PB
Q
Q
PB
K
PB
a
0
0
1/3ta
30
Table 1. The equilibrium strategy of player in Kuhn poker, where E [0, 1/3). It can be considered that player has countless equilibrium strategies in this game.
5.2. Pref-CFR Converges to Different Styles in Texas Hold'em em
The traditional CFR algorithm does not account for goals beyond expected payoffs. In Texas Hold' em, two significant "style demands" have consistently emerged. yet the previous CFR algorithm was unable to address them.
The equilibrium strategy for Player I can be defined by the parameter a, which corresponds to the probability of choos- ing to bet in the information set J.. After iterating through 10 nodes, the exploitability of the strategy is reduced to below 0.001. At this point, we approximate the equilibrium strategy with a = aJ_ Bet).
1. Aggressive strategy A prominent feature of this strat- egy is its higher probability of raising. For some am- ateur players with an abundance of chips, sensitivity to chip loss is minimal. Their primary motivation for participating in Texas Hold' e' is not necessarily to win more chips but rather to seek novel experiences or enhance the atmosphere social gatherings. As a result, these players often look forward to engaging in games with larger pot amounts. In training, we set (I, raise) = 5 and 3 = 0.05 at the first decision node of player 1.
We initialize the CFR training with a randomly distributed strategy. As depicted in Figure 1, although Kuhn poker admits multiple NEs, CFR converges to a single equilibrium at a = 0.2, irrespective of the initial strategy. This behavior is not exclusive Kuhn poker; as Conjecture 3.1 posits, altering the initial state of RM does not affect its final con- vergence point. Furthermore, the strategies generated by the CFR algorithm closely resemble traditional "machine strategics," " which are complex, stable, and lacking distinct characteristics. In practical gameplay, human players can readily recognize and memorize simpler strategies, such as and a= 1/3. For example, when a O, player I always selects Pass, embodying cautious and conser- vative playstyle. Conversely, Pref-CFR can identify these characteristic equilibria.
Loose passive strategy A significant characteristic of this strategy is that the probability of folding is lower than that of typical strategy. This phenomenon arises from the difficulty many players with small number of chips experience when it comes to folding. After in- vesting a considerable amount of chips during the flop or turn stages, these players often feel uncomfortable quitting the game without knowing their opponent's 's hole cards. The discomfort is especially pronounced when they suspect they have been successfully bluffed. By minimizing folds, this strategy provides psycholog- ical comfort for these players, helping them avoid the frustration of being outplayed by an opponent's 's bluff. In training, we set OT, call) = 5 and 3 = 0.05 at the first decision node of Player
The parameter configurations for these experiments are de- tailed in Appendix C. As shown in Figure 2, the convergence rates across all settings remain comparable to that of the original CFR. However, all variants of Pref-CFR yield equi- librium that deviate from a 0.2, with higher preference degree settings resulting in more pronounced deviations. Moreover, the performance of Pref-CFR(RM) is notably in- ferior to that of Pref- (BR(R.....BR......... Consequently, we advocate for the use of the Pref-CFR(BR) method, which was also utilized in all subsequent experiments.
Kuhn poker, game with multiple NEs, allows PrefCFR to converge to different NEs when 3 = 0. In contrast, as Fig. 3 shows, in the Ledue poker experiment, with 6rraise) = 10, B = 0 and 8(call) = 10, B = 0, both eventu- ally converge like the original CFR. Only by introducing 3 can the strategy deviate stably from the standard one, sug-
Before analyzing style variations among AI models, we first compare their performance. Our goal is to develop Als with distinct play styles without sacrificing significant payoffs. Given the challenge of evaluating strategy effectiveness in large-scale games, we use head-to-head matches to assess different algorithms. As shown in Table 2, performance
6

---

Preference-CFR, Beyond Nash Equilibrium for Better Game Strategies
Kuhn Poker
NE
03
0.7
log(0(Exploitabiigg
-1.0
0.5
-13
The value the
0.4
20
02
-30
log (O(Noess touched)
log (oddess touched)
Figure 1. Convergence rate of CFR in Kuhn poker (left) and fluctuation of in CFR algorithm iterations (right). Thirty experiments performed for each setting, and the shaded area indicates the 90% confidence interval these trials (the settings remain unchanged subsequent experiments). It can be seen that regardless of the initial strategy, all CFR iterations converge 0.2.
Kuhn Poker
of appoximatte NT
1.0
-0.
Ogg0(Exploitabllgg
-1.0
0.0
-1.
-20
0.4
-25
The alpta P
-30
0.0
log O(Nodes touched)
log ONNoess touched)
PreSTTR/RM
PeeCFR/BR) Ier
ProICFR(BR) of
ProICFR(RM) def'
Figure Convergence rate CFR/PIef-CFF Kuhn poker (left) fluctuation a value during CFR/Pref- CFR iterations (right). It is evident that the Pref-CFR algorithm can still converge to equilibrium, with convergence speed comparable to that of the original CFR. Additionally, the right figure clearly shows Pref-CFR, strategies converge to results deviating from the NE, and these deviations become larger with higher values of (3.
Unit: mill',,
Big Blind (Plaeer 2)
skill levels.
90% confidence interval: 1.0
Aggressive
Narmal
Loose
Small Blind
Apgressive
68.6
58.7
67.6
Our experiments clearly demonstrate how the algorithm
Normal
73.2
(Payer 1)
Looge-Pansive
74,7
66.9
73.8
influences the final strategies in the first row of Figure 5. In standard CFR training for two-player Texas Hold'em, the average preflop strategy distribution [5.4%, 52.7%, 42.0%, 0.0%, 0.0%) for folding, calling, raising to 2, raising to 3, and going all-in, respectively. In contrast, the loose Al's strategy distribution shifts [0.3%, 73.3%, 21.7%, 4.7%, 0.0%], while the aggressive AI shows [4.2%, 64.8%, 9.8%, 21.1%, 0.0%]. As expected, the loose AI exhibits much lower folding probability, dropping from 5.4% to 0.3%-a 94.4% decrease. The differences between aggressive and standard strategies are also pronounced, with the aggressive AI raising to a 21.1% rate compared 0% in the standard strategy.
Table 2. The battle results hetween different AIs in two-player Texas Hold' em. In Texas Hold's em, mBB/h represents the thou- sandth of big blind measure player's profit or loss for each individual hand.
lost per hand, used to accurately
differences across styles remain minimal in both two-player and three-player games, staying within 10mBB/h. Prior search suggests that training error below ImBB/h indicates convergence to NE (Bowling et al., 2015). For reference, Pluribus achieved a 32mBB/h win rate against top human players (Brown & Sandholm, 2019b). Thus, the observed 10mBB/h difference suggests these Als are of comparable
We believe these Al strategies provide valuable insights for human players. For example, traditional human ex-

---

Preference-CFR, Beyond Nash Equilibrium for Better Game Strategies
Ledue Poker
1.0
1.0
The probubiltty of choose Call
0.5
0.3
log10Exxloitahiing
0.
0.5
0.4
1.2
-1.5
4.0
4.5
50
5.5
6.0
6.5
7.0
7.5
3.0
4.0
5.0
5.5
6.0
6.5
7.0
75
3.0
log Nodes touched)
loglO(Nodes touched)
Nonmll
1ccall)-10, p-0.05
6friise)-100 10, 8-0.00
Bemiso)-55 $+0.05
dcally 10, p-0.00
Kcalll 5, p-0.05
6traise) p-0.05
Figure 3. Convergence rates of ES-MCCFR/Pref-ES EEEEEE--MII MCCFR in Ledue poker (left) and the fluctuations in the probability of choosing Call during MS-MCCFR/PPef-ES-MCCC iterations (right). This figure shows that Ledue poker, strategies converge to different equilibria only when B > 0 is set.
Leduc Poker
1.0
Call
0.5
0.8
log 1gg10xxxpotbbiitty
0.0
The probubilitt
0.4
-1.0
0.0
log OONoess
log (NodesNodes touched)
Nonual
40mise)-55 Stmiso)=3, (-0.20
Sirise))- p-0.50
80mise)-5, 8-0.05
Figure 4. Convergence rates of T.S-MCCFFR/PPEATS-MCCC in Ledue poker (left) and the fluctuations in the probability of choosing Call during IS-MCCCRR/reL-ES-MCCCC iterations (right). Obviously, the larger /3 is, the higher the probability of choosing to Call and the more obvious the strategy style is.
perts and GTO-based AIs typically fold weak hand com- binations like 82o and 72o, considered poor due low straight potential, unsuitedness, and low card ranks. In con- trast, our loose-passive Al opts to call almost 100% of the time, while the aggressive AI may even raise with these hands. Such stylized strategies cleverly adjust hand dis- tributions to obscure hand strength, akin to "hiding leaf in the forest." The loose-passive Al's higher calling rates make it difficult for opponents to distinguish between a heak-hand bluff strong-hand slow-play. Similarly, the aggressive AI increases raising frequency across all hands, potentially challenging conventional Texas Hold'en hold'em tactics. While loose-passive play was historically deemed subop- timal, our results show can be a viable strategy when balanced across all hand combinations, offering a fresh tac- tical perspective. The results of three-player AT matches,
Unit mBRS, confidence
Small Blind (Player 3) Bip Blind (Plsyer
Agurasive
Normal
Louse-pasrive
Aggressive
247.2
251.0
259.1
(Payer 1)
Normal
256.3
252.2
200.0
Table 3. The battle results between different Als in three-player Texas Hold'ent
displayed in Table 3 and the second of Figure 5, show that our algorithm achieves even pronounced effects in three-player scenarios. In standard three-player training, the average strategy distribution is [61.4%, 0.0%, 38.6 38.6%, 0.0%]. This example illustrates why the loose boose-passive style was previously considered ineffective: according to GTO calculations, is typically not recommended. However, our
8

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
(a) 2P Loose passive Al
(b) 2P Normal Al
(c) 2P Aggressive Al
(d) 3P Loose passive AT
(c) 3P Normal AT
(f) 3P Aggressive AT
Figure 5. Strategy display for Texas Hold' em. In the top left corner of each image, the current player's 's information and available actions are displayed. The central showeases the strategies for different hand combinations this stage. In Texas Hold' em poker, there are 13 ranks across 4 suits, with no distinction in value between suits, resulting in 169 unique hand combinations. These are represented x3x 13 matrix, where the lower left displays offsuit hands and the upper right shows suited hands. Each matrix element's color indicates the strategic choice for the corresponding hand: blue for folding, green for calling, red shades for raising (with deeper red shades indicating higher raises), and black-red for going all-in. The bottom row provides an overview the average strategies across all hands, allowing for a visual understanding of the overall strategy distribution.
experiments reveal that the Loose-Passine strategy not only avoids any loss in earnings but actually gains an advantage of O.2mBB/h compared to the Normal strategy, while sig- nificantly increasing the calling probability from 0.0% to 23.9%. We speculate that the added complexity of three- player poker compared to two-player games accentuates the impact of style changes.
of preference degrees for actions in each information set, restricting its adaptability to varied player styles. Given the game's complexity, human users face challenges in implementing context-dependent strategies. To enhance practical utility, we propose automating the translation of user-specified metrics (e.g.. betting frequencies or pot en- try rates) into informatiion-set-ppecii preference weights, streamlining the customization workflow. Additionally, ex- tending this framework to interdisciplinary domains-sur as behavioral economics or market dynamics dynanics--peseents presents an intriguing avenue for bridging game-theoretic algorithms with weal-world decision systems.
6. Conclusion and Prospect
The Pref-CFR algorithm proposed in this study addresses a key limitation of traditional CFR by enabling the discovery of diverse equilibria. Our experiments demonstrate suc- cessful training of Texas Hold'em em AIs with distinct strategic profiles. For example, in three-player games, the Aggressive Al increased its 3-bet probability from 0.0% to 16.0% with only 10mBB/h performance decrement, while the Loose Al raised its calling rate from 0.0% to 23.9%. Notably, the algorithm maintains training efficiency and supports real-time strategic guidance for human players.
However, the current framework requires manual calibration

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Impact Statement
Hannan, J. Approximation to bayes risk in repeated play. Contributions to the Theory of Games, 3(2):97-139, 1957.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Ju, Q.. Hei, F., Feng, T., Yi, D., Fang, Z., and Luo, Y. Ac- celerating nash equilibrium convergence in monte carlo settings through counterfactual value based fictitious play. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. 62103158).
Kuhn, H. W. A simplified two-person poker. Contributions to the Theory of Games, 1:97-103, 1950.
References
Lanctot, M., Waugh, K., Zinkevich, M., and Bowling, M.
Monte carlo sampling for regret minimization in exten- sive games. Advances in neural information processing systems, 22, 2009.
Aberntthy, J., Bartlett, L., and Hazan, E. Blackwell approachability and no-regret learning are equivalent. In Proceedings of the 24th Annual Conference on Learning Theory, Pp. 27-46. JMLR Workshop and Conference Proceedings, 2011.
Moravčík, M., Schmid, M., Burch, N., Lisy, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508 513, 2017.
Aumann, J. J. Subjectivity and correlation in randomized strategies. Journal of mathematical Economics, 1(1): 67-96, 1974.
Schelling, C. The Strategy of Conflict: with new Preface by the Author. Harvard university press, 1980.
Bowling, M., Burch, N., Johanson, M., and Tammelin, O. Heads-up limit hold' em poker is solved. Science, 347 2015.
Shi, J. and Littman, M. L. Abstraction methods for game theoretic poker. In Computers and Games: Second In- ternational Conference, CG 2000 Hamamatsu, Japan, October 26-28, 2000 Revised Papers 2, pp. 333-345. Springer. 2001.
Brown, G. W. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1):374, 1951.
Brown, N. and Sandholm, T. Safe and nested subgame solving for imperfect-information games. Advances in neural information processing systems, 30, 2017.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of with deep neural networks and tree search. nature, 529(7587)4444448,, 2016.
Brown, N. and Sandholm, T. Solving imperfect-inforee intommatinnn games via discounted regret minimization. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1829-1898888 1836, 2019a.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel. T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Sci- 2018.
Brown, N. and Sandholm, T. Superhuman ai for multiplayer poker. Science, 365(6456) 2019b.
ence, 362(6419):1140
Brown, N., Sandholm, T., and Amos, B. Depth-limited solving for imperfect-informatior games. Advances in Neural Information P Processing Systems (NeurIPS), 31, 2018.
Tammelin, O. Solving large imperfect information games using cfr+, arXiv preprint 1eXiv:1407.5044 5042, 2014.
Vinyals, O., Babuschkin, I., Czamecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350 354-344,,, 2019.
Campbell, M., Hoane Jr, A. J., and Hsu, F.-h. Deep blue. Artificial intelligence, 134(1-2):57-83, 2002.
Fudenberg, D. and Levine, D. K. The theory of learning in games, volume 2. MIT press, 1998.
Wakabayashi, D. and Young, J. Y. Defeated by a.i.,
legend in the board game go warns: Get ready for what's next. The New York Times, 2024. ISSN 0362-4331. URL 10 https::/www.nytee
Ganzfried, S. Safe equilibrium. In 2023 62nd IEEE Con- ference on Decision and Control (CDC), pp. 5230-5236. IEFE, 2023.
com/ 20////224///// atmmm......... html.
10

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Zinkevich, M., Johanson, M., Bowling, M., and Piccione, C. Regret minimization in games with incomplete informa- tion. Advances in neural information processing systems, 20, 2007.

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
A. Game introduction
A.1. Kuhn poker
Kuhn poker a simple poker game. Here is a detailed rules introduction to it:
Card composition: Kuhn poker uses only three cards: J, Q, and K. Each player can only get one card at time.
Action sequence: Usually, two players participate in the game. At the beginning of each round, each player places a blind bet first. Then each player will randomly receive a card. Subsequently, players take actions in turn. The action options are Bet and Pass. If one player bets, the other player can choose to call(Bet) or fold(Pass).
Winning determination: Finally, if player calls, both players show their cards and compare the sizes. The player with the larger card wins and takes the chips in the pot. In Kuhn poker, K > Q> J.
Kuhn poker is a classic model in game theory research. Due to its simple rules and limited card types and action choices, it is convenient for mathematical analysis and theoretical derivation.
A.2. Leduc poker
Ledue poker is a simplified poker game that extends Kuhn poker with additional complexity, making it key model for game theory research. Here is detailed introduction to its rules:
Ledue poker uses six cards from two suits (e.g., spades and hearts), specifically J. Q, K of each suit. Each player is dealt one hole card, and one community card is placed face up.
Card composition:
Action sequence: The game involves two players and two betting rounds:
First round: Players post blinds, then receive hole cards. Actions include Bet or Check (pass). If player bets, the opponent can Call or Fold.
2. Second round: A community card is dealt, and players act again. Actions remain Bet, Check, Call, or Fold, with betting limits typically set to standardize stakes.
Winning determination: If both players call by the end of the second round, they reveal their hole cards. Hand strength is determined by:
1. Pair: Hole card and community card of the same rank (e.g., J-J) is the strongest.
High card: If neither pair nor flush, compare the higher card (K & Q ? J); tied, spades suit prevails over hearts.
Ledue poker bridges the gap between simple and complex games, featuring two betting rounds and community cards that introduce strategic depth. Its structured complexity- intricate than Kuhn poker but less complex than Texas Hold' hold'em-makee it ideal for testing algorithms in extensive-form games and studying equilibrium strategies in partial- information scenarios. Our experiments used a 12-card (6-pair) setup.
A.3. Texas Hold'em
Texas Hold' em is poker game that has large player population and extensive influence globally. In terms of tournaments, numerous international Texas Hold' em competitions draw significant attention. High prize money attracts many top players to participate. Socially, it is popular activity for people to enhance communication during leisure gatherings. The following is an introduction to some basic Texas Hold' em poker terminology. For more content, you can refer to Wikipedia.
The stage before the community cards are dealt. Decisions are made based on hands, position, etc., like raising, calling or folding.
Preflop:
Flop: The first three community cards dealt. Evaluate competitiveness combined with hands and decide strategies.
Turn: The fourth community card after the flop.
12

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
River: The last community card. Decides the final decision.
Big blind (BB): The big blind is forced bet made by one of the players before the cards are dealt. It is typically twice the size of the small blind.
Small blind (SB): The small blind is also forced bet made by player before the cards dealt.
Button: The button indicates which player is the dealer for that hand. The player on the button has certain advantages in terms of position and play order.
mBB/h: In Texas Hold' em, mBB/h represents the thousandth of big blind lost per hand, used to accurately measure player's profit or loss for each individual hand.
Play many hands with a low entry criterion. Often participate even with weak hands. High risk but may win with weak hands against strong ones.
Loose:
Actively bet or raise to put pressure on opponents and control the game to win the pot. The hand may not be strong.
Aggressive:
Bet: Raise again after someone has raised. Often indicates strong hand or wanting to pressure opponents and increase the pot.
All-in: Bet all chips. Due to confidence in the hand or having few chips and wanting to pressure.
Call: Follow by betting the same amount of chips as the opponent. Want continue to see the cards and compete.
Fold: Give the hand and not participate in the pot. Due to weak hands or high risk from large bets by opponents.
Bluff: When having a weak hand, make large bets and let opponents think the hand is strong so they fold to win the pot. High risk.
B. Proof of Convergence of Pref-CFR
B.1. Blackwell Approachability Game
Definition B.1. A Blackwell approachability game in normal-form two-player games can be described as tuple
(E, u, S., ,S²), where a strategy profile set, u is the payoff function, and Si = =R a closed convex target cone. The Player i's regret vector of the strategy profile a is (e(a)) E R, for each component (a az) = w² (az, 0 u² (a), ag E A the average regret vector for players to take actions T time a is Ri
T
Σ (o)),
(17)
T
at each time 1, the two players interact in this order:
Player I chooses strategy of E1
Player 2 chooses an action of E L2 which can depend adversarially on all the o, output so far;
Player 1 gets the vector value payoff (ot) E 11
The goal Player is to select actions of oz....
... E " such that matter what actions of, o2, ................ 22 played by Player 2, the average payoff vector converges to the target set S!
min R/ +
T - xo.
(18)
Before explaining how to choose the action a, to ensure this goal achieve, we first need to define the forceable half-space:
13

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Definition B.2. Let H C Ra as half-space, that is, for some E IR, b E R. H { Rd a UJ. In Blackwell approachability games, the halfspace H is said to be forceable there exists strategy o Ex of Player that guarantees that the regret vector (o) is in H matter the strategy played by Player -in such that
R¹ (o" (
E H
Vo-- E
(19)
and ot* is forcing action for H.
Blackwell's s approuchability theorem states the following.
Theorem B.3. Goal 18 can be attained and only if every halfspace H4 2 S is forceable.
The relationship between Blackwell approachability and no-regret learning is: Theorem B.4. Any strategy (algorithm) that achieves Blackwell approachability can be converted into algorithm that achieves no-regret, and vice versa (Abernethy et al., 2011).
If the algorithm achieves Blackwell approachability, the average strategy o/. will converge equilibrium with cc. The rate of convergence is e & Ry & I. Lv/IAT/
B.2. Pref-CFR Achieves Blackwell Approachability
Preference
transformation
R
8R
V
H, PM..-RRM
HRRM
Yt
74
S
Figure 6. The differences in the selected foreing actions and foreing half-spaces of RM, FP and Pref-RM in the two-dimensional planc.
The proof concept for the Pref-CFR algorithm is as follows:
an algorithm guarantees that the regret value converges to zero in normal-form games, applying this algorithm to two-player zero-sum games will converge to the NE and to the CCE in multi-playe general-sum games. Furthermore, Theorem B indicates that any algorithm satisfying Blackwell approachability is equivalent to a no-regret algorithm. Additionally, as shown in paper (Zinkevich et al., 2007), if an algorithm ensures that regret converges to zero normal-form games, then its application in extensive-form games leads to the convergence counterfactual regret to zero (i.e., behavioral strategy convergence to NE two-player zero-sum games). Thus, to demonstrate that Pref-CFR converges NE in two-player zero-sum extensive-form games, we need only establish that it satisfies Blackwell approachability.
In this proof, we primarily show that Pref-CFR(BR) meets the criteria for Blackwell approachability, while Pref-CFR(RM) can derive similar results through a relatively straightforward conversion.
Let Ri represent any component of the vector IN such that R > 0. Next, we identify the point bb, R R2 E R on the axis (noting that this point is not necessarily located on the surface of the target cone S²). We can define the normal R which allows us to determine the half-space defined by this normal vector and point ub,
vector as m.
मt = fz E ERRAAAA
(R xb,)) S (RC TT $1
(20)
14

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Since Ri 4/,- Ri and (Rg 1b))) TTT, O, we can simplify this to:
He { E : ( Ri 2 =0)
(21)
for any point s' E St there is R - 0. Then we need to find the forcing action that matches H.. According to
Definition B.2, need to find 141 E' that achieves R (o (41 Hiff for any 0 E- For simplicity, let t= [u' (a1.0 ".... E Bld'l, we rewrite the regret vector as R2 ( = -e(,0 1, we are looking for a o E E such that:
Rt 0 Hi
R l (l,ott1) 1) <0
R
(l,0tt1) (Ri 1 0
(k,o(tz) | || 50
Rs
(22)
R
l
(l, ( 50
R
R
l
in 1+1
50. 0.
R
Therefore, the strategy 0:41 = || ||, can guarantee KP to be forceable half-space. Figure 6 intuitively shows the relationship between these points, half-spaces and the target set in a two-dimensional plane.
R:
should be noted that the half-space Hf41 can only prove that the distance from the average regret value RG to the point bb, converges to O, but bb is not necessarily on the target cone St. So it cannot be shown that Ri will converge to S The projection of R2 to the target cone S' is point yi [R]] R min (0, R We need keep the distance from the R to point ub, and point 7; within certain range, and preference degree 8(a) just plays this role.
Define a = arg max (a), a. = arg max 8(a) (a). The distance from the average regret value R2 to 7 is | 2|,, and the distance from R2 to the point b, is d'(a")) (od"))) R (a"".. Define dist(z, y) as the distance between points r and y.
dist R2, 71) R 12 S VI (R)
(23)
at the same time,
dist(It,, (b) 8' ('(aP) + (a 2 (R. ) (a (RB))
(24)
since (aBB)) 2 1, so:
((A dist( Ri, v.). 8'(aBR)
dist(Ri, 7.) s
(25)
Define 8" = ( Therefore, from Blackwell's approachability, we know that dist tb,) < Lv Ay/8" Finally, we get:
vt
L|
dist(R), 7e)
(26)
Vi
The convergence speed will linearly slow down as 8* increases. As long as 8* set within a reasonable range, will not significantly reduce the convergence speed of Pref-CFR.
B.3. Vulnerability CFR Will Converge to an e-NE
The proof for the Vulnerability CFR algorithm is more straightforward In the original scenario, the historical strategy oj
converging to a NE is equivalent to the regret R. converging to the convex target cone S R go
This holds true because,
15

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
R
-HRM
B
Ytt
$2 3
H
RM,3
B
Figure 7. The differences in the selected forcing actions and forcing half-spaces of RMM and 3 vulnerability RM in the two-dimensional plane.
given the opponent's strategy o, player identifies a strategy oj such that regardless of which pure strategy player selects, their payoff will not increase (since [2) (a) < for all AA' + oo), thus satisfying the conditions for NE.
However, what if our target cone is not S* RR but instead S IR AA'l for some 3 Z 0? This means that the regret of any pure strategy will not exceed 3, indicating that we have identified B-NE
In the iteration, we only need to adjust the translation. The original projection point is 7 = R and the distance that needs to reduced is given by:
dist( RG, 7.) = ||; 7ell2 = || 112.
(27)
Now, the projection point becomes 7 = R2 -3 where R min, 33. At this point, the distance that needs to be reduced is:
dist 7:) = R 7i l| = ||A - -112.
(28)
Thus, in the CFR iteration, we simply need to replace (a) with (a) = (a) - to ensure that the final strategy converges to a B-NE. Figure intuitively illustrates the relationship between these points, the half-spaces, and the target set in two-dimensional plane.
Strictly speaking, in many of our subsequent experiments, both Pref-CFR and Vulnerability CFR used simultancously. However, Pref-CFR is the more critical component of the algorithm. Using Vulnerability CFR alone does not produce meaningful results. Therefore, even though Vulnerability CFR is utilized, it will not be mentioned in the name of the algorithm.
C. Kuhn Poker Experiment Setup
The settings in the experiment are follows:
1. Pref-CFR(BR) with 8[L_Bet), ((Q..BBe) o(K_ 10.
2. Pref-CFR(BR) with 8I__Bet), (QQ..eet
5.
3. Pref-CFR(RM) with ( 8(K.. 5.
4. Pref-CFR(RM) with dJ 6(K_,
((L_Paass),(((_PPss)) 5. Pref-CFR(BR) with {J__Pass), ((Q_.a(s)) 5(__Passs) 5.
6. Pref-CFR(BR) with ((L_Pass)) (as_ 10.
In Kuhn poker, there are 12 information sets. For all actions in all information sets except for those on the information sets specified here, 5(T,a) a) =
16

---

Preference-CRR: Beyond Nash Equilibrium for Better Game Strategies
D. The RM Algorithm is GWFP Process
The final strategy of RM is:
OT T
RM
(29)
where
R7 (a) Doc (a)
if (a') + (aa)
RL,RM =
(30)
otherwise,
Compared with Formula 10, in the GWFP process, it equivalent to as 1/t, M, 0, and be, ORRM- Therefore, in two-player zero-sum game, only need to prove
lim eT = 0, T T-000
(31)
where
CT u' (o7'), o,7") u² u' oT')
(32)
to show that the RM algorithm is GWFP process.
First, recall that in the FP process, the essence to find BR strategy to the historical strategy.
1
σ+ 1+1
o. t+1 b(o.).
(33)
In two-player zero-sum sum normal-from game, this means performing a matrix multiplication. Let the pay-off matrix of the game be E A'|x|4"), Then, to find b(ō,), we need to calculate
6² (o,) arg maxU
(34)
dEAA
Here, we start from the perspective of player 1, and the calculation method for player 2 is symmetric. Specifically:
ba (0 arg maxUo? arg max acA 0
aEAt
(35)
arg max f aEAl
Uo..
k=1
Define (a) w' o' and Q+ - lt then
6'(0,t) = arg max
(36)
aEdt
Define Q1 - +r. and the value obtained is:
u' ('((TT').0T)) =('((((7'))
(37)
the value obtained by the RM strategy is:
u uo T.RM )T
Σ oTRM (a) (a)
(38)
aeAA"
When Ra') = 0, it means that all (a) are equal, and naturally:
n' (b(oT")) ,, o..
w u'(o (oT,RM) ,.. = 0.
(39)
When ( * O, in the RM process, define V = Σt= u² (ore). The regret of each action can be rewritten as:
(a) (a) -V
(40)
17

---

Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Formula 32 can be rewritten
4 = (b((TT'') - w (TT,
= (T(('TT)))
Σ
(a)
oEA Rij (a)s0
Σ
@T)M(())
-((((1)))(((((()))
(41)
aE and f. (a)>0
Σ
(.) (R) R2
oEAt and (0)20
Since ?a)( (a) > O,
0 < (a) S Ri(o, ')).
(42)
in two-player zero-sum game, limy m (a) = 0. We have:
Σ
(a)) (T) ( -() = 0,
(43)
aEAt
(a)>u
holds for all i E N. Therefore, RM satisfies all the conditions of GWFP, and RM is GWFP process. Q.E.D.
18


Reviews:
[{"review_id": "aPA3BCdtWC", "reviewer": "Reviewer_FBND", "summary": "This paper proposes a method for finding approximate equilibrium strategies in extensive-form games and claims to achieve strategies that balance “style” and “diversity” with playing strength. The authors present theoretical ideas in the context of two-player, zero-sum (2P0S) games (e.g., Kuhn poker), and also provide preliminary experiments in multiplayer Texas Hold’em. Their overall goal is to develop agents that have low exploitability while also satisfying certain preferences about how the agent plays.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We first answer your questions and comments in order (due to word limit, the original text is not quoted), and then we summarize some of the Experiments questions and answer them. If there are still questions or new questions arise, please feel free to discuss further.\n\n---\n\n## Questions\n\n1. Indeed, this is a mistake and we will fix it.\n2. We need to re-write the descriptions of $β$ and $δ$. In this section, I accidentally mistyped diversity and style. Diversity refers to the scale of the acceptable strategy space. Style refers to how close a trained strategy is to another preferred strategy.   $δ$ incentivises a style and $β$ enforces it (at least in poker). Without $β$ (i.e. with a $β$ of 0), if there is only one Nash equilibrium, $δ$ is useless: Pref-CFR will find the original NE.\n3. Setting only $δ$ indeed means setting $β$ to 0. $ε$ is a property inherent in the strategy, while $β$ is an artificially set parameter. These are two concepts so we use two symbols. We should say $ε \\leq β$, and we will fix this problem.\n4. \"ju et al.\" should be fixed.\n5. We will clarify that FP converges in 2P0S.\n6. See (7).\n7. The goal of this paper is to find a strategy that can take into account people's needs for \"style\" and \"diversity\" better than the equilibrium strategies (the NE strategy in two-player zero-sum games and the coarse correlated equilibrium (CCE) strategy in multi-player games). This divides our research into two parts:\n    * How to define the strategies of \"style\" and \"diversity\"?\n    * How to find strategies with different styles and diversities during the iteration?\nTherefore, the related work of this article is introduced in two parts. Lines 75 to 99 introduce how previous articles solved the equilibrium (finding the NE in 2p0s games and finding the CCE in multi-player games). The content after line 100 introduces what other studies, besides the traditional NE and CCE, have been conducted by predecessors to attempt to find \"better\" strategies.\nThe writing of the related work in this paper does not present the above content well, and we will appropriately modify the wording.\n8. “Strategy profile” is indeed more precise.\n9. In general cases, we should indeed replace “exploitability” with NashConv/NE. Moreover, Equation 2 measures \"Player i’s deviation incentive\" rather than exploitability.\n10. See (2).\n11.  A set of Nash equilibria does not always form a convex polyhedron. For instance, Rock-Paper-Scissors has a unique NE at $[1/3,1/3,1/3]$ (a single point).\n12. Deltas cannot all be 1, We will emphasize this in the main text. And, for instance, if degrees are [1, 1, 2, 3] we choose actions with probability: [0, 0, ⅓, ⅔].\n13. We indeed meant “alternatively”.\n14. To clarify, we reiterate our key proofs:\n    * Appendix B.2 demonstrates that introducing preference parameter $δ$ in CFR preserves Blackwell approachability.\n    * Appendix B.3 proves that the vulnerability parameter $β$ modifies the convergence target set of no-regret algorithms.\n\nImplications:\n\n    - Based on B.2, our algorithm inherits the convergence guarantees of no-regret learning (equivalence to Blackwell approachability), converging to NE in 2p0s games and CCE in multiplayer settings.\n    - B.3 establishes that convergence to the target cone $S_{≤β}$ corresponds to the epsilon-equilibrium class.\n\nWe acknowledge the insufficient discussion of these connections in the main text and will:\n\n    - Add explicit signposts in Section 4\n    - Include a new subsection in Appendix B to detail the proof roadmap.\n\n15. This is a mistake. Thanks for spotting it.\n16. This metric aligns with both Brown et al.’s results and our experimental findings.\n\n\n---\n## More Experiments\n\nFor more experiments on smaller games, we have integrated Pref-CFR into OpenSpiel, and calculated the exploitability of 2p Texas Hold'em. These experiments show the good scalability and robustness of our algorithm, and we have redrawn the picture. The results of the experiment can be viewed at: https://drive.google.com/file/d/1F-oNqLm_cVqBNHvDYUf5dtQ_Rk1SdMkH/view?usp=drive_link\n\nIn addition, we will add more details to Texas Hold'em training in later versions. We cannot show it to you here due to character limitations. You can check our new experiments in the response of the reviewer CeZM and BR1q.\n\n---\n\nWe deeply appreciate your rigorous review. All revisions will be prominently marked in the final manuscript.", "claims_and_evidence": "- The paper claims that their method produces strategies which play according to the desired \"style\", while not increasing exploitability by too much. They claim this produces interesting, low-exploitability strategies in poker, whereas existing AI strategies for poker all play in the same style.\n\nThe experimental results shown in Figures 1 and 2 (Kuhn poker) are convincing in demonstrating the method’s effectiveness in a toy domain. They convincingly show that their method can result in strategies that behave very differently, and without the cost of increasing exploitability by much.  \nFor the Texas Hold’em experiments, the evidence that the trained agents are low-exploitability is not as strong. The authors’ discussion under Table 2 attempts to convince the reader of the strength of the agents, but they are unconvincing to me. More direct exploitability evaluation on a smaller domain where exploitability can be calculated (e.g., Leduc poker, Turn/River hold'em) would strengthen the claims.", "methods_and_evaluation_criteria": "The proposed method is a modification of CFR, parameterized by two parameters. The proposed evaluation criteria are Kuhn poker, 2-player Texas hold'em, and 3-player Texas hold'em.\n\nThe paper needs to be much more clear about which parts apply to the 2P0S (two-player zero sum) setting only, and which claims pertain to the general setting. Almost nowhere in the paper do the authors specify that they specifically care about 2P0S, yet much of the discussion makes sense only in the 2P0S setting (e.g. characterization of Nash equilibria in Introduction as \"maximizing expected payoffs under worst-case scenarios\", and the rest of the paragraph and following paragraphs) -- e.g. the CFR-based methods surely only have convergence guarantees in 2P0S settings, yet experiments are also run on a 3-player domain.\n\nAs mentioned above, there is not convincing evidence that the Texas Hold'em experiments produced low-exploitability agents. This is very difficult to do in a domain like Texas hold'em, so experiments in a more tractable domain would improve the paper.", "theoretical_claims": "I did not review the proofs of the method for correctness.", "experimental_designs_or_analyses": "See: Methods and Evaluation Criteria.", "relation_to_broader_scientific_literature": "The paper’s motivation draws heavily from the poker community, especially research on finding low-exploitability strategies using CFR variants. This is a well-established area with strong ties to multi-agent reinforcement learning and computational game theory. The paper aims to bridge preference-based methods (style/diversity) with standard CFR-like approaches.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "In all, the paper is well-motivated, the research direction is especially interesting to the poker community. I did not review the proofs of the methods for convergence, but the results in Kuhn poker look very good, and the qualitative Texas hold'em results are interesting. Nevertheless, I'm hesitant to recommend this version of the paper for this conference -- the writing still needs a lot of work. Additionally, the Texas hold'em experiments don't contain enough detail.", "other_comments_or_suggestions": "- Figures 1 and 2 are excellent results. They are convincing evidence of the effectiveness of the paper's method (albeit in a toy domain), and also clearly communicates the aims of the research direction. The charts could use some cleanup though -- all the text/labels in the charts are too small and hard to read.", "questions_for_authors": "- In Introduction, \"Nevertheless, strategies with winning probability as the sole objective\" should probably say something like \"EV\" or \"maximizing expected payoffs\" instead of \"winning probability\", to be precise (for domains like chess with loss/draw/win and poker with integer-valued payoffs).\n- Bottom of page 1/top of page 2: The descriptions of \\beta and \\delta sound a little silly to me: \"style\" corresponds to the degree of exploitability of the strategy, and \"diversity\" corresponds to closeness to a particular type of play? Neither of those sound intuitive.\n- \"setting only \\delta\" is not the most understandable -- this means setting \\beta to 0, right? Why not just say that?\n\\epsilon = \\beta? Why do we need the notation for both then? Or, should this say \\epsilon \\leq \\beta?\n- Related Work (line 94/95): \"ju et al.\" should be capitalized?\n- Should clarify that FP converges in 2P0S.\n- I don't find the paragraph starting with line 100 to be super relevant -- too broad. Besides, CEs are NEs in 2P0S.\n- \"This paper will synthesize the research from these two aspects\" -- what two aspects?\n- Section 2.1.2 Extensive-Form Games: Last sentence \"when all players select actions according to the strategy\" should be \"when all players select actions according to the strategy profile\", to be precise.\n- Section 2.1.3. Nash Equilibrium:\nIf we're not going to narrow the focus to 2P0S, then let's not use the terms \"exploitability\". Instead, call it NashConv or Nash distance (and you can state that this is called \"exploitability\" in 2P0S games). Regardless, I don't think we should call Equation 2 the \"exploitability of player i\", since it's about i deviating, not -i.\n- Section 3.1.: Again, we have description of \\beta and \\delta that do NOT match the descriptions given in the Introduction. Now, diversity is the size of an acceptable strategy space, and style is the similarity between a strategy distribution and a preferred strategy. However, later in the section, style is defined as the distance between a single strategy (not distribution) and a preferred strategy.\n- Section 3.2.: Conjecture 3.1: \"In a 2P0S game, if the set of Nash equilibria forms a convex polyhedron\" -- isn't this always true?\n- Section 4.1: Equation 13: why does the \"otherwise\" case have a \"-1\" in the numerator and the denominator? E.g. if the deltas are all 1, then we get 0/0, which is undefined.\n- After Equation 13, maybe \"additionally\" should say \"alternatively\"?\n- \"We prove in the Appendix B.2 that the Pref-CFR algorithm can converge to the NE\" -- shouldn't this clarify that this is only in 2P0S? In fact, even in the appendix, I don't see anywhere that clarifies that this is only in 2P0S.\n- Equation 16 notation needs work: argmax over a^i, over a function of a, not a^i? There should be an indicator function with equality to a, right?\n- Experiments:\n\"In Heads-Up play, this setup achieved exploitability below 4 mBB/h\" -- is this referring to Brown et al., 2018, or to your reproduced results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NAqTcseacg", "reviewer": "Reviewer_BR1q", "summary": "This paper introduces **Preference Counterfactual Regret Minimization (Pref-CFR)**, an extension of **Counterfactual Regret Minimization (CFR)**, designed to incorporate **strategy diversity** and **playstyle customization** in game AI. While standard CFR focuses on computing **Nash Equilibrium (NE)**, the authors argue that optimality alone is insufficient for practical applications requiring diverse strategies.\n\n### **Main Contributions** ###\n\nPref-CFR introduces **Preference and Vulnerability Degrees** to adjust strategy tendencies, enabling AI to exhibit different playstyles while maintaining competitiveness.\n\n### **Experimental Results** ###\n\nApplied to **Texas Hold’em**, Pref-CFR successfully learns **Aggressive** and **Loose Passive** playstyles, performing competitively with standard CFR-trained strategies while demonstrating distinct stylistic differences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. The strengths of the paper were perfectly understood, so we will only address weaknesses and questions below.\n\n---\n## Comparisons to state-of-the-art AI\nOur algorithm is not aimed at defeating top-level AI, so we didn't conduct this experiment at the start. However, competing against top AI can also prove the effectiveness of our algorithm, so we added this part of the experiment.\n\nFor this paper, the algorithm and the strategy profile it produces needed to satisfy the following constraints:\n1. Style customization, and possibly its usefulness\n2. Beta-exploitability\n3. Ability to work on a large-scale game\n\nChallenging a state-of-the-art AI is a third-party verification that ensures the correctness of the strategy profile produced and its lack of exploitability (2). It proves (3) even more, since it would use the full game (without bet or cards abstraction).\n\nHowever, it is not a direct way to evaluate the exploitability of the algorithm, which we did later in Head’s Up Texas Hold’em Poker. As expected, exploitability was under Beta. To be precise, **passive solutions were very close to the normal version of CFR (6 mBB/h vs 4 mBB/h), while aggressive versions were very close to Beta (23 mBB/h vs 25 mBB/h).** We believe that this difference is explained by the average size of the pot: in terms of %pot exploitability, results are closer.\n\nNow, to get back to the state-of-the-art AI point, **the blueprint (no real-time solving) strategy used in the paper defeated Slumbot 2019 after 100,000 hands, by 34.2 mBB/h $\\pm$ 12 mBB/h**, which is a performance similar to the one achieved by Baby Tartanian8 against Slumbot 2017. Moreover, while blueprints are more exploitable in later streets due to post-flop cards’ abstraction, preflop is barely exploitable, and we only showed preflop results.\n\nFinally, you mentioned “the paper does not benchmark Pref-CFR against state-of-the-art models like DeepStack and Pluribus, making it unclear whether it offers meaningful advantages”. Pref-CFR should offer no advantage against top AIs, its goal is not to play optimally but to offer a trade-off between customization and performance, and to allow humans to understand how much wiggle room they have when deviation from Game Theory Optimal (GTO) solutions.\n\n---\n\n## Is Pref-CFR applicable to a broader range of imperfect information games?\n\n**In fact, the scope of application of our algorithm is consistent with CFR, which is proved in our Appendix B.2.** To avoid misunderstanding, we will emphasize this result in the main text. In addition,\n\nIn addition, **We have integrated Pref-CFR into OpenSpiel**, a game-theoretic learning framework that supports over 100+ games. We successfully conducted two additional experiments in this environment. Due to character limitations, this part of the experiment can be found in the reply of reviewer CeZM. However, large-scale games require more specialized frameworks, which are often closed source, so we will limit ourselves to Texas Hold’em for now.\n\n---\n## Have the strategies generated by Pref-CFR been tested by human players?\n\n\n\n\n\nHiring professional players for evaluation is extremely challenging. For example, Pluribus paid more than 50,000 dollars for professional players to appear in the experiment. The focus of our paper is to propose a method to find different styles of strategies in the game. It is too early to play against professional players. We conducted some tests among amateur players to prove that our algorithm is effective, and we will add this part in subsequent versions.\n\n---\n\n## What is the practical value of having different playstyles in the generated strategies？\n\nIn Texas Hold'em, we analyze the practicality of our algorithm from two perspectives: **novel experience** and **winning more money**. \nIf it is to give players a novel experience, our current strategy is already very recognizable. In fact, our strategy can already become a qualified companion AI. We have verified the content of this part in the games played by amateur players.\n\nHowever, if you want to win more money with this strategy, there are two difficulties.\n1. The strategies are hardly memorizable: there is a mixed strategy for every combination of private cards, requires players to practice for a long time.\n2. If a player plays a particular style in order to win money, it is often to exploit a particular opponent’s weakness, while we do it without direct purpose. \n\nFor the above two problems, we can use preference degree and beta to model the opponent's strategy profile, and then find the best response that is less likely to be exploited. Here, Pref-CFR will be used for modeling. It requires sufficient domain knowledge to model such a role, as well as an algorithm that produces safe exploits and meets player-friendly characteristics. We will continue to improve our algorithm in subsequent research.", "claims_and_evidence": "- #### **Strengths:** ####\n  \n  - **Pref-CFR retains the convergence properties of CFR** and can still reach **Nash Equilibrium (NE)** under certain conditions, as theoretically proven in **Appendix B.2**.\n  - The algorithm extends to **ϵ-NE**, demonstrating mathematical soundness even with some loss tolerance.\n  - Pref-CFR introduces **novel strategy discoveries**, such as **raising with weaker hands**, which may offer insights for professional players.\n  \n- #### **Weaknesses:** ####\n  \n  - While theoretically applicable to other **extensive-form games**, there is **no empirical validation** beyond **Texas Hold’em**, limiting its demonstrated generalizability.\n  - The **lack of human player testing** makes it uncertain whether Pref-CFR’s strategies provide meaningful improvements for real-world play.", "methods_and_evaluation_criteria": "- #### **Strengths:** ####\n   \n    - Pref-CFR effectively **adjusts strategic tendencies** through **preference degree and vulnerability degree**, enabling AI to learn diverse playstyles while maintaining competitiveness.\n    - The **evaluation criteria are reasonable**, focusing on **strategy diversity** and **competitiveness**, which align with common practices in game AI research.\n    - AI playstyle variation is **demonstrated through parameter adjustments**, providing an important metric for evaluating strategic flexibility.\n    \n- #### **Weaknesses:** ####\n    \n    - **Limited validation**: The method has only been tested in **Texas Hold’em**, while **CFR applies to a broader range of imperfect information games**. Additional experiments in **other domains (e.g., StarCraft, economic games)** are needed to establish generalizability.\n    - **Lack of human player feedback**: Pref-CFR aims to improve strategies for human players, but **no real-world testing with professional players** has been conducted.\n    - **No comparison with state-of-the-art game AIs**: The study does not evaluate **whether more advanced AI models can achieve similar playstyle control**, leaving Pref-CFR’s comparative advantage uncertain.", "theoretical_claims": "The core theoretical proofs in the paper are correct, and the mathematical derivations contain no obvious errors.", "experimental_designs_or_analyses": "- #### **Strengths:** ####\n    \n    - The experimental design effectively validates Pref-CFR’s ability to train AI with different playstyles in Texas Hold’em, demonstrating both Aggressive and Loose Passive strategies.\n    \n    - Pref-CFR’s competitiveness is assessed by comparing it with standard CFR-trained AI, showing that distinct playstyles can be achieved without significant performance loss.\n    \n    - The algorithm discovers new strategies , suggesting that Pref-CFR may offer novel insights beyond conventional CFR-based approaches.\n    \n- #### **Weaknesses:** ####\n    \n    - Limited scope: Experiments are restricted to Texas Hold’em, with no validation in other imperfect information games , limiting generalizability.\n    \n    - Lack of human player evaluation: The study does not test whether Pref-CFR-generated strategies are beneficial for real-world human players, which is crucial given its focus on strategic diversity.\n    \n    - No comparison with advanced game AIs: The paper does not benchmark Pref-CFR against state-of-the-art models like DeepStack and Pluribus, making it unclear whether it offers meaningful advantages.\n    \n    - Future research should expand experimental settings and incorporate human feedback to further validate Pref-CFR’s real-world applicability.", "relation_to_broader_scientific_literature": "This study builds on **CFR**, which has been widely used in imperfect information games. However, traditional CFR focuses only on **optimal strategy** and lacks control over **strategy diversity**. This paper introduces **preference degree and **vulnerability degree, allowing AI to **adjust strategies across different playstyles**, filling the gap in **personalized strategy learning** and providing a new direction for research on **adaptive strategies in game AI**.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "- #### **Weaknesses:** ####\n\n - **Limited experimental scope**: The experiments are currently restricted to **Texas Hold’em**, lacking validation in other imperfect information games (e.g., **StarCraft, economic games, bridge**).\n\n - **No comparison with more advanced game AIs**: The advantages of Pref-CFR in strategy style control and competitiveness have not been fully evaluated against state-of-the-art AI models.\n\n - **Lack of human player evaluation**: If Pref-CFR aims to generate strategies that align better with human playstyles, professional players should be involved in testing, and their feedback should be collected to assess its usability and impact on human players.\n\n- **Limited practical significance**: While the generated strategies show diverse playstyles, the **real-world impact** of such stylistic variations remains unclear. What is the **practical value** of these playstyles, and how do they **contribute to improving gameplay** in real-world settings?", "other_comments_or_suggestions": "- **Clarify Scope of Applicability**: The paper primarily conducts experiments in **Texas Hold’em**, but its applicability to other **imperfect information games** remains unclear. Discussing how the method could generalize to **multi-agent strategy games, economic simulations, and other domains** would strengthen the paper's contributions.\n\n - **Consider Human Player Evaluation**: Since the method aims to adjust strategy styles, incorporating **feedback from professional players** would help assess its practical effectiveness in real-world gameplay.\n\n - **Improve Formula Notation and Explanation**: Some **formula symbols and derivations** could be further clarified, particularly in **Appendix B**, to enhance readability.\n\n - **Figure 3 Strategy Visualization Improvements**:\n  (1) The **current strategy display for Texas Hold’em** is informative but lacks clarity and aesthetic appeal.\n  (2) Consider a **more intuitive representation**, such as a cleaner color scheme or a different layout to enhance interpretability", "questions_for_authors": "- **Is Pref-CFR applicable to a broader range of imperfect information games?**\n The current experiments are limited to **Texas Hold’em**, but **CFR** is applicable to a wider range of games.\n\n- **Have the strategies generated by Pref-CFR been tested by human players?**\n Is there a plan to have **professional players** test the strategies generated by Pref-CFR and collect feedback to verify their feasibility in real-world gameplay?\n\n- **What is the practical value of having different playstyles in the generated strategies?**\n Given that the generated playstyles (e.g., Aggressive and Loose Passive) are largely theoretical, how do these stylistic variations **translate into real-world advantages**, particularly for human players? Does this diversity actually improve gameplay, or is it primarily a theoretical construct?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA."}}, {"review_id": "7H21Bp8zCY", "reviewer": "Reviewer_CeZM", "summary": "This paper tackles a key limitation of standard CFR-based methods. Typically, these methods converge to a single Nash equilibrium (NE) and struggle to accommodate risk preferences or different playing styles. To address this, the authors propose Preference-CFR (Pref-CFR), which introduces two key parameters at each decision point: a preference degree, $\\delta(I,a)$, that amplifies or suppresses specific actions, and a vulnerability degree, $\\beta(I)$, which controls how much exploitability is allowed, guiding the solution toward a $\\epsilon$-NE instead of a strict NE.\n\nUsing Blackwell approachability, the authors show that Pref-CFR maintains low regret, ensuring strong near-NE performance. In Kuhn poker experiments, the method steers strategy toward different equilibria by adjusting $\\delta$, whereas standard CFR always converges to the same equilibrium. In larger-scale tests on two and three player Texas Hold'em, Pref-CFR effectively shapes playstyles, such as increasing aggression or reducing folds, resulting in distinctive, human-like strategies like \"aggressive\" or \"loose-passive\". These adjustments maintain high performance with only a slight trade-off in exploitability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We will first answer your questions and comments in order, and then summarize and answer some questions from previous sessions. If there are still questions or new questions arise, please feel free to discuss further.\n\n---\n\n## Questions 1\n\nOur algorithm adopts a similar idea to Pluribus, that is, only the pre-flop strategy is saved, and the subsequent strategy is obtained by real-time search, so we only show the per-flop results.\n\n---\n\n## Questions 2\n\nConducting such an experiment directly in large-scale problems is extremely costly. Therefore, we carried out more experiments in Leduc. The experimental charts can be found at: https://drive.google.com/file/d/1F-oNqLm_cVqBNHvDYUf5dtQ_Rk1SdMkH/view?usp=drive_link\n\nNo unexpected instability was found in the results, which is in line with theoretical expectations, which also proves the robustness of our algorithm.\n\nIn Figure 1, we doubled the value of $\\delta$ (from 5 to 10). It can be seen that the improvement in the stylistic characteristics of the final result is negligible. Thus, we have every reason to believe that a larger $\\delta$ is unnecessary.\n\nIn Figure 2, we set different values of $\\beta$. It can be observed that the influence of different $\\beta$ values on the results is fully consistent with the theoretical expectations:\n1. The $\\epsilon$ of the final strategy will not exceed the set $\\beta$;\n2. A larger $\\beta$ can lead to a more pronounced stylistic strategy.\n\nTips: In this game, there are only two actions, namely \"raise\" and \"call\", and $\\sigma(\\text{raise}) + \\sigma(\\text{call}) = 1$.  \n\n---\n\n## More extensive preference experimentation\n\nThis seems unnecessary because a strategy cannot have two styles simultaneously. For example, at the beginning, we set $\\delta$ as an aggressive strategy (with a relatively large $\\delta(raise)$), and then change it to a conservative strategy in the middle of the training process (reducing the value of $\\delta(raise)$), which seems meaningless. However, this suggestion has inspired us. When training strategies of different styles, we don't have to start the training from a completely random strategy. Instead, we can first train a \"Normal\" strategy, and then use this \"Normal\" strategy as a warm start. Subsequently, strategies of different styles can continue to be trained based on this \"Normal\" strategy. We will try this approach in our subsequent work.  \n\n---\n\n## Discussing the strengths and limitations of other CFR-based methods\n\nIndeed, we lack the comparison with previous methods, and there is also a lack of technical introductions of key concepts and terms in the introduction stage. We will add this part of content in the subsequent versions.", "claims_and_evidence": "Most of the paper's claims are reasonably supported, but there are places where the evidence could be more thorough. The authors assert that Pref-CFR can not only converge to different equilibria but also preserve near-optimal performance. They demonstrate examples of altered strategies in Kuhn poker and Texas Hold'em, along with head-to-head matches measuring exploitability. If we look at the claim that increasing the preference parameters $\\delta$ or the vulnerability parameters $\\beta$ \"does not significantly slow down convergence,\" the supporting data come primarily from small-scale Kuhn experiments and selected subgame analyses in Texas Hold'em. These examples, while informative, do not fully guarantee scalability or robustness in larger or more complex scenarios, especially when very large $\\delta$ or $\\beta$ values are applied across numerous information sets.\n\nWhile the experiments demonstrate a few distinct playstyles, such as \"aggressive\" and \"loose-passive,\" they don't rigorously test whether the method can handle a large number of style parameters simultaneously or adapt to more nuanced, context-specific preferences. It would have been useful to see broader experiments systematically exploring a wider range of preference settings, such as analyzing how performance changes when $\\delta$ is adjusted incrementally across all betting rounds. Without these deeper evaluations, it's fair to question whether the method truly scales to diverse playstyles as seamlessly as the paper suggests. The claims aren't entirely unsupported, but more extensive and systematic testing would have made the evidence far more compelling.", "methods_and_evaluation_criteria": "The choice of poker, Kuhn and Texas Hold'em, as a testing ground does make sense, given that poker has a long history in computational game theory. The proposed approach, Pref-CFR, specifically targets scenarios where one needs flexible or style-driven equilibria rather than a single exploitability-minimizing solution, and poker is a natural fit for demonstrating that kind of variety.\n\n* While Kuhn poker is a classic toy example and works nicely to illustrate theoretical proofs, it's extremely small in scale and doesn't necessarily expose all the complications that arise in real-world poker settings. The paper then jumps to two and three player Texas Hold'em, which is certainly more complex, but the presentation of final strategies (e.g., aggregated preflop charts) and small-sample matchups may not fully convey whether the style modifications maintain robustness across the entire decision tree. More rigorous exploitability analyses, or thorough cross-play among multiple different stylized agents, might have better established how well the approach generalizes.\n\n* While the evaluation criteria of measuring head-to-head performance in terms of mBB/h is standard in poker, there's some risk in concluding broad \"style\" success based primarily on changes to preflop charts. One could argue that style adjustments should manifest consistently across multiple betting rounds, and the current paper only provides a snapshot of the final strategy. So, even if the overall benchmarks make sense, the completeness of the evaluation might be limited when it comes to verifying that the agent truly embodies a different style (as opposed to just shifting its action frequencies slightly at the earliest decision points).\n\nIn summary, the methods and evaluation criteria make sense given the application, but they could be more thorough in probing deeper behaviors and broader ranges of scenarios.", "theoretical_claims": "The proofs in the paper appear consistent with known results on no-regret learning and Blackwell approachability. The authors' main technical claim is that, by introducing preference parameters $\\delta(I,a)$ and vulnerability $\\beta(I)$ in the update steps, they can still guarantee convergence to a (possibly $\\beta$-relaxed) Nash equilibrium. Their proofs hinge on two well-known building blocks: (1) standard regret-minimization arguments, and (2) Blackwell approachability ensuring that the average regret vector converges.", "experimental_designs_or_analyses": "The experiments in the paper are generally appropriate for illustrating the potential of the proposed method, yet there are a few points:\n\n* The authors test on Kuhn poker (a very small game) and on subgame approximations of Texas Hold'em (two and three player). This does show that their algorithm can scale beyond toy scenarios, but the evaluation is limited in how comprehensively it inspects all stages of a large game. By focusing on subgame methods, we do not necessarily see full-tree exploitability or style changes across every betting round.\n* The experiments tend to vary only a handful of preference degrees $(\\delta)$ or vulnerability degrees $(\\beta)$ in fairly simple ways (e.g., drastically increasing raise frequency or almost never folding). It is plausible that moderate or more nuanced adjustments could produce different equilibrium behaviors. This keeps the experiments intuitive, yet it leaves open questions about how the method behaves when many parameters are tweaked in more subtle ways.\n\nOverall, the experimental designs serve their main purpose, showing that Pref-CFR can shift an equilibrium strategy's style with minimal performance loss.", "relation_to_broader_scientific_literature": "see next section", "essential_references_not_discussed": "Since this paper is grounded in CFR models, it would be beneficial to include a more comprehensive discussion of related works within this domain. While the current related work section primarily provides background information on existing approaches, it lacks a direct comparison between these methods and the proposed approach. A stronger comparative analysis would help contextualize the contributions of this paper by highlighting how it improves upon prior work. Specifically, discussing the strengths and limitations of other CFR-based methods and demonstrating where your approach offers advantages would make the contribution clearer. This would not only strengthen the justification for your method but also provide a more complete picture of the research landscape", "other_strengths_and_weaknesses": "Strengths:\n* Novel Extension to CFR: The introduction of preference and vulnerability parameters is a straightforward yet effective modification of the CFR framework, allowing for different equilibria based on user-defined stylistic or risk constraints.\n* Practical Demonstration: Experiments on both Kuhn poker and Texas Hold'em games convincingly show how adjusting the parameters leads to distinct, human-like playing styles (aggressive, loose-passive) with minimal performance degradation.    \n\nWeaknesses:\n* While the experiments are illustrative, they focus primarily on a few stylized adjustments and measure changes mostly at preflop or initial decision points. Postflop actions in full-scale Texas Hold'em and extended multi-street analysis are not exhaustively demonstrated.\n* The related work section primarily provides background information but lacks a direct comparison with similar CFR-based methods. A more comprehensive discussion of related works and a stronger comparative analysis would better highlight the contributions of this paper. Addressing the strengths and limitations of existing approaches while emphasizing the advantages of the proposed method would clarify its impact and strengthen its justification.", "other_comments_or_suggestions": "* The introduction section would benefit from a more technical explanation of key concepts and terminology to better introduce the research area to the reader.\n* The technical background in Section 2 is well-presented and effectively enhances the clarity of the paper.", "questions_for_authors": "1. Your experiments mostly highlight changes in preflop or early-round decision-making. Did you conduct any deeper analyses on later betting rounds to confirm that style-related shifts remain consistent throughout the entire game tree? If so, do you observe similarly clear style distinctions in postflop play?\n2. While it is shown that extreme values can push the final strategy toward distinct equilibria, have you observed any surprising instability when $\\delta$ or $\\beta$ become very large (for example, extremely aggressive or extremely passive settings)? Does the algorithm remain stable, or do you see any oscillations or convergence slowdowns in such scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fTn4PhoQ4X", "reviewer": "Reviewer_XPTK", "summary": "The paper introduces Preference-CFR (Pref-CFR), an extension of Counterfactual Regret Minimization (CFR) aimed at generating diverse strategies in extensive-form games. Traditional CFR-based approaches focus on solving for Nash Equilibria (NE), which prioritize optimality under worst-case conditions but lack strategic diversity. Pref-CFR introduces preference degree and vulnerability degree to control playstyle and allowable exploitability, respectively. The method enables the training of AI agents with distinct styles, such as Aggressive and Loose Passive strategies, in Texas Hold’em. Experimental results show that Pref-CFR achieves comparable performance to standard CFR-trained strategies while demonstrating diverse playstyles.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We summarize some of the questions and answer them. If there are still questions or new questions arise, please feel free to discuss further.\n\n---\n\n## How sensitive are the results to the heuristic tuning of $δ$ and $β$, particularly in large-scale games?\n\nConducting experiments in large-scale games is highly costly, and comprehensive analysis of these two parameters would require significant time and hardware resources. However, we have successfully migrated the PrefCFR method to OpenSpiel, a game learning framework developed by DeepMind. By employing a larger game instance of Leduc Poker, we can conduct detailed analysis of these parameters' effects. The results can be viewed at: https://drive.google.com/file/d/1F-oNqLm_cVqBNHvDYUf5dtQ_Rk1SdMkH/view?usp=drive_link\n\nExperiments show that the introduction of the preference parameter alone has little effect on the convergence of the game. Even if it is set to 10, the convergence speed is only reduced but not significantly. When fragility is introduced, the convergence speed in the early stage of training is basically the same as that of the original CFR algorithm, and only slows down when it approaches $β$.\n\n---\n\n## Heuristic tuning of $δ$ and $β$ requires manual selection.\n\nRegarding manual selection of $δ$ and $β$: Manual parameter selection is currently required, but in the context of Texas Hold'em poker - where player styles have distinct categorizations and sufficient expert knowledge exists - this manual selection does not create a bottleneck for the current algorithm implementation. We plan to further investigate this aspect in future research.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I didn't check the proof of theory section in the appendix in detail.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Traditional CFR and its extensions (CFR+, DCFR, MCCFR).\nCorrelated Equilibria and other refinements beyond NE.\nOpponent-exploiting AI (e.g., safe equilibrium models), which could serve as an additional baseline.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n- Novel conceptual contribution: Introducing preference-based parameters into CFR is an interesting extension.\n- Empirical validation: The method performs well in Texas Hold’em, an established benchmark.\n- Theoretical soundness: The algorithm has theoretical proof on convergence.\n\nWeaknesses\n- Heuristic tuning of $\\delta$ and $\\beta$ requires manual selection. The sensitivity of parameter selection in large-scale game is not discussed.", "other_comments_or_suggestions": "See strengths and weaknesses", "questions_for_authors": "How sensitive are the results to the heuristic tuning of  $\\delta$ and $\\beta$ , particularly in large-scale games?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that the proposed approach is interesting and important at least to a reasonably large community in AI—the poker community. Reviewers found the theoretical results sound and the experimental results supportive of the conclusion. On the other side, reviewers raised on concerns about the clarity and the generality of the proposed framework beyond the three types of games tested in the paper. Some concerns were addressed during the rebuttal and all reviewers agreed that the paper is acceptable.
Decision: Accept (poster)