Do LLM Agents Have Regret? A Case Study in Online Learning and Games

## Abstract

Abstract Large language models (LLMs) have been increasingly employed for (interactive) decision-making, via the development of LLM-based autonomous agents. Despite their emerging successes, the performance of LLM agents in decision-making has not been fully investigated through quantitative metrics, especially in the multi-agent setting when they interact with each other, a typical scenario in real-world LLM-agent applications. To better understand the limits of
LLM agents in these interactive environments, we propose to study their interactions in benchmark decision-making settings in online learning and game theory, through the
performance metric of regret .
We first empirically study the no-regret behaviors of LLMs in canonical (non-stationary) online learning problems, as well as the emergence of equilibria when LLM agents interact through playing repeated games.
We then provide some theoretical insights into the no-regret behaviors of LLM agents, under certain assumptions on the supervised pre-training and the rationality model of human decision-makers who generate the data. Notably, we also identify (simple) cases where advanced LLMs such as GPT-4 fail to be no-regret. To promote the no-regret behaviors, we propose a novel unsupervised training loss of regret-loss , which, in contrast to the supervised pre-training loss, does not require the labels of (optimal) actions. We then establish the statistical guarantee of generalization bound for regret-loss minimization, followed by the optimization guarantee that
minimizing such a loss may automatically lead to known no-regret learning algorithms.
Our further experiments demonstrate the effectiveness of our regret-loss, especially in addressing the above “regrettable” cases.

## 1 Introduction

> Live Life with No Excuses. Travel with No Regret.

Large language models (LLMs) have recently exhibited remarkable reasoning capabilities (Bubeck et al., ; Achiam et al., ; Wei et al., ; Yao et al., ). As a consequence, a burgeoning body of work has been investigating the employment of LLMs as central controllers for (interactive) decision-making, through the construction of *LLM-based autonomous agents* (Hao et al., ; Shen et al., ; Yao et al., ; Shinn et al., ; Wang et al., ; Significant Gravitas, ). Specifically, the LLM agent interacts with the (physical) world in a *dynamic/sequential* way: it
uses LLMs as an oracle for reasoning, then acts in the environment based on the reasoning and the feedback it perceives over time. LLM agent has achieved impressive successes in embodied AI (Ahn et al., ; Huang et al., ; Wang et al., ), natural science (Wu et al., ; Swan et al., ), and social science (Park et al., , ) applications.

Besides being *dynamic*, another increasingly captivating feature of LLM-based decision-making is the involvement of *strategic* interactions, oftentimes among multiple LLM agents. For example, it has been continually reported that the reasoning capability of LLMs can be improved by interacting with each other through negotiation and/or debate games (Fu et al., ; Du et al., ); LLM agents have now been widely used to *simulate* the strategic behaviors for social and economic studies, to understand the emerging behaviors in interactive social systems (Aher et al., ; Park et al., ).
Moreover, LLMs have also exhibited remarkable potential in solving various games (Bakhtin et al., ; Mukobi et al., ), and in fact, a rapidly expanding literature has employed *repeated games* as a fundamental benchmark to understand the strategic behaviors of LLMs (Brookins and DeBacker, ; Akata et al., ; Fan et al., ). These exciting empirical successes call for a rigorous examination and understanding through a theoretical lens of decision-making.

*Regret*, on the other hand, has been a core metric in (online) decision-making. It measures how “sorry” the decision-maker is, in retrospect, not to have followed the best prediction in hindsight
(Shalev-Shwartz, ). It provides not only a sensible way to *evaluate* the intelligence level of online decision-makers, but also a quantitative way to measure their *robustness* against arbitrary (and possibly adversarial) environments. More importantly, it inherently offers a connection to modeling and analyzing strategic behaviors: the long-run interaction of no-regret learners leads to certain *equilibria* when they repeatedly play games (Cesa-Bianchi and Lugosi, ). In fact, *no-regret* learning has served as a natural model for predicting and explaining human behaviors in strategic decision-making, with experimental evidence (Erev and Roth, ; Nekipelov et al., ; Balseiro and Gur, ). It has thus been posited as an important model of “rational behavior” in playing games (Blum et al., ; Roughgarden, ; Roughgarden et al., ).
Thus, it is natural to ask:

*Can we
examine and better understand the online and strategic decision-making
behaviors
of LLMs through the lens of *regret*?*

Acknowledging that LLM(-agents) are extremely complicated to analyze,
to gain some insights into the question,
we focus on benchmark decision-making settings: online learning with convex (linear) loss functions, and playing repeated games.
We summarize our contributions as follows.

##### Contributions.

First, we carefully examine the performance of several representative pre-trained LLMs in aforementioned benchmark online decision-making settings,
in terms of *regret*. We observe that oftentimes, LLM agents exhibit no-regret behaviors in these (non-stationary) online learning settings, where the loss functions change over time either arbitrarily (and even adversarially), or by following some patterns with bounded variation, and
in playing both representative and randomly generated repeated games. For the latter, equilibria will emerge as the long-term behavior of the interactions when all LLM agents are no-regret. Second, we provide some theoretical insights into the observed no-regret behaviors, based on some hypothetical model of
the
human decision-makers who generate the data, and certain assumptions on the *supervised pre-training* procedure, a common practice in training large models for decision-making.
In particular, we make a connection of pre-trained LLMs to the known no-regret algorithm of *follow-the-perturbed-leader* (FTPL) under such assumptions. Third, we also identify (simple) cases where advanced LLMs as GPT-4 fail to be
no-regret. We thus propose a novel *unsupervised* training loss, *regret-loss*, which, in contrast to the supervised
pre-training loss, does not require the *labels* of
(optimal) actions.
We then establish both statistical and optimization guarantees
for regret-loss minimization, showing that
minimizing such a loss may automatically lead to known no-regret learning algorithms.
Our further experiments demonstrate the effectiveness of regret-loss, especially in addressing the above “regrettable” cases.

### 1.1 Related Work

##### LLM(-agent) for decision-making.

The impressive capability of LLMs for *reasoning* (Bubeck et al., ; Achiam et al., ; Wei et al., , ; Srivastava et al., ; Yao et al., ) has inspired a growing line of research on *LLM for (interactive) decision-making*, i.e., an LLM-based autonomous agent interacts with the environment by taking actions repeatedly/sequentially, based on the feedback it perceives. Some promises have been shown from a *planning* perspective (Hao et al., ; Valmeekam et al., ; Huang et al., ; Shen et al., ). In particular, for embodied AI applications, e.g., robotics, LLMs have achieved impressive performance when used as the controller for decision-making (Ahn et al., ; Yao et al., ; Shinn et al., ; Wang et al., ; Driess et al., ; Significant Gravitas, ). However, the performance of decision-making has not been rigorously characterized via the regret metric in these works. Very recently, Liu et al. () has proposed a principled architecture for LLM-agent, with provable regret guarantees in stationary and stochastic decision-making environments, under the Bayesian adaptive Markov decision processes framework. In contrast, our work focuses on online learning and game-theoretic settings, in potentially
adversarial and non-stationary environments.
Moreover, (first part of) our work focuses on *evaluating* the intelligence level of LLM per se in decision-making (in terms of the regret metric), while Liu et al. () focused on *developing* a new architecture that uses LLM as an oracle for reasoning, together with memory and specific planning/acting subroutines, *to achieve* sublinear (Bayesian) regret, in stationary and stochastic environments.

##### LLMs in multi-agent environments.

The interaction of multiple LLM agents has garnered significant attention lately.
For example, Fu et al. () showed that LLMs can autonomously
improve each other in a negotiation game by playing and criticizing each other. Similarly, (Du et al., ; Liang et al., ; Xiong et al., ; Chan et al., ; Li et al., ) showed that multi-LLM *debate* can improve the reasoning and evaluation capabilities of the LLMs. Qian et al. (); Schick et al. (); Wu et al. () demonstrated the potential of multi-LLM interactions and collaboration in software development, writing, and problem-solving, respectively. Zhang et al. () exhibited a similar potential in embodied cooperative environments. More formally, multi-LLM interactions have also been investigated under a *game-theoretic* framework, to characterize the *strategic* decision-making of LLM agents. Bakhtin et al. (); Mukobi et al. () and Xu et al. (, ) have demonstrated the promise of LLMs in playing Diplomacy and WereWolf games, respectively, which are both language-based games with a mixture of competitive and cooperative agents.
Note that these works utilized LLM to solve a specific rather than a general game. Related to our work, Brookins and DeBacker (); Akata et al. (); Lorè and Heydari (); Brookins and DeBacker (); Fan et al. () have also used (repeated) matrix games as a benchmark to evaluate the reasoning capability and rationality of LLM agents.
In contrast to our work, these empirical studies have not formally investigated LLM agents using the metric of *regret*, nor through the lenses of *online learning* and *equilibrium-computation*, which are all fundamental in modeling and analyzing strategic multi-agent interactions. Moreover, our work also provides theoretical results to explain and further enhance the no-regret property of LLM agents.

##### LLMs & Human/Social behavior.

LLMs have also been used to *simulate* the behavior of human beings, for social science and economics studies (Engel et al., ). The extent of LLMs simulating human behavior has been claimed as a way to evaluate the level of its intelligence in a controlled environment (Aher et al., ; Tsai et al., ). For example, Li et al. (); Hong et al. (); Zhao et al. () showed that by specifying different “roles” to LLM agents, certain collaborative/competitive behaviors can emerge. Argyle et al. () showed that LLMs can emulate response distributions from diverse human subgroups, illustrating their adaptability. Horton () argued that an LLM, as a computational model of humans, can be used as *homo economicus* when given endowments, information,
preferences, etc., to gain new economic insights by simulating its interaction with other LLMs. Park et al. (, ) proposed scalable simulators that can generate realistic social behaviors emerging in populated and interactive social systems, and the emerging behaviors of LLM agents in society have also been consistently observed in Chen et al. (, ). Li et al. (, ) studied the opinion/behavioral dynamics of LLM agents on social networks.
These empirical results have inspired our work, which can be viewed as an initial attempt towards quantitatively understanding the *emerging behavior* of LLMs as computational human models, given the well-known justification of *equilibrium* being a long-run emerging behavior of *learning dynamics* (Fudenberg and Levine, ) and strategic interactions (Young, ; Camerer, ).

##### Transformers & In-context-learning.

LLMs nowadays are predominantly built upon the architecture of Transformers (Vaswani et al., ). Transformers have exhibited a remarkable capacity of *in-context-learning* (ICL), which can construct new predictors from sequences of labeled examples as input, without further parameter updates. This has enabled the *few-shot learning* capability of Transformers (Brown et al., ; Garg et al., ; Min et al., ). The empirical successes have inspired burgeoning theoretical studies on ICL. Xie et al. () used a Bayesian inference framework to explain how ICL works, which has also been adopted in Wang et al. (); Jiang (). Akyürek et al. (); Von Oswald et al. (); Dai et al. (); Giannou et al. () showed (among other results) that ICL comes from the fact that Transformers can implement the gradient descent (GD) algorithm. Bai et al. () further established that Transformers can
implement a broad class of machine learning algorithms in context.
Moreover, Ahn et al. (); Zhang et al. (); Mahankali et al. () proved that a *minimizer* of the certain training loss
among single-layer Transformers is equivalent to a single step of GD for linear regression.
Li et al. () established generalization bounds of ICL from a multi-task learning perspective. Zhang et al. () argued that ICL implicitly implements Bayesian model averaging, and can be approximated by the attention mechanism. They also established a result on some *regret* metric. However, the regret notion is not defined for (online) decision-making, and is fundamentally different from ours that is standard in online learning and games. Also, we provide extensive experiments to validate the no-regret behavior by our definition. More recently, the ICL property has also been generalized to decision-making settings. Laskin et al. (); Lee et al. (); Lin et al. () investigated the in-context reinforcement learning (RL) property of Transformers under supervised pre-training, for solving
stochastic bandits and Markov decision processes.
In contrast, our work focuses on online learning settings with an arbitrary and *potentially adversarial* nature, as well as *game-theoretic* settings. We also provide a new *unsupervised* loss to promote the no-regret behavior in our settings.

##### Online learning and games.

Online learning has been extensively studied to model the decision-making of an agent who interacts with the environment sequentially, with a potentially arbitrary sequence of loss functions (Shalev-Shwartz, ; Hazan, ), and has a deep connection to game theory (Cesa-Bianchi and Lugosi, ). In particular, regret, the difference between the incurred accumulated loss and the best-in-hindsight accumulated loss, has been the core performance metric, and a good online learning algorithm should have regret at most sublinear in time $T$ (i.e., of order $o(T)$), which is referred to as being *no-regret*.
Many well-known algorithms can achieve no-regret against *arbitrary* loss sequences, e.g., multiplicative weight updates (MWU)/Hedge (Freund and Schapire, ; Arora et al., ), EXP3 (Auer et al., ),
and more generally follow-the-regularized-leader (FTRL) (Shalev-Shwartz and Singer, ) and follow-the-perturbed-leader (FTPL) (Kalai and Vempala, ). In the bandit
literature (Lattimore and Szepesvári, ; Bubeck et al., ), such a setting without any statistical assumptions on the losses is also referred to as the *adversarial/non-stochastic* setting.
Following the conventions in this literature, the online settings we focus on
shall not be confused with the stationary and *stochastic*(-bandit)/(-reinforcement learning) settings that have been explored in several other recent works on *Transformers for decision-making* (Lee et al., ; Lin et al., ). Centering around the regret metric, our work has also explored the
non-stationary bandit setting (Besbes et al., ), as well as the repeated game setting where the environment itself consists of strategic agents (Cesa-Bianchi and Lugosi, ).

## 2 Preliminaries

##### Notation.

We use $\mathbb{N}$ and $\mathbb{N}^{+}$
to denote the sets of non-negative and positive integers, respectively.
For a finite set ${\mathcal{S}}$, we use $\Delta({\mathcal{S}})$ to denote the simplex over ${\mathcal{S}}$. For $d\in\mathbb{N}^{+}$, we define $[d]:=\{1,2,\dots,d\}$. For two vectors $x,y\in\mathbb{R}^{d}$, we use $\langle x,y\rangle$ to denote the inner product of $x$ and $y$. We define $\boldsymbol{0}_{d}$ and $\boldsymbol{1}_{d}$ as a $d$-dimensional zero or one vector, and $\boldsymbol{O}_{d\times d}$ and $I_{d\times d}$ as a $d\times d$-dimensional zero matrix and identity matrix, respectively. We omit $d$ when it is clear from the context. We define $e_{i}$ as a unit vector (with proper dimension) whose $i$-th coordinate equal to $1$. For $p\in\mathbb{R}^{d},R>0$ and $C\subseteq\mathbb{R}^{d}$ is a convex set, define $B(p,R,\|\cdot\|):=\{x\in\mathbb{R}^{d}\mid\|x-p\|\leq R\}$, $\texttt{Proj}_{C,\|\cdot\|}(p)=\operatorname*{arg\,min}_{x\in C}\|x-p\|$ (which is well defined as $C$ is a convex set), and $\texttt{clip}_{R}(x):=[\texttt{Proj}_{B(0,R,\|\cdot\|_{2}),\|\cdot\|_{2}}(x_{i})]_{i\in[d]}$. Define $\texttt{Softmax}(x):=\left(\frac{e^{x_{i}}}{\sum_{i\in[d]}e^{x_{i}}}\right)_{i\in[d]}$ and $\texttt{ReLU}(x)=\max(0,x)$ for $x\in\mathbb{R}^{d}$. For $A\in\mathbb{R}^{m\times n}$ with $A_{i}$ denoting its $i$-th column, we define $\|A\|_{\texttt{op}}:=\max_{\|x\|_{2}\leq 1}\|Ax\|_{2}$, $\|A\|_{2,\infty}:=\sup_{i\in[n]}\|A_{i}\|_{2}$,
$\|A\|_{F}$
as the Frobenius norm, and $A_{-1}:=A_{n}$ to denote the last column vector of $A$. We define $\mathbb{R}^{+}:=\{x\mid x\geq 0\}$. For a set $\Pi$, define $\text{diam}(\Pi,\|\cdot\|):=\sup_{\pi_{1},\pi_{2}\in\Pi}\|\pi_{1}-\pi_{2}\|$. We define $\mathbbm{1}(\mathcal{E}):=1$ if $\mathcal{E}$ is true, and $\mathbbm{1}(\mathcal{E}):=0$ otherwise. For a random variable sequence $(X_{n})_{n\in\mathbb{N}}$ and random variables $X,Y$, we denote $F_{X}$ as the cumulative distribution function of a random variable $X$, $X_{n}\overset{p}{\to}X$ if $\forall\epsilon>0,\lim_{n\to\infty}\mathbb{P}(|X_{n}-X|>\epsilon)=0$, $X_{n}\overset{d}{\to}X$ if $\lim_{n\to\infty}F_{X_{n}}(x)=F_{X}(x)$ for all $x$ where $F_{X}(x)$ is continuous, $X\overset{d}{=}Y$ if $F_{X}(x)=F_{Y}(x)$ for all $x$, $X_{n}\overset{a.s.}{\to}X$ if $\mathbb{P}(\lim_{n\to\infty}X_{n}=X)=1$, and $\operatorname{esssup}(X):=\inf\{M\in\mathbb{R}:\mathbb{P}(X>M)=0\}$. For a random variable $X$, we use $\text{supp}(X)$ to denote its support. For functions $f,g:\mathbb{R}\to\mathbb{R}$, we define $g(x)=\mathcal{O}(f(x))$ if there exist $x_{0},M<\infty$ such that $|g(x)|\leq M|f(x)|$ for all $x>x_{0}$. We use $f^{\prime}$ to denote the derivative of $f$. Let $F:\Omega\rightarrow\mathbb{R}$ be a continuously-differentiable, strictly convex function defined on a convex set $\Omega$. The Bregman divergence associated with $F$ for points $p$, $q$ is defined as $D_{F}(p,q):=F(p)-F(q)-\langle\nabla F(q),p-q\rangle$. For a sequence $(\ell_{t})_{t\in[T]}$ for some $T\in\mathbb{N}^{+}$, we define $\ell_{a:b}:=(\ell_{a},\cdots,\ell_{b})$ for $1\leq a\leq b\leq T$. If $a>b$, we define $\ell_{a:b}=\emptyset$.

### 2.1 Online Learning & Games

##### Online learning.

We first consider the online learning setting where an agent interacts with the environment for $T$ rounds, by iteratively making decisions based on the feedback she receives. Specifically, at each time step $t$, the agent chooses her decision policy $\pi_{t}\in\Pi$ for some bounded domain $\Pi$, and after her commitment to $\pi_{t}$, a bounded
loss function $f_{t}:\Pi\to{[-B,B]}$ for some constant $B>0$ is chosen by the environment, potentially in an adversarial fashion. The agent thus incurs a loss of $f_{t}(\pi_{t})$, and will update her decision to $\pi_{t+1}$ using the feedback. We focus on the most basic setting where the agent chooses actions from a finite set $\mathcal{A}$ every round, which is also referred to as the *Experts Problem* (Littlestone and Warmuth, ; Hazan, ), without loss of much generality (c.f. [Section A.4](#A1.SS4) for a discussion). In this case, $\Pi$ becomes the simplex over $\mathcal{A}$, i.e., $\Pi=\Delta(\mathcal{A})$, and $f_{t}(\pi_{t})=\langle{\ell}_{t},\pi_{t}\rangle$ for some loss *vector* ${\ell}_{t}\in\mathbb{R}^{d}$ that may change over time, where $d:=|\mathcal{A}|$.

At time step $t\in[T]$, the agent may receive either the full vector ${\ell}_{t}$, or only the realized loss ${\ell}_{ta_{t}}$ (we sometimes also interchangeably write it as $\ell_{t}(a_{t})$), the $a_{t}$th element of $\ell_{t}$, for some $a_{t}\sim\pi_{t}(\cdot)$, as feedback, which will be referred to as online learning with *full-information feedback*, and that with *bandit feedback*, respectively. The latter is also referred to as the *adversarial/non-stochastic bandit* problem in the multi-armed bandit (MAB) literature. Note that hereafter, we will by default refer to this setting that does *not* make any assumptions on the loss sequence $({\ell}_{t})_{t\in[T]}$ simply as *online learning*.
Moreover, if the loss functions change over time (usually with certain bounded variation), we will refer to it as
*non-stationary online learning* for short, whose bandit-feedback version is also referred to as the *non-stationary bandit* problem.

##### Repeated games.

The online learning setting above has an intimate connection to game theory. Consider a normal-form game $\mathcal{G}=\langle N,\{\mathcal{A}_{n}\}_{n\in[N]},\{r_{n}\}_{n\in[N]}\rangle$, where $N$ is the number of players, $\mathcal{A}_{n}$ and $r_{n}:\mathcal{A}_{1}\times\dots\times\mathcal{A}_{N}\rightarrow{[-B,B]}$ are the action set and the payoff function of player $n$, respectively. The $N$ players repeatedly play the game for $T$ rounds, each player $n$ maintains a strategy $\pi_{n,t}\in\Delta(\mathcal{A}_{n})$ at time $t$, and takes action $a_{n,t}\sim\pi_{n,t}(\cdot)$. The *joint* action $a_{t}=(a_{1,t},\cdots,a_{N,t})$ determines the payoff of each player at time $t$, $\{r_{n}(a_{t})\}_{n\in[N]}$. From a single-player’s (e.g., player $n$’s) perspective, she encounters an online learning problem with (expected) loss function ${\ell}_{t}:=-\mathbb{E}_{a_{-n,t}\sim\pi_{-n,t}}[r_{n}(\cdot,a_{-n,t})]$ at time $t$, where $-n$ denotes the index for all the players other than player $n$.
We will refer to it as the *game setting* for short,
and use the terms of “agent” and “player” interchangeably hereafter. The key difference between online learning and repeated games is in their interaction dynamics: online learning involves an agent facing a potentially adversarial, changing environment (or sequence of loss functions), while in repeated games, agents interact by playing the same game repeatedly, which might be less adversarial when they follow specific learning algorithms.

### 2.2 Performance Metric: Regret

We now introduce *regret*, the core performance metric used in online learning and games.
For a given algorithm $\mathscr{A}$,
let $\pi_{\mathscr{A},t}$ denote the decision policy of the agent at time $t$ generated by $\mathscr{A}$. Then, the regret, which is the difference between the accumulated (expected) loss incurred by implementing $\mathscr{A}$ and that incurred by
the best-in-hindsight fixed decision, can be defined as

$$ $\displaystyle\text{Regret}_{\mathscr{A}}\left((f_{t})_{t\in[T]}\right):=\sum_{t=1}^{T}f_{t}(\pi_{\mathscr{A},t})-\inf_{\pi\in\Pi}\sum_{t=1}^{T}f_{t}(\pi).$ $$

In the Experts Problem, the definition can be instantiated as $\text{Regret}_{\mathscr{A}}(({\ell}_{t})_{t\in[T]}):=\sum_{t=1}^{T}\langle{\ell}_{t},\pi_{\mathscr{A},t}\rangle-\inf_{\pi\in\Pi}\sum_{t=1}^{T}\langle{\ell}_{t},\pi\rangle$. With bandit-feedback, a commonly used metric may also take further expectation for $\text{Regret}_{\mathscr{A}}$, over the randomness of the generated policies $(\pi_{\mathscr{A},t})_{t\in[T]}$.
An algorithm $\mathscr{A}$ is referred to as being *no-regret*, if $\text{Regret}_{\mathscr{A}}((f_{t})_{t\in[T]})\sim o(T)$, i.e., the regret grows sublinearly in $T$. Widely-known no-regret algorithms include follow-the-regularized-leader (FTRL) (Shalev-Shwartz and Singer, ), follow-the-perturbed-leader (Kalai and Vempala, ) (See [Section A.3](#A1.SS3) for a detailed introduction).

In non-stationary online learning, one also uses the metric of *dynamic regret* (Zinkevich, ), where the *comparator* in the definition also changes over time, as the best decision policy at each individual time $t$: $\text{D-Regret}_{\mathscr{A}}((f_{t})_{t\in[T]}):=\sum_{t=1}^{T}f_{t}(\pi_{\mathscr{A},t})-\sum_{t=1}^{T}\inf_{\pi\in\Pi}f_{t}(\pi)$,
which is a stronger notion than $\text{Regret}_{\mathscr{A}}((f_{t})_{t\in[T]})$ in that $\text{Regret}_{\mathscr{A}}((f_{t})_{t\in[T]})\leq\text{D-Regret}_{\mathscr{A}}((f_{t})_{t\in[T]})$.

## 3 Do Pre-Trained LLMs Have Regret? Experimental Validation

In this section, we explore the no-regret behaviors of representative pre-trained LLMs (i.e., GPT-4 Turbo, GPT-4, and GPT-3.5 Turbo),
in the context of online learning and games.
All experiments with LLMs
are conducted using the public OpenAI Python API (Openai, ).

##### Intuition why pre-trained language models may exhibit no-regret behavior.

Transformer-based LLMs have demonstrated impressive *in-context-learning* and few-/zero-shot learning capabilities (Brown et al., ; Garg et al., ; Min et al., ). One theoretical explanation is that, trained Transformers can implement the *gradient descent algorithm* on the testing loss in certain supervised learning problems (Akyürek et al., ; Von Oswald et al., ; Dai et al., ; Ahn et al., ; Zhang et al., ; Mahankali et al., ), which is inherently *adaptive* to the loss function used at test time. On the other hand, it is known in online learning that the simple algorithm of *online gradient descent* (Zinkevich, ) can achieve no-regret. Hence, it seems reasonable to envision the no-regret behavior of such meta-learners in online learning, due to their fast adaptability. However, it is not straightforward due to the fundamental difference between multi-task/meta-learning and online learning settings, as well as the difference between
*stationary* and *non-stationary/adversarial* environments in decision-making. Next, we provide both experimental and theoretical studies
to validate this intuition.

##### Interaction protocol.

To enable the sequential interactions with LLMs, we first describe the setup and objective of our experimental study. At each round, we incorporate the entire history of loss vectors of past interactions into our prompts, as concatenated texts, and ask the LLM agent to determine a policy that guides the decision-making for the next round. Note that since we hope to *evaluate* the intelligence level of pre-trained LLMs through online learning or games, we only provide simple prompts that she should utilize the history information, without providing explicit rules of *how* to make use of the history information, nor asking her to *minimize regret* (in any sense). A detailed description and an ablation study of the prompts are deferred to [Section B.1](#A2.SS1), and an illustration of the protocol in playing repeated games is given in [Figure 3](#S3.F3).

### 3.1 Framework for No-Regret Behavior Validation

Before delving into the results, we note that to the best of our knowledge, we are not aware of any principled framework for validating no-regret behaviors with finite-time experimental data. Therefore, we propose two frameworks to rigorously validate the no-regret behavior of algorithms over a *finite* $T$, which might be of independent interest.

##### Trend-checking framework.

We propose the following hypothesis test:

$$ $\displaystyle H_{0}$ $\displaystyle:\text{The sequence }\left(\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t\right)_{t=1}^{\infty}\text{ either diverges or converges to a positive constant}$ $\displaystyle H_{1}$ $\displaystyle:\text{The sequence }\left(\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t\right)_{t=1}^{\infty}\text{ converges to 0}$ $$

with $H_{0}$ and $H_{1}$ denoting the null and alternative hypotheses, respectively.
The notion of convergence is related to $T\to\infty$ by definition, making it challenging to verify directly. As an alternative, we propose a more tractable hypothesis test, albeit a weaker one, that still captures the essence of our objective:

$$ $\displaystyle H_{0}$ $\displaystyle:\text{The sequence }\left(\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t\right)_{t\in[T]}\text{ does not exhibit a decreasing trend}$ $\displaystyle H_{1}$ $\displaystyle:\text{The sequence }\left(\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t\right)_{t\in[T]}\text{ shows a decreasing trend}.$ $$

Ideally, one should check if $\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t$ approaches zero as $t$ goes to infinity. With a finite $T$ value, testing these hypotheses provides a method to quantify this – whether we reject $H_{0}$ offers a way to measure it.
To this end, one needs to count the number of $\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t-\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t+1]}\right)/(t+1)>0$, for which we use [Proposition 1](#Thmproposition1) below (whose proof is deferred to [Section B.2](#A2.SS2)) to provide some understanding of (how small) the probability it happens under various counts. For example, with the default choice of $T=25$ in our experiments later, one can see from [Proposition 1](#Thmproposition1) that: $\mathbb{P}_{H_{0}}(\mathcal{E}(17,25))<0.032,\mathbb{P}_{H_{0}}(\mathcal{E}(19,25))<0.0035,\mathbb{P}_{H_{0}}(\mathcal{E}(21,25))<0.00014$, i.e., one can easily reject $H_{0}$ with high probability. We will report the $p$-value of $H_{0}$, denoted as $p_{trend}$, as the output of this framework.

###### Proposition 1 .

*($p$-value of the null hypothesis).* Define the event

$$ $\displaystyle\mathcal{E}(s,T):=\left\{\text{The~{}number~{}of~{}}\frac{\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)}{t}-\frac{\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t+1]}\right)}{t+1}>0\text{~{}~{}for~{}~{}}t=1,\dots,T\text{~{}is~{}at~{}least~{}~{}}s\geq\frac{T-1}{2}\right\}.$ $$

Under the assumption that the null hypothesis $H_{0}$ holds, the probability of this event happening is bounded as $\mathbb{P}_{H_{0}}(\mathcal{E}(s,T))\leq\frac{1}{2^{T-1}}\sum_{t=s}^{T-1}\begin{pmatrix}T-1\\
t\end{pmatrix}$.

###### Proposition 1 .

##### Regression-based framework.

In complement to the statistical framework above, we propose an alternative approach by
fitting the data with regression.
In particular, one can use the data

$$ $\displaystyle\left\{\left(t,\log\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)\right)\right\}_{t\in[T]}$ $$

to fit a linear function $g(t)=\beta_{0}\log t+\beta_{1}$, where the estimate of $\beta_{0}$, i.e., $\widehat{\beta}_{0}$, satisfying $\widehat{\beta}_{0}<1$
may be used to indicate the no-regret behavior, i.e., the *sublinear* growth of $\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)$ over time.
While being simple, it cannot be directly used when $\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)<0$, so we set $\log\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)$ as $-10$ if this happens. We define $p_{reg}$ as the $p$-value of the regression parameter $\widehat{\beta}_{0}$, and will report the pair of $(\widehat{\beta}_{0},p_{reg})$ as the output of this framework.

### 3.2 Results: Online Learning

We now present the experimental results on the no-regret behavior of pre-trained LLMs in online learning in: 1) arbitrarily changing environments, 2) non-stationary environments, and 3) bandit-feedback environments.

##### Online learning in arbitrarily changing environment.

We first consider the setting with arbitrarily changing environments, which are instantiated as follows:
1) *Randomly-generated loss sequences*. At every timestep, we generate a random loss vector ${\ell}_{t}\sim\operatorname{Unif}([0,10]^{d})$ or ${\ell}_{t}\sim\mathcal{N}(5\boldsymbol{1}_{d},I)$
with clipping to $[0,10]$
to ensure the boundedness, such that the loss vectors of different timesteps can be arbitrarily different;
2) *Loss sequences with a predictable trend.* Although many real-world environments may change, they often change following certain patterns. Therefore, we consider two representative trends, a *linear* trend and a *periodic* (sinusoid) trend. For the linear trend, we sample $a,b\sim\text{Unif}([0,10]^{d})$ and let $\ell_{t}=(b-a)\frac{t}{T}+a$ for each $t\in[T]$. For the periodic trend, we sample $a,b\sim\text{Unif}([0,10]^{d})$ and let $\ell_{t}=5(1+\sin(at+b))$ for each $t\in[T]$. In the experiments, we choose $d=2$. The average regret (over multiple randomly generated instances) performance is presented in [Figure 1](#S3.F1),
where we compare GPT-4 with well-known no-regret algorithms, FTRL with entropy regularization and FTPL with Gaussian perturbations (with tuned parameters). It is seen that these pre-trained LLMs can indeed achieve no-regret and often have smaller regrets than these baselines.

##### Online learning (in non-stationary environment).

We then experiment on the setting where the losses are still changing over time, but their total variations across time are bounded, more concretely, sublinear in $T$. Correspondingly, we consider the stronger metric of *dynamic regret* here to measure the performance. Note that without constraining the variation of the loss vectors, dynamic regret can be linear w.r.t. $T$ in the worst case.
Hence, we generate the loss vectors in two different ways:
1) *Gradual variation.* We firstly sample ${\ell}_{1}\sim\text{Unif}([0,10]^{d})$. Then for each $t\geq 2$, we uniformly and randomly generate ${\ell}_{t+1}$ under the constraint $\|{\ell}_{t+1}-{\ell}_{t}\|_{\infty}\leq\frac{1}{\sqrt{t}}$, such that the variations over time are guaranteed to satisfy $\sum_{t=1}^{T-1}\|{\ell}_{t+1}-{\ell}_{t}\|_{\infty}={o(T)}$;
2) *Abrupt variation.* We randomly generate ${\ell}_{1}\sim\text{Unif}([0,10]^{d})$ and $m$ time indices $\{t_{i}\}_{i\in[m]}$ from $\{1,2,\cdots,T\}$. At each time step $t_{i}$ for $i\in[m]$, the sign of the loss vector ${\ell}_{t_{i}}$ is flipped, i.e., we let
${\ell}_{t_{i}}\leftarrow 10\boldsymbol{1}_{d}-{\ell}_{t_{i}}$.
For the specific choice of $T=25$ in our experiments, we choose $m=3$.
For both cases, the average dynamic regret results are presented in [Table 1](#S3.T1). It can be seen that GPT-4 achieves sublinear dynamic regret and outperforms *FTRL/FTPL with Restart*, a standard variant of FTRL/FTPL for non-stationary online learning (see e.g., Besbes et al. ()). We refer to [Section A.3](#A1.SS3.SSS0.Px5) for a more detailed introduction of FTRL/FTPL with Restart.

##### Extension to bandit-feedback settings.

Although pre-trained LLMs have achieved good performance in online learning with full-information feedback, it is unclear whether they can still maintain no-regret with only bandit feedback.
For such problems, we modify the prompt and protocol of interactions slightly, where we still ask the LLM agent to provide a policy $\pi_{t}$ at time step $t$, then sample one $a_{t}\sim\pi_{t}(\cdot)$. In the bandit setting, the LLM agent can only access $(a_{t},\ell_{ta_{t}})$. Instead of directly feeding it to the agent, we feed an estimate of the loss vector $\widehat{\ell}_{t}\in\mathbb{R}^{d}$, where $\widehat{\ell}_{t}(a)\leftarrow\frac{\ell_{t}(a)}{\pi_{t}(a)}\mathbbm{1}(a_{t}=a)$ for all $j\in[d]$. Note that such an operation of *re-weighting* the loss by the inverse of the probability is standard in online learning when adapting full-information-feedback no-regret algorithms to the bandit-feedback ones (Auer et al., ).
Later, we will also show the benefits of such operations (c.f. [Section 4](#S4)). We compare the performance of pre-trained LLMs with that of the counterparts of FTRL with bandit feedback, e.g., EXP3 (Auer et al., ) and the bandit-version of FTPL (Abernethy et al., ), in both [Figure 2](#S3.F2) and [Table 1](#S3.T1), where GPT-4 consistently achieves lower regret.

Figure: Figure 1: Regret of GPT-3.5 Turbo/GPT-4 for online learning with full-information feedback in $4$ different settings. It performs comparably and sometimes even better than well-known no-regret learning algorithms, FTRL and FTPL.
Refer to caption: /html/2403.16843/assets/x1.png

Figure: Figure 2: Regret of GPT-3.5 Turbo/GPT-4 for online learning with bandit feedback in $4$ different settings. It performs comparably and sometimes even better than well-known no-regret learning algorithms, variants of FTRL and FTPL with bandit-feedback.
Refer to caption: /html/2403.16843/assets/x2.png

Figure: Figure 3: Demonstration of the prompts and interaction protocol for multi-player repeated games. A human moderator does not provide the game’s payoff matrices to the LLMs. Instead, at each round, the human moderator provides each player’s own payoff vector history.
Refer to caption: /html/2403.16843/assets/x3.png

### 3.3 Results: Multi-Player Repeated Games

We now consider the setting when multiple LLMs make online strategic decisions in a *shared* environment repeatedly.
Specifically, at each round, the loss vectors each agent receives are determined by both her payoff matrix and the strategies of all other agents. Note that the payoff matrix is not directly revealed to the LLM agent, but she has to make decisions in a completely online fashion based on the payoff vector marginalized by the opponents’ strategies (See [Figure 3](#S3.F3) for a prompt example). This is a typical scenario in learning in (repeated) games (Fudenberg and Levine, ).

##### Representative games.

We first test the repeated play of pre-trained LLMs on $6$ representative general-sum games (*win-win, prisoner’s dilemma, unfair, cyclic, biased,* and *second best*) studied in Robinson and Goforth () (see a detailed introduction of these games in [Section A.5](#A1.SS5)). For each type of the game, we conduct $20$ repeated experiments.

##### Randomly generated games.

To further validate the no-regret behavior of LLMs, we also test on $50$ randomly generated three-player general-sum games, and $50$ randomly generated four-player general-sum games, where each entry of the payoff matrix is sampled randomly from $\operatorname{Unif}([0,10])$. These are larger and more challenging settings than the structured and representative ones above.

We summarize the experimental results in [Figure 4](#S3.F4), which
are similar to the above in the online setting: for all types of games, GPT-4 achieves sublinear regret, which is comparable with that obtained by FTRL for most games. We provide six instances of three-player general-sum games and six instances of four-player general-sum games in [Figure 5](#S3.F5) and [Figure 6](#S3.F6), respectively. Occasionally, GPT-4 even provides a negative regret value.

Figure: Figure 4: Regret of GPT-3.5 Turbo/GPT-4 for repeated games of $3$ different game sizes, where the sublinear regret is validated by both of our statistical frameworks. Due to the symmetry of agents in randomly generated games, we report the regret of one agent for ease of presentation.
Refer to caption: /html/2403.16843/assets/x4.png

Figure: Figure 5: Regret of GPT-4 and the FTRL algorithm in $6$ randomly generated three-player general-sum games. GPT-4 has comparable (even better) no-regret properties when compared with the FTRL algorithm.
Refer to caption: /html/2403.16843/assets/x7.png

Figure: Figure 6: Regret of GPT-4 and the FTRL algorithm in $6$ randomly generated four-player general-sum games. GPT-4 has comparable (even better) no-regret properties when compared with the FTRL algorithm, according to the frameworks in [Section 3.1](#S3.SS1) and the graphic trends.
Refer to caption: /html/2403.16843/assets/x8.png

Figure: Figure 7: (left) Regret of both GPT-4 and GPT-4 Turbo under the canonical counterexample for FTL (Hazan, , Chapter 5). (mid, right) Failure of GPT-4 and GPT-4 Turbo on two scenarios for regrettable behavior of GPT, while Transformers with regret-loss provide no-regret behaviors.
Refer to caption: /html/2403.16843/assets/x9.png

**Table 1: Dynamic regret of GPT-3.5 Turbo/GPT-4 in a non-stationary environment with either full-information or bandit feedback. Every experiment is conducted with 25 rounds. No-regret behaviors of GPT-3.5 Turbo/GPT-4 are validated by both of our frameworks (low $p$-values and $\widehat{\beta}_{0}<1$). The only exception is GPT-3.5 Turbo on loss sequence with abrupt variations under bandit feedback. This indicates that GPT-3.5 Turbo may not be capable of dealing with an abruptly changing environment with limited feedback, although the average regret achieved eventually is still lower than that of other baselines.**
| Dynamic regret | GPT-4 | GPT-3.5 Turbo | FTRL | FTPL |  |
| --- | --- | --- | --- | --- | --- |
| Full<br>information | Gradual variation | $12.61\pm 7.01$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.0,0.58,0.0)$ | $19.09\pm 11.33$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.0,0.83,0.0)$ | $36.58\pm 24.51$ | $35.19\pm 22.51$ |
| Abrupt variation | $30.0\pm 19.91$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.01,0.87,0.0)$ | $33.65\pm 22.51$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.08,0.96,0.0)$ | $36.52\pm 27.68$ | $36.24\pm 28.22$ |  |
| Bandit | Gradual variation | $21.39\pm 10.86$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.0,0.78,0.0)$ | $28.42\pm 21.6$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.0,0.83,0.0)$ | $37.64\pm 21.97$ | $36.37\pm 20.7$ |
| Abrupt variation | $35.94\pm 28.93$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.42,0.95,0.0)$ | $30.76\pm 25.48$<br>$(p_{trend},\widehat{\beta}_{0},p_{reg})=(0.92,1.01,0.0)$ | $36.52\pm 27.68$ | $38.82\pm 26.17$ |  |

### 3.4 Pre-Trained LLM Agents May Still Have Regret

It seems tempting to conclude that pre-trained LLMs can achieve no-regret in both online learning and playing repeated games. However, is this capability *universal*? We show that the no-regret property might break for LLM agents if the loss vectors are generated in a more adversarial way.

##### Canonical counterexample for follow-the-leader.

To begin with, we consider a well-known example that *follow-the-leader* (FTL) algorithm (Shalev-Shwartz, ) suffers from linear regret (Hazan, , Chapter 5), where $\ell_{1}(1)=5,\ell_{1}(2)=0$ and $\ell_{t}(2-t\%2)=10,\ell_{t}(1+t\%2)=0$ for $t\geq 2$, where $\%$ is the modulo operation. Interestingly, GPT-4 agent can easily identify the pattern for the loss sequence that the optimal action *alternates*, thus accurately predicting the loss it will receive and achieving nearly zero regret in [Figure 7](#S3.F7). In other words, GPT-4 agent seems to not fail in the same way as FTL, which is known to be due to the lack of randomness in prediction.

##### Less predictable loss sequence.

Inspired by the counterexample above, we design a new loss sequence that is *similar but less predictable*. Specifically,
we construct the following (simple) loss sequence with $2$ actions such that ${\ell}_{t}(1+t\%2)=\min(25/t,10),{\ell}_{t}(2-t\%2)\sim\text{Unif}([9,10])$ for $t\in[25]$.

##### Adaptive loss sequence.

We also develop a simpler but more *adaptive* loss sequence that takes the full power of the adversary in the online learning setup. After the GPT-4 agent provides $\pi_{t}$, we choose ${\ell}_{t}$ such that ${\ell}_{t}(\operatorname*{arg\,max}_{i}\pi_{ti})=10$ and ${\ell}_{t}(3-\operatorname*{arg\,max}_{i}\pi_{ti})=0$. We report the average regret over $20$ repeated experiments for the later two settings using GPT-4 and more advanced GPT-4 Turbo in [Figure 7](#S3.F7), where we cannot reject the hypothesis that GPT-4 (Turbo) has linear-regret by either our trend-checking or regression-based framework.

These observations have thus motivated us to design new approaches to better promote the no-regret property of LLM agents, with additional training, as to be detailed in [Section 5](#S5). Before delving into the design of such a *stronger* LLM agent, we first provide some theoretical insights into why pre-trained LLMs have already exhibited good no-regret behaviors oftentimes.

## 4 Why Are Pre-Trained LLMs (No-)Regret? A Hypothetical Model and Some Theoretical Insights

We now provide some plausible explanations about the no-regret behavior of pre-trained LLMs, as observed in [Sections 3.2](#S3.SS2) and [3.3](#S3.SS3). Note that our explanations have to be *hypothetical* by nature, since to the best of our knowledge, the details of pre-training these popular LLMs (e.g., GPT-3.5 Turbo and GPT-4), regarding data distribution, training algorithm, etc., have not been revealed. We instead make the explanations based on some common assumptions and arguments in the literature for modeling human behaviors, and the recent literature on understanding LLMs/Transformers.

### 4.1 Pre-Trained LLMs Have Similar Regret as Humans (Who Generate Data)

We first provide a direct observation based on some existing speculation on the capability of Transformer-based LLMs.
Recently, a growing literature has evidenced that the intelligence level
of LLM agents are determined by, and in fact mimic, those of human beings who generate the data for pre-training the models (Park et al., ; Argyle et al., ; Horton, ).
The key rationale was that, LLMs (with Transformer parameterization) can approximate the *pre-training data distribution* very well (Xie et al., ; Zhang et al., ; Lee et al., ).
In such a context, one can expect that LLM agents can achieve similar regret as human decision-makers who generate the pre-training data, as we formally state below.

###### Observation 1 .

An LLM agent is said to be pre-trained with an *$\epsilon$-decision error* if, for any arbitrary $t$ and loss sequences $({\ell}_{i})_{i\in[t]}$, the following condition holds:

$$ $\displaystyle\sup_{\pi\in\Pi}\quad\big{|}P_{\text{data}}(\pi{\,|\,}({\ell}_{i})_{i\in[t]})-P_{\text{LLM}}(\pi{\,|\,}({\ell}_{i})_{i\in[t]})\big{|}\leq\epsilon,$ $$

where $P_{\text{data}}$ and $P_{\text{LLM}}$ are the pre-training data distribution and the decision policy distribution of the pre-trained LLM, respectively.
Then, the regret of an LLM agent with $\epsilon$-decision error is bounded as:

$$ $\displaystyle\text{(D-)Regret}_{\text{LLM}}\left(({\ell}_{t})_{t\in[T]}\right)\in\left[\text{(D-)Regret}_{\text{data}}\left(({\ell}_{t})_{t\in[T]}\right)\pm\epsilon\|{\ell}_{t}\|\sup_{\pi\in\Pi}\|\pi\|\right],$ $$

where $[a\pm b]:=[a-b,a+b]$.

[1](#Thmobservation1) shows that the pre-trained LLM-agent’s regret can be controlled by that of the pre-training dataset and the decision error $\epsilon$. A small $\epsilon$ can be achieved if LLM is constructed by a rich function class, e.g., the Transformer architecture (Zhang et al., ; Lin et al., ). We defer a proof of the result to [Appendix C](#A3).

Note that the above observation does not provide the full picture – the $\text{(D-)Regret}_{\text{data}}\left(({\ell}_{t})_{t\in[T]}\right)$, which denotes the (dynamic-)regret of using $P_{\text{data}}\left(\pi{\,\big{|}\,}({\ell}_{i})_{i\in[t]}\right)$ for online learning, is not necessarily sublinear in $T$.
We discuss next under what (natural) models and pre-training processes,
the no-regret behavior of pre-trained LLMs may emerge.

###### Observation 1 .

### 4.2 A Human Decision-Making Model: Quantal Response

A seminal model for human decision-making behavior is the *quantal response* model, which assumes that humans are often *imperfect* decision-makers, and their bounded rationality can be modeled through unseen latent variables that influence the decision-making process (McFadden, ; McKelvey and Palfrey, ). Formally, the quantal response is defined as follows:

###### Definition 1 (Quantal response) .

Given a loss vector $\ell\in\mathbb{R}^{d}$, a noise distribution $\epsilon\sim P_{\text{noise}}$, and $\eta>0$, the quantal response is defined as

$$ $\displaystyle P_{\text{quantal}}^{\eta}\big{(}a{\,\big{|}\,}\ell\big{)}=\mathbb{P}\left(a\in\operatorname*{arg\,min}_{a^{\prime}\in\mathcal{A}}~{}~{}z(a^{\prime})\right),\qquad\text{where~{}~{} $z=\ell+\eta\epsilon$.}$ $$

In essence, this implies that humans are rational but with respect to (w.r.t.) the latent variable $z$, a perturbed version of $\ell$, instead of $\ell$ per se. This addition of noise to the actual loss vector characterizes the bounded rationality of humans in decision-making.

The traditional quantal response formulation primarily focused on scenarios with a single loss vector. In online decision-making, given the *history* information, the human at each time $t$ is faced with *multiple* loss vectors. Hence, we adopt the following generalization to model the decision-making behavior in this setting.

###### Definition 2 (Quantal response against multiple losses) .

Given a set of loss vectors $({\ell}_{i})_{i\in[t]}$, a noise distribution $P_{\text{noise}}$, and $\eta_{t}>0$, the generalized quantal response against $({\ell}_{i})_{i\in[t]}$ is defined as

$$ $P_{\text{quantal}}^{\eta_{t}}\left(a{\,\big{|}\,}({\ell}_{i})_{i\in[t]}\right):=P_{\text{quantal}}^{\eta_{t}}\left(a{\,\bigg{|}\,}\sum_{i=1}^{t}{\ell}_{i}\right).$ $$

For $t=0$, we define $P_{\text{quantal}}^{\eta_{t}}\left(a{\,\big{|}\,}({\ell}_{i})_{i\in[t]}\right):=P_{\text{quantal}}^{\eta_{0}}\left(a{\,\big{|}\,}\emptyset\right)=\frac{1}{d}$.

In simpler terms, the generalized quantal response
is defined as the standard quantal response against some scaled summation of the losses. Note that such a *dynamic* version of quantal response also has implications from behavior economics, and has been recently used to model human behaviors in sequential decision-making (Ding et al., ) (in stochastic and stationary environments). Indeed, such a response against multiple loss vectors is believed to be natural, and has also been widely adopted in well-known no-regret learning algorithms of *smooth/stochastic fictitious play* (Fudenberg and Kreps, ) and *follow-the-perturbed-leader* (Kalai and Vempala, ),
whose formal definitions can be found in [Section A.3](#A1.SS3). Finally, note that the response model in [Definition 2](#Thmdefinition2) does not necessarily involve a *sequential* decision-making process, i.e., the set of losses may not come from the history of an online learning process. With this (natural) human response model in hand, we hypothesize that powerful pre-trained LLMs also behave similarly as such a model. To illustrate how [Definition 2](#Thmdefinition2) can potentially capture also the behavior of LLMs instead of only humans, we have also provided examples of LLMs *explaining* how they output their policies in [Section B.4](#A2.SS4). It can be observed that the LLM agent tends to use the history of the reward vectors by looking at the “sum/average of the accumulated rewards”, and tends to introduce “randomization” in decision-making, in order to be “less predictable and exploitable”. These are known to be key to achieving no-regret behaviors in online learning and games (Cesa-Bianchi and Lugosi, ). Hence, we analyze a case where pre-training under certain canonical data assumptions can provably lead to the quantal response behaviors and further yield no-regret guarantees, to gain some insights into the empirical observations in [Sections 3.2](#S3.SS2) and [3.3](#S3.SS3).

###### Definition 1 (Quantal response) .

###### Definition 2 (Quantal response against multiple losses) .

### 4.3 Case Study: Pre-Training under Canonical Data Distribution

Pre-training of LLMs often involves the approach of *next-token prediction*. When applying LLMs to sequential decision-making, the model receives the context of the decision-making sequence and then generates the *action*. This process can be conceptualized as *predicting the optimal action* in the form of the next token prediction. For instance, Yao et al. (); Shinn et al. (); Liu et al. (, ) demonstrated how decision-making may be framed in this way.
Concretely, LLM receives the context as $(x_{1},x_{2},\cdots,x_{N_{1}})$ and output $(x_{N_{1}+1},\cdots,x_{N_{2}})$ for some $N_{1},N_{2}\in\mathbb{N}^{+}$ and $N_{1}<N_{2}$, where each $x_{i}\in\mathcal{V}$ represents one *natural language token* for $i\in[N_{2}]$, and $\mathcal{V}$ is the finite token set. Afterwards, the human user extracts the corresponding decision action $a$ from the text $(x_{N_{1}+1},\cdots,x_{N_{2}})$.

Meanwhile, in these cases, large models such as Transformers are often (pre-)trained for (sequential) decision-making under several *fixed/stationary* environments (Laskin et al., ; Lin et al., ; Lee et al., ; Reed et al., ), which may limit their ability to handle *arbitrary/non-stationary/adversarial* loss sequences in our online learning setup. Thus, it is natural to ask:
*Is it possible to have no-regret behaviors emerging as a consequence of this (optimal) action prediction, under only a fixed pre-training distribution of stationary environments?*

Here we analyze a standard pre-training objective on a token sequence distribution $x_{1:N_{t+1}}\sim P^{text}_{t}$ for given $t\in[T]$,
which is the expected log-likelihood maximization for next-token prediction:

$$ $\displaystyle\max_{\theta\in\Theta}\quad\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log\text{LLM}_{\theta}\left(x_{j}{\,|\,}x_{1:j-1}\right),$ (1) $$

where we define $\text{LLM}_{\theta}\left(x_{1}{\,|\,}x_{1:0}\right)=\text{LLM}_{\theta}\left(x_{1}\right)$, and $\Theta$ denotes the parameter space of the LLM.

For the pre-training data distribution, intuitively, we model it as follows: there exists a latent variable $z$, representing the loss of the underlying *static* decision-making problem.
The pre-training dataset, however, only contains *partial observations* $x_{1:N_{t}}$ (a natural language representation of ${\ell}_{1:t}$) of $z$ due to imperfect data collection. The presence of partial observations could be attributed to the fact that $z$ is only *privately* known by the data-generator (human), representing the intentions of a human being/data-generator. Hence, LLM will only be pre-trained with partial and noisy information about $z$. Meanwhile, we assume that the optimal action $x_{N_{t}+1:N_{t+1}}$ (a natural language representation of $a$) w.r.t. the underlying loss vector $z$ is available in the pre-training dataset as the *label*, which could come from user surveys, personal blogs, or explicit data annotation. We formalize such a scenario in the assumption below:

###### Assumption 1 .

Given $T\in\mathbb{N}^{+}$, $t\in[T]$, and $N_{t+1}\in\mathbb{N}^{+}$, there is a latent variable $z$, $\ell_{1:t}$, $N_{1},\cdots N_{t}\in[N_{t+1}]$, and $N_{0}=0$, such that
$\mathbb{P}(z,\ell_{1:t},x_{1:N_{t+1}})=\mathbb{P}(z,\ell_{1:t})\mathbb{P}(x_{1:N_{t}}{\,|\,}\ell_{1:t})\mathbb{P}(x_{N_{t}+1:N_{t+1}}{\,|\,}z),$
and

$$ $\displaystyle P^{text}_{t}(x_{1:N_{t+1}})=\mathbb{P}(x_{1:N_{t+1}})=\int_{z}\int_{\ell_{1:t}}\mathbb{P}(z,\ell_{1:t},x_{1:N_{t+1}})d\ell_{1:t}dz,$ $$

where we assume tokens $x_{1:N_{t}}$ encode the context information, i.e., information for $\ell_{1:t}$, and the user will decode action $a$ from $x_{N_{t}+1:N_{t+1}}$. More formally, we denote the mappings that decode such semantic information into numeric values as $f$, $g$, such that $f(x_{N_{i-1}+1:N_{i}})=\ell_{i}\in\mathbb{R}^{d}$ for each $i\in[t]$ and $g(x_{N_{t}+1:N_{t+1}})=a\in\mathcal{A}$.

To further understand our assumption, we provide the following example and lemma, which recovers the prominent human behavior model introduced in [Section 4.2](#S4.SS2), quantal response, under some canonical data distributions.

###### Example 1 (An example instantiating 1 ) .

We consider a common decision-making task that may generate the training data, *recommender systems*. An instance of the text data could be: “On September 29, 2023, user X clicked movie A three times, movie B eight times, and movie C five times”. This sentence corresponds to $x_{N_{i-1}+1:N_{i}}$ for some $i\in[t]$ and serves as a natural language depiction of the numerical $\ell_{i}$.
The corresponding label $x_{N_{t}+1:N_{t+1}}$ can be obtained by some user survey: “User X’s favorite movie is movie B”. Meanwhile, $z$ represents user X’s latent, genuine preference for each movie – information that is private to the user, and cannot be observed or collected in the pre-training dataset. In this example, [1](#Thmassumption1) suggests that $x_{1:N_{t}}$, which records the frequency of interactions with each movie, serves as an *imperfect estimate* of the user’s latent, genuine preference for the movies, while the text $x_{N_{t}+1:N_{t+1}}$ depicts the user’s favorite movie only based on her latent $z$.

###### Lemma 1 .

*(Alignment of [1](#Thmassumption1) with quantal response).* Fix $t\in[T]$, $\sigma>0$. If we model the noise of data collection to be i.i.d. Gaussian distribution in the numeric value space, i.e.,

$$ $\mathbb{P}\left(\left\{f(x_{N_{i-1}+1:N_{i}})\right\}_{i\in[t]}{\,\big{|}\,}z\right)=\prod_{i=1}^{t}\mathbb{P}\left(f(x_{N_{i-1}+1:N_{i}}){\,\big{|}\,}z\right)\propto\prod_{i=1}^{t}\exp\left(-\frac{\|f(x_{N_{i-1}+1:N_{i}})-z\|_{2}^{2}}{2\sigma^{2}}\right),$ $$

the prior distribution of the latent variable $z$ is also Gaussian, i.e., $z\sim\mathcal{N}(\boldsymbol{0}_{d},\sigma^{2}I)$, and the text labels satisfy that
$\mathbb{P}(g(x_{N_{t}+1:N_{t+1}}){\,|\,}z)=\mathbbm{1}\left(g(x_{N_{t}+1:N_{t+1}})\in\operatorname*{arg\,min}_{a\in\mathcal{A}}z_{a}\right)$, then we have

$$ $\mathbb{P}\left(g(x_{N_{t}+1:N_{t+1}}){\,\big{|}\,}x_{1:N_{t}}\right)=P_{quantal}^{\sigma\sqrt{t+1}}\left(g(x_{N_{t}+1:N_{t+1}}){\,\big{|}\,}\left\{f(x_{N_{i-1}+1:N_{i}})\right\}_{i\in[t]}\right),$ $$

with $P_{noise}=\mathcal{N}(\boldsymbol{0}_{d},I)$ in [Definition 2](#Thmdefinition2), i.e., the action $a=g(x_{N_{t}+1:N_{t+1}})$ extracted from the text $x_{N_{t}+1:N_{t+1}}$ is a quantal response w.r.t. the loss vectors $\left(f(x_{N_{i-1}+1:N_{i}})\right)_{i\in[t]}$.

We defer the proof of the lemma to [Section C.2](#A3.SS2). Now based on this lemma, we provide the no-regret guarantees of the corresponding pre-trained LLM.

###### Theorem 1 .

*(Emergence of no-regret behavior).*
Under the assumptions of Lemma [1](#Thmlemma1), suppose the function class of $\text{LLM}_{\theta}$ is expressive enough such that for all $t\in[T]$, $\max_{\theta\in\Theta}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log\text{LLM}_{\theta}\left(x_{j}{\,|\,}x_{1:j-1}\right)=\max_{\left\{q_{j}\in\left\{\mathcal{V}^{j-1}\rightarrow\Delta(\mathcal{V})\right\}\right\}_{j\in[N_{t+1}]}}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log q_{j}\left(x_{j}{\,|\,}x_{1:j-1}\right)$, where we define $q_{1}(x_{1}{\,|\,}x_{1:0}):=q_{1}(x_{1})$, and
$\theta^{\star}$ maximizes [Equation 1](#S4.E1). Then,
there exist (simple) algorithms using $\text{LLM}_{\theta^{\star}}$ to achieve no (dynamic) regret for (non-stationary) online learning with full-information/bandit feedback.
To be specific, for (2) and (4), by defining the variation bound $\sum_{t=1}^{T-1}\|\ell_{t+1}-\ell_{t}\|_{\infty}\leq V_{T}$ such that $V_{T}\leq T$ and $V_{T}=\Theta(T^{\rho})$ for some $\rho\in(0,1)$, it holds that for large enough $T$, $d$:

(1) For online learning with full-information feedback, $\text{Regret}_{\text{LLM}{{}_{\theta^{\star}}}}\left(({\ell}_{t})_{t\in[T]}\right)\leq\mathcal{O}\left(\sqrt{T\log d}\right)$;

(2) For non-stationary online learning with full-information feedback,

$$ $\text{D-Regret}_{\text{LLM}{{}_{\theta^{\star}}}}\left(({\ell}_{t})_{t\in[T]}\right)\leq\mathcal{O}\left((\log d\ V_{T})^{1/3}T^{2/3}\right);$ $$

(3) For online learning with bandit feedback,

$$ $\mathbb{E}\left[\text{Regret}_{\text{LLM}{{}_{\theta^{\star}}}}\left(({\ell}_{t})_{t\in[T]}\right)\right]\leq\mathcal{O}\left((\log d)^{1/2}dT^{1/2+1/\log T}\log T\right);$ $$

(4) For non-stationary online learning with bandit feedback,

$$ $\mathbb{E}\left[\text{D-Regret}_{\text{LLM}{{}_{\theta^{\star}}}}\left(({\ell}_{t})_{t\in[T]}\right)\right]\leq\mathcal{O}\left((T^{2}d^{2}V_{T})^{1/3}(\log d)^{1/2}T^{1/\log T}\log T\right).$ $$

###### Remark 1 (Implication for playing repeated games) .

First, we note that the no-regret guarantee in the online setting is stronger than and thus implies that in the game setting, since regret by definition handles arbitrary/adversarial environments, while in playing games the opponents are not necessarily as adversarial. Second, it is folklore that if all players in the repeated game follow no-regret learning algorithms, then the time-average policies of all players during learning constitute an approximate *coarse correlated equilibrium* of the game (Cesa-Bianchi and Lugosi, ). Hence, the results (1) and (2) in [Theorem 1](#Thmtheorem1) imply that a coarse correlated equilibrium will emerge in the long run from the interactions of the LLM agents (under certain assumptions as in the theorem).

We present proofs of the non-asymptotic bounds for (dynamic) regret
in [Section C.4](#A3.SS4). Furthermore, we demonstrate that the prior distribution of
$z$ could also be replaced by a general distribution (c.f. [Theorem 5](#Thmtheorem5)), in order to obtain the above results. We also point out that [1](#Thmassumption1) may be further relaxed to better match the actual LLMs’ pre-training data distributions from diverse sources (c.f. [Section C.5](#A3.SS5)).

It is important to observe that even when pre-training is conducted solely with *stationary* loss vector generation ($\ell_{1:t}$ are i.i.d. conditioned on $z$), it can still lead to the emergence of no-regret behavior in online learning with potentially adversarial losses. Key in the proof is a connection of pre-trained LLM models to the online learning algorithm of FTPL.

###### Assumption 1 .

###### Example 1 (An example instantiating 1 ) .

###### Lemma 1 .

###### Theorem 1 .

###### Remark 1 (Implication for playing repeated games) .

##### Comparison to Lee et al. ( 2023 ); Lin et al. ( 2024 ) .

Intriguingly, similar assumptions and pre-training objectives have also been considered in the very recent work of Lee et al. (); Lin et al. () for studying in-context reinforcement learning property of Transformers under supervised pre-training. Lee et al. () established its equivalence to *posterior sampling* (Osband et al., ), an important RL algorithm with provable regret guarantees when the environments are *stationary*, and Lin et al. () generalized the study to the setting of algorithm distillation as in Laskin et al. (). However, their results cannot directly imply the no-regret guarantee in our online learning setting, due to the fact that posterior sampling can perform poorly under potentially *adversarial* or *non-stationary* environments (Zimmert and Seldin, ; Liu et al., ).
In contrast, we here establish the equivalence of the pre-trained LLM to the FTPL algorithm (under different pre-training distribution specifications), with the ability to handle arbitrary loss sequences, even though the LLMs are only trained on a fixed distribution of texts (tasks).

##### Calibrating the degree of bounded rationality of actual LLMs.

To further validate our model and data distribution assumptions, we also propose to *calibrate* the parameter $\{\eta_{t}\}_{t\in[T-1]}$ in [Definition 2](#Thmdefinition2), the degree of bounded rationality, by estimating the parameters of $\{\eta_{t}\}_{t\in[T-1]}$ using data from interacting with LLMs (following the same protocol as before),
with $P_{{noise}}$ being a standard normal distribution (note that we do not need to calibrate $\eta_{0}$ by [Definition 2](#Thmdefinition2)).
Specifically, given $n$ episodes of the LLM agent’s behavior $\{({\ell}_{t}^{(j)},\pi_{t}^{(j)})_{t\in[T]}\}_{j\in[n]}$, motivated by our [Lemma 1](#Thmlemma1) and [Theorem 1](#Thmtheorem1),
we calibrate $\{\eta_{t}\}_{t\in[T-1]}$ by solving the following problem

$$ $\displaystyle\sigma^{\star}\in\arg\min_{\sigma>0}\quad\sum_{t\in[T-1]}\sum_{j\in[n]}\Big{\|}\pi_{t+1}^{(j)}-P^{\sigma\sqrt{t+1}}_{quantal}\Big{(}\cdot{\,\bigg{|}\,}{\ell}_{1:t}^{(j)}\Big{)}\Big{\|}_{1},\qquad\quad\eta_{t}^{\star}=\sigma^{\star}\sqrt{t+1},\quad\forall t\in[T-1].$ $$

We solve this single-variable optimization problem by grid search over $[0,10]$. We then run the generalized quantal response model with the calibrated $\{\eta_{t}^{\star}\}_{t\in[T-1]}$ on another *unseen test set*, and compare it with the behavior of the actual LLM agents. We use all the interaction data from [Section 3.2](#S3.SS2) and split it in half for training and testing. In [Figure 8](#S4.F8), we show the averaged regret for the LLM agent and the calibrated generalized quantal response. It can be seen that calibrated generalized quantal response can *very well capture* the behavior of the LLM agent for all problem instances in [Section 3.2](#S3.SS2), justifying the applicability of our hypothetical model and assumptions.

Figure: Figure 8: (left) Comparison of GPT-4 with a calibrated agent on the test set, where the calibrated quantal response can perfectly capture the behavior of the GPT-4 agent. (mid, right) The calibrated agent on the less predictable and adaptive loss sequences failed to make accurate predictions for the GPT-4 anymore.
Refer to caption: /html/2403.16843/assets/x12.png

We also use the same framework to understand the regrettable behaviors in [Section 3.4](#S3.SS4). This analysis uses all the data from [Section 3.4](#S3.SS4). We first find that such fitting procedures do not yield good predictions for LLMs on those counter-examples. Therefore, we resort to a more expressive model by directly fitting each $\eta_{t}$ as
$\eta_{t}^{\star}\in\arg\min_{\eta_{t}>0}\sum_{j\in[n]}\Big{\|}\pi_{t+1}^{(j)}-P^{\eta_{t}}_{quantal}\Big{(}\cdot{\,\bigg{|}\,}{\ell}_{1:t}^{(j)}\Big{)}\Big{\|}_{1}$ separately for each $t\in[T-1]$.
Even under the expressive model, LLMs fail to follow the generalized quantal response for the counter-examples with less predictable or adaptive loss sequences, as [Figure 8](#S4.F8) shows the gap between GPT-4 (dynamic) regret and the calibrated agent (dynamic) regret.

Finally, we acknowledge that for most existing pre-trained LLMs like GPT-4, the canonical assumptions above, though may be further relaxed (c.f. [Remark 4](#Thmremark4)), may not hold in general. More importantly, the *supervision labels*, i.e., the optimal action given $z$, may be sometimes imperfect or unavailable during the dataset collection. Hence, it is completely possible to observe regrettable behaviors (c.f. LABEL:{sec:yes_regret_example}). Motivated by these caveats, we next propose a new training loss that is *unsupervised*, and can promote no-regret behavior provably.

## 5 Provably Promoting No-Regret Behavior by an Unsupervised Loss

In light of the observations in [Section 3](#S3), we ask the question:

*Is there a way to further enhance the no-regret property of LLM agents,
hopefully without (optimal) action labels?*

To address this question, we propose to train LLMs with a new *unsupervised learning* loss that naturally provides no-regret behaviors. This approach is akin to the process of “instruction tuning” (Wei et al., ), which was shown to have enhanced LLMs’ ability when learning from context, with both theoretical (Ahn et al., ; Mahankali et al., ; Zhang et al., ) and empirical (Lu et al., ) evidence.

### 5.1 A New Unsupervised Training Loss: Regret-Loss

Intuitively, our new training loss is designed to enforce the trained LLM to minimize the regret under an arbitrary sequence of loss vectors. Specifically,
we define the training loss as

$$ $\displaystyle{\mathcal{L}(\theta)}:=\max_{\ell_{1},\dots,\ell_{T}}~{}~{}~{}\text{Regret}_{\text{LLM}_{\theta}}\left((\ell_{t})_{t\in[T]}\right)$ (2) $$

where $\|\ell_{t}\|_{\infty}\leq B$ for $t\in[T]$.
As discussed in Kirschner et al. (), directly minimizing the max regret can be computationally challenging, except for superficially simple problems.
Hence, in practice, one may parameterize the LLM and resort to differentiable programming to solve it approximately. However,
[Equation 2](#S5.E2) is not necessarily differentiable with respect to parameter $\theta$, if it does not satisfy the condition of Danskin’s Theorem (Danskin, ); or even if it is differentiable (i.e., the maximizer of $(\ell_{t})_{t\in[T]}$ is unique), computation of derivatives can be challenging since we need to calculate $\operatorname*{arg\,max}_{(\ell_{t})_{t\in[T]}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})$ while we have $\inf$ in the definition of regret.
Therefore, we provide a general framework so that we can approximate [Equation 2](#S5.E2) by the following surrogate:

$$ $\displaystyle\mathcal{L}(\theta,k,N):=\mathbb{E}\left[\frac{\sum_{j\in[N]}h(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))f(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}{\sum_{j\in[N]}f(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}\right],$ (3) $$

where
$k\in\mathbb{N}^{+}$, $N\in\mathbb{N}^{+}$, $h:\mathbb{R}\to\mathbb{R}^{+}$ is a continuous function, with continuous derivative $h^{\prime}$, and $f(\cdot,k):\mathbb{R}\to\mathbb{R}^{+}$ is a continuous function for each $k\in\mathbb{N}^{+}$, satisfying $\lim_{k\to\infty}\frac{f(R_{1},k)}{f(R_{2},k)}=\infty\cdot\mathbbm{1}(R_{1}>R_{2})+\mathbbm{1}(R_{1}=R_{2})$, where we use the convention of $\infty\cdot 0=0$. These conditions on $h,f$ will be assumed throughout the paper.
Examples of such an $f$ include $f(x,k)=x^{k}$ and $\exp(kx)$.
We will sample $N$ trajectories of loss sequences $(\ell_{t}^{(j)})_{t\in[T],j\in[N]}$ from some continuous
probability distribution supported on $[-B,B]^{T\times N}$, and the expectation in [Equation 3](#S5.E3) is thus taken with respect to this distribution. Note that we do not have any statistical assumption on $(\ell_{t}^{(j)})_{t\in[T],j\in[N]}$ (except being continuous and supported on $[-B,B]^{T\times N}$),
in contrast to those in [Section 4](#S4) to *justify* the no-regret property of pre-trained LLMs.

In [Section D.2](#A4.SS2), we prove that under certain regularity conditions of $f$ and $h$, we have

$$ $\lim_{N,k\to\infty}\mathcal{L}(\theta,k,N)=h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right),$ $$

as well as the uniform convergence of $\mathcal{L}(\theta,k,N)$:
$\lim_{N,k\to\infty}\sup_{\theta\in\Theta}\Big{|}h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))-\mathcal{L}(\theta,k,N)\Big{|}=0,$
where $\Theta$ is some compact set of the LLM parameter. Hence, one can expect that minimizing the loss function in [Equation 3](#S5.E3) with large enough $k$ and $N$ may promote the trained LLM to have a small regret value. We will hereafter refer to [Equation 3](#S5.E3) as the regret-loss. Similarly, we can also define dynamic-regret-loss, and the results to be presented next can also generalize to this case (c.f. [Remark 5](#Thmremark5) in [Section D.3](#A4.SS3)).

### 5.2 Guarantees via Regret-Loss Minimization

We first establish a *statistical* guarantee under general parameterizations of $\text{LLM}_{\theta}$ that is Lipschitz with respect to $\theta$, including the Transformer-based models as used in GPT-4 and most existing LLMs (see [Proposition 2](#Thmproposition2) for an example with formal statement).
This guarantee focuses on their *generalization ability* when trained to minimize the empirical regret loss, which is defined as follows:

###### Definition 3 (Empirical loss function) .

We define the empirical loss $\widehat{\mathcal{L}}$ computed with $N_{T}$ samples as follows:

$$ $\displaystyle\widehat{\mathcal{L}}(\theta,k,N,N_{T}):=\frac{1}{N_{T}}\sum_{s=1}^{N_{T}}\left[\frac{\sum_{j\in[N]}h\left(\emph{Regret}_{\emph{LLM}_{\theta}}((\ell_{s,t}^{(j)})_{t\in[T]})\right)f\left(\emph{Regret}_{\emph{LLM}_{\theta}}((\ell_{s,t}^{(j)})_{t\in[T]}),k\right)}{\sum_{j\in[N]}f\left(\emph{Regret}_{\emph{LLM}_{\theta}}((\ell_{s,t}^{(j)})_{t\in[T]}),k\right)}\right]$ (4) $$

where $(\ell_{s,t}^{(j)})_{j\in[N],t\in[T]}$ denotes the $s$-th sample of $(\ell_{t}^{(j)})_{j\in[N],t\in[T]}$ for estimating $\mathcal{L}(\theta,k,N)$.

We denote $\widehat{\theta}_{k,N,N_{T}}\in\operatorname*{arg\,min}_{\theta\in\Theta}~{}\widehat{\mathcal{L}}(\theta,k,N,N_{T})$, and present the generalization guarantee below.

###### Theorem 2 .

*(Generalization gap).* Suppose $\emph{LLM}_{\theta}$ is Lipschitz-continuous with respect to the model parameter $\theta$, then
for any $0<\epsilon<1/2$, with probability at least $1-\epsilon$, we have

$$ $\displaystyle\mathcal{L}\left(\widehat{\theta}_{k,N,N_{T}},k,N\right)$ $\displaystyle-\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)\leq\widetilde{\mathcal{O}}\left(\sqrt{\frac{d_{\theta}+\log(1/\epsilon)}{N_{T}}}\right),$ (5) $$

for any $N$ and sufficiently large $k$, where $d_{\theta}$ is the dimension of the parameter $\theta$.

Through a careful use of Berge’s Maximum Theorem (Berge, ), we prove that the right-hand side of [Equation 5](#S5.E5) does *not* depend on $k$ and $N$, which allows us to take the limit of $\lim_{N\to\infty}\lim_{k\to\infty}$ without affecting the generalization bound. Thanks to the uniform convergence of $\mathcal{L}(\theta,k,N)$ (c.f. [Section D.2](#A4.SS2)), we further obtain the following corollary on the regret guarantee:

###### Corollary 1 .

*(Regret).*
Suppose(^§^§§Note that these conditions on $h,f$ are in addition to those specified after [Equation 3](#S5.E3).) for any $k\in\mathbb{N}^{+}$, $h,f(\cdot,k)$ are non-decreasing, and $\log f$ is a supermodular function (i.e., $\log f(R_{1},k_{1})-\log f(R_{1},k_{2})\geq\log f(R_{2},k_{1})-\log f(R_{2},k_{2})$ for $R_{1}\geq R_{2}$ and $k_{1}\geq k_{2}$). Then, with high probably,
we have

$$ $\displaystyle h\left(\lim_{N\to\infty}\lim_{k\to\infty}\max_{\|\ell_{t}\|_{\infty}\leq B}\emph{Regret}_{\emph{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}\left((\ell_{t})_{t\in[T]}\right)\right)\leq h\left(\inf_{{\theta\in\Theta}}\max_{\|\ell_{t}\|_{\infty}\leq B}\emph{Regret}_{\emph{LLM}_{{\theta}}}\left((\ell_{t})_{t\in[T]}\right)\right)+\widetilde{\mathcal{O}}\left(\sqrt{\frac{d_{\theta}}{{N_{T}}}}\right).$ (6) $$

###### Corollary 2 .

*(Emerging behavior: Coarse correlated equilibrium)*.
For a sufficiently large $N_{T}$, if each agent in the matrix game plays according to $\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}$, then the time-averaged policy for each agent will constitute an approximate coarse correlated equilibrium of the game.

Proofs of [Theorem 2](#Thmtheorem2) and [Corollary 1](#Thmcorollary1) are deferred to [Section D.3](#A4.SS3), and [Corollary 2](#Thmcorollary2) follows from the folklore result discussed in [Remark 1](#Thmremark1).
Therefore, if additionally, the LLM parameterization (i.e., Transformers) can realize a no-regret algorithm (for example, the single-layer self-attention model can construct FTRL, as to be shown next in [Section 5.3](#S5.SS3)), then [Corollary 1](#Thmcorollary1) means that with a large enough number of samples $N_{T}$, the learned $\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}$ becomes a *no-regret* learner, i.e., $\text{Regret}_{\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}\left((\ell_{t})_{t\in[T]}\right)=o(T)$, since the first term on the right-hand-side of [Equation 6](#S5.E6) would directly be $o(T)$ under the choice of $h(x)=\max\{0,x\}$. For other choices of $h$, one can use the inverse function of $h$, i.e., $h^{-1}$ (which always exists by our requirement of $h$), to ensure $\text{Regret}_{\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}((\ell_{t})_{t\in[T]})$
is of order $o(T)$.

Despite the power and generality of the previous results, one cannot use an *infinitely large* $N$ and $k$ in practical training. Hence, in the next subsection, we provide results when $N$ is finite, for the specific parameterization of the LLMs using Transformers.

###### Definition 3 (Empirical loss function) .

###### Theorem 2 .

###### Corollary 1 .

###### Corollary 2 .

### 5.3 Minimizing Regret-Loss Can Automatically Produce Known Online Learning Algorithms

We now study the setting of minimizing [Equation 3](#S5.E3) when $\text{LLM}_{\theta}$ is specifically parameterized by Transformers. As an initial step, we focus on single-layer (linear) self-attention models, as in most recent theoretical studies of Transformers (Ahn et al., ; Zhang et al., ; Mahankali et al., ), and the more practical setting with a finite $N=1$. Note that in this case, the choice of $f$ (and thus $k$) is not relevant. Thus, throughout this section, we drop superscript $(j)$ in [Equation 3](#S5.E3) for notational convenience. We sample $\ell_{t}$ for $t\in[T]$ as realizations of some random variable $Z$. Here, we assume $Z$ is symmetric about zero (i.e., $Z\overset{d}{=}-Z$), $\text{Var}(Z)=\Sigma$ is positive definite. We also assume that the support of $Z$ has an interior such that it contains a ball centered at $\boldsymbol{0}_{d}$.

Firstly, we consider the following structure of single-layer self-attention model $g$ (see a formal introduction in [Section A.1](#A1.SS1)):

$$ $\displaystyle g(Z_{t};V,K,Q,v_{c},k_{c},q_{c}):=(V\ell_{1:t}+v_{c}\boldsymbol{1}_{t}^{\intercal})\texttt{Softmax}\left((K\ell_{1:t}+k_{c}\boldsymbol{1}_{t}^{\intercal})^{\intercal}\cdot(Qc+q_{c})\right),$ (7) $$

where $Z_{t}=(\ell_{1},\dots,\ell_{t},c)$, $\ell_{1:t}\in\mathbb{R}^{d\times t}$ denotes a matrix with each column corresponding to each $\ell_{i}$, and $V,K,Q\in\mathbb{R}^{d\times d}$ correspond to the value, key, and query matrices, respectively, $v_{c},k_{c},q_{c}\in\mathbb{R}^{d}$ correspond to the bias terms associated with $V,K,Q$, and $c\neq\boldsymbol{0}_{d}$ is a constant vector. We then have the following result.

###### Theorem 3 .

Consider the policy space $\Pi=B(0,R_{\Pi},\|\cdot\|)$ for some $R_{\Pi}>0$.
The configuration of a single-layer self-attention model in [Equation 7](#S5.E7) $(V,K,Q,v_{c},k_{c},q_{c})$ such that $K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d}$ and

$$ $V=-R_{\Pi}\frac{T}{\sum_{t=1}^{T-1}1/t}\Sigma^{-1}\mathbb{E}\left[\bigg{\|}{\sum_{t=1}^{T}\ell_{t}}\bigg{\|}\ell_{1}\ell_{2}^{\intercal}\right]\Sigma^{-1}$ $$

is a first-order stationary point of [Equation 3](#S5.E3) with $N=1$, $h(x)=x^{2}$.
Moreover, if $\Sigma$ is a diagonal matrix, then plugging this configuration into [Equation 7](#S5.E7), and projecting the output with $\texttt{Proj}_{\Pi,\|\cdot\|}$ would perform FTRL with an $L_{2}$-regularizer for the loss vectors $(\ell_{t})_{t\in[T]}$.

In practical training, such stationary points of the loss may be attained by first-order optimization algorithms of (stochastic) gradient descent, the workhorse in machine learning.
Moreover, we also consider the single-layer *linear* self-attention model as follows, for which we can strengthen the results above from a stationary-point to an *optimal-solution* argument:

$$ $\displaystyle g(Z_{t};V,K,Q,v_{c},k_{c},q_{c})=\sum_{i=1}^{t}(V\ell_{i}+v_{c})\left((K\ell_{i}+k_{c})^{\intercal}\cdot(Qc+q_{c})\right).$ (8) $$

###### Theorem 4 .

Consider the policy space $\Pi=B(0,R_{\Pi},\|\cdot\|)$ for some $R_{\Pi}>0$.
The configuration of a single-layer linear self-attention model in [Equation 8](#S5.E8) $(V,K,Q,v_{c},k_{c},q_{c})$ such that $K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d}$ and $V=-2R_{\Pi}\Sigma^{-1}\mathbb{E}\left(\|\sum_{t=1}^{T}\ell_{t}\|\ell_{1}\ell_{2}^{\intercal}\right)\Sigma^{-1}$ is a global optimal solution of [Equation 3](#S5.E3) with $N=1$, $h(x)=x^{2}$.
Moreover, every global optimal configuration of [Equation 3](#S5.E3) within the parameterization class of [Equation 8](#S5.E8) has the same output function $g$. Additionally, if $\Sigma$ is a diagonal matrix, then plugging any global optimal configuration into [Equation 8](#S5.E8), and projecting the output with $\texttt{Proj}_{\Pi,\|\cdot\|}$
would perform FTRL with an $L_{2}$-regularizer for the loss vectors $(\ell_{t})_{t\in[T]}$.

[Theorem 4](#Thmtheorem4) shows the capacity of self-attention Transformer models to realize online learning algorithms, thanks to the regret-loss we proposed. In particular, this can be achieved automatically by optimizing the new loss, *without* hard-coding the parameters of the Transformer.

The above results are for the case of FTRL with an $L_{2}$-regularizer, and it is possible to consider FTRL with an *entropy regularizer*, leading to the well-known Hedge algorithm (Freund and Schapire, ) that is more compatible with the simplex constraint on $\pi$ in the Experts Problem. We defer the discussion of this case to [Section D.7](#A4.SS7). Through these results, we can also guarantee in the repeated game setting that approximate coarse correlated equilibria would emerge in the long run,
since each player will exhibit no-regret behavior, using a similar argument as that for [Corollary 2](#Thmcorollary2).

###### Remark 2 .

The very recent studies by (Ahn et al., ; Zhang et al., ; Mahankali et al., ) have demonstrated that if $Z_{t}=((x_{1},y_{1}),\dots,(x_{t},y_{t}),(x_{t+1},0))$ and the “instruction tuning” loss (i.e., $\mathbb{E}[\|{\widehat{y}_{t+1}-y_{t+1}}\|^{2}]$) is being minimized with a single-layer linear self-attention model, then a global optimizer among single-layer linear self-attention models yields the output $\widehat{y}_{n+1}=\eta\sum_{i=1}^{n}y_{i}x_{i}^{\intercal}x_{n+1}$. This output can be interpreted as a *gradient descent* algorithm, indicating that a single-layer linear self-attention model implicitly performs gradient descent. However, in the online learning setting where there are no $y$-labels, such an implicit gradient descent update-rule is hard to define. Compared to the previous studies, our global optimizer among single-layer linear self-attention models is an *explicit* and *online* gradient descent update for online learning. With a different loss (regret-loss v.s. instruction-tuning-loss), the techniques to obtain the seemingly similar results are also fundamentally different.

###### Theorem 3 .

###### Theorem 4 .

###### Remark 2 .

### 5.4 Experimental Results for Minimizing Regret-Loss

We now provide experimental results for minimizing our *regret-loss*, and evaluate in the following environments: 1) randomly-generated loss sequences ([Figure 9](#S5.F9)); 2) loss sequences with a predictable trend ([Figure 10](#S5.F10)); 3) repeated games ([Figure 11](#S5.F11)); and 4) counterexamples for pre-trained LLMs to be regrettable ([Figure 7](#S3.F7)). Details of the training setup can be found in [Section D.8](#A4.SS8). We also provide an ablation study for the training of the loss [Equation 3](#S5.E3) in [Section D.9](#A4.SS9).

##### Randomly generated loss sequences.

We use the same loss vectors as those in [Section 3.2](#S3.SS2) for randomly generated loss functions, and compare the results with that using GPT-4. The results show that with regret-loss, both the trained single-layer self-attention model and the trained Transformers with multi-layer self-attention structures can achieve comparable regrets as FTRL and GPT-4. The results can be found in [Figure 9](#S5.F9).

Figure: Figure 9: Regret performance for the randomly generated loss sequences that are generated by Gaussian with truncation and uniform distribution. No-regret behaviors of single-layer and multi-layer self-attention models are validated by both of our frameworks (low $p$-values and $\widehat{\beta}_{0}<1$).
Refer to caption: /html/2403.16843/assets/x15.png

Figure: Figure 10: Regret performance for the randomly generated loss sequences that are generated by linear-trend and sine-trend. No-regret behaviors of single-layer and multi-layer self-attention models are validated by both of our frameworks (low $p$-values and $\widehat{\beta}_{0}<1$).
Refer to caption: /html/2403.16843/assets/x17.png

##### Loss sequences with a predictable trend.

We investigate the case where the loss sequences have predictable trends such as linear-trend or sine-trend. One might expect that the performance of the trained Transformer would surpass the performance of traditional no-regret learning algorithms such as FTRL, since they may not be an optimal algorithm for the loss sequence with a predictable trend. We modify the training distribution by changing the distribution of random variable $Z$ (which generates the loss vectors $\ell_{t}$) to follow two kinds of trends: linear and sine functions. The results, as illustrated in [Figure 10](#S5.F10), show that the trained single-layer self-attention model and the trained Transformer with multi-layer self-attention structures with regret-loss outperformed GPT-4 and FTRL in terms of regret, when the loss sequence is a linear trend. Similarly, [Figure 10](#S5.F10) shows that the trained Transformer with multi-layer self-attention structures with regret-loss is comparable to GPT-4 and outperformed FTRL in terms of regret, when the loss sequence is a sine-trend. Note that the training dataset does not contain the sequence of losses. Nonetheless, by focusing on the overall trend during training, we can attain performance that is either superior to or on par with that of FTRL and GPT-4.

##### Repeated games.

We then investigate the case of multi-player repeated games. We study 2x2, 3x3x3, 3x3x3x3 games, where each entry of the payoff matrix is sampled randomly from $\operatorname{Unif}([0,10])$. The results, as illustrated in [Figure 11](#S5.F11), show that the trained single-layer self-attention model and the trained Transformer with multi-layer self-attention structures with regret-loss have a similar performance as that of FTRL. However, GPT-4 still outperforms the trained single-layer self-attention model and the trained Transformer with multi-layer self-attention structures in terms of regret. Since for repeated games (in which the environment faced by the agent can be less adversarial than that in the online setting), there might be a better algorithm than FTRL (see e.g., Daskalakis et al. ()), while our self-attention models have a similar structure as FTRL ([Theorem 3](#Thmtheorem3) or [Theorem 4](#Thmtheorem4)). Also, in practical training (with the empirical loss in [Equation 4](#S5.E4)), we possibly did not find the exact global minimum or stationary point of the *expected* loss in [Equation 3](#S5.E3). Hence, it is possible that GPT-4 may have lower regret than our trained models with the regret-loss.

Figure: Figure 11: Regret performance for the game with two players, three players, and four players general-sum games. No-regret behaviors of single-layer and multi-layer self-attention models are validated by both of our frameworks (low $p$-values and $\widehat{\beta}_{0}<1$).
Refer to caption: /html/2403.16843/assets/x19.png

##### Two scenarios that caused regrettable behaviors of GPT-4.

Finally, we investigate the cases that have caused GPT-4 to have regrettable performance in [Section 3.2](#S3.SS2).
The results, which can be found in [Figure 7](#S3.F7), show that both the trained single-layer self-attention model and the trained Transformer with regret-loss can achieve comparable no-regret performance as FTRL, and outperforms that of GPT-4. This validates that our new unsupervised training loss can address the regrettable cases, as our theory in [Sections 5.2](#S5.SS2) and [5.3](#S5.SS3) has predicted.

## 6 Concluding Remarks

In this paper, we studied the online decision-making and strategic behaviors of LLMs quantitatively, through the metric of regret. We first examined and validated the no-regret behavior of several representative pre-trained LLMs in benchmark settings of online learning and games. As a consequence, (coarse correlated) equilibrium can oftentimes emerge as the long-term outcome of multiple LLMs playing repeated games. We then provide some theoretical insights into the no-regret behavior, by connecting pre-trained LLMs to the follow-the-perturbed-leader algorithm in online learning, under certain assumptions. We also identified (simple) cases where pre-trained LLMs fail to be no-regret, and thus proposed a new unsupervised training loss, *regret-loss*, to provably promote the no-regret behavior of Transformers without the labels of (optimal) actions. We established both experimental and theoretical evidence for the effectiveness of our regret-loss.

As a first attempt toward rigorously understanding the online and strategic decision-making behaviors of LLMs through the metric of regret, our work has opened up fruitful directions for future research:

- •
There are more than one definitions of (dynamic-)regret in the online learning literature, and we mainly focused on the so-called *external-regret* in the literature. It would be interesting to study the no-regret behavior of LLMs in terms of other regret metrics, e.g., swap-regret (Blum and Mansour, ), which may lead to stronger equilibrium notions in playing repeated games.
- •
Our new regret-loss has exhibited promises in our experiments for training modest-scale Transformers. We are currently generalizing it to training other larger-scale models, such as Foundation Models, for decision-making.
- •
No-regret behavior can sometimes lead to better outcomes in terms of social efficiency (Blum et al., ; Roughgarden, ; Nekipelov et al., ). It would thus be interesting to further validate the efficiency of no-regret LLM agents in these scenarios, as well as identifying new prompts and training losses for LLMs to promote the efficiency of the outcomes.
- •
To evaluate the performance quantitatively, we focused on online learning and games with *numeric valued* payoffs. It would be interesting to connect our no-regret-based and game-theoretic framework with existing multi-LLM frameworks, e.g., debate, collaborative problem-solving, and human/social behavior simulation, with potentially new notions of regret (defined in different spaces) as performance metrics.

## Acknowledgement

The authors thank Dingwen Kong for discussing the truncation idea in proving [Lemma 8](#Thmlemma8). Also, the authors thank Kristian Georgiev and Aleksander Madry for the helpful feedback.

## Appendix A Deferred Background

### A.1 Additional Definitions for Appendix

##### (Linear) Self-attention.

One key component in Transformers (Vaswani et al., ), the backbone of modern language models, is the *(self-)attention* mechanism. For simplicity, we here focus on introducing the *single-layer* self-attention architecture.
The mechanism takes a sequence of vectors $Z=[z_{1},\dots,z_{t}]\in\mathbb{R}^{d\times t}$ as input, and outputs some sequence of $[\widehat{z}_{1},\dots,\widehat{z}_{t}]\in\mathbb{R}^{d\times t}$. For each $i\in[t]$ where $i>1$, the output is generated by $\widehat{z}_{i}=(Vz_{1:{i-1}})\sigma((Kz_{1:i-1})^{\intercal}(Qz_{i}))$, where $z_{1:{i-1}}$ denotes the $1$ to $i-1$ columns of $Z$, $\sigma$ is either the Softmax or ReLU activation function, and for the initial output, $\widehat{z}_{1}=\boldsymbol{0}_{d}$. Here, $V,Q,K\in\mathbb{R}^{d\times d}$ are referred to as the *Value*, *Query*, and *Key* matrices, respectively. Following the theoretical framework in Von Oswald et al. (); Mahankali et al. (), we exclude the attention score for a token $z_{i}$ in relation to itself.
For theoretical analysis, we also consider the *linear* self-attention model, where $\widehat{z}_{i}=(Vz_{1:{i-1}})((Kz_{1:i-1})^{\intercal}(Qz_{i}))$. We write this (linear) self-attention layer’s output as $\texttt{(L)SA}_{(V,Q,K)}(Z)$. We define an $M$-head self-attention layer with $\theta=\{(V_{m},Q_{m},K_{m})\}_{m\in[M]}$ as $\texttt{M-(L)SA}_{\theta}(Z):=\sum_{m=1}^{M}\texttt{(L)SA}_{(V_{m},Q_{m},K_{m})}(Z)$. We define $\|\cdot\|_{\texttt{M-(L)SA}}$ as $\|\theta\|_{\texttt{M-(L)SA}}:=\max_{m\in[M]}\left\{\|Q_{m}\|_{\texttt{op}},\|K_{m}\|_{\texttt{op}}\right\}+\sum_{m=1}^{M}\|V_{m}\|_{\texttt{op}}$.

##### Transformers.

For a multi-layer perceptron (MLP) layer, it takes $Z=[z_{1},\dots,z_{t}]\in\mathbb{R}^{d\times t}$ as input, with parameter $\theta=(W_{1},W_{2})\in\mathbb{R}^{d^{\prime}\times d}\times\mathbb{R}^{d\times d^{\prime}}$ such that for each $i\in[t]$, the output is $\widehat{z_{i}}:=W_{2}\sigma(W_{1}z_{i})$ where $\sigma$ is either Softmax or ReLU. We write the output of an MLP layer with parameter $\theta$ as $\texttt{MLP}_{\theta}(Z)$. Defining $\|\cdot\|_{\texttt{MLP}}$ as $\|\theta\|_{\texttt{MLP}}:=\|W_{1}\|_{\texttt{op}}+\|W_{2}\|_{\texttt{op}}$ and $\texttt{ResNet}(f,Z):=Z+f(Z)$, we can define an $L$-layer Transformer with parameter $\theta=(\theta^{(lm)},\theta^{(la)})_{l\in[L]}$ as

$$ $\displaystyle\texttt{TF}_{\theta}(Z):=Z^{(L)},$ $$

where the output $Z^{(L)}$ is defined iteratively from $Z^{(0)}=\texttt{clip}_{R}(Z):=\min(-R,\max(R,Z))$ and

$$ $\displaystyle Z^{(l)}=\texttt{clip}_{R}\left(\texttt{ResNet}\left(\texttt{MLP}_{\theta^{(la)}},\texttt{ResNet}\left(\texttt{M-(L)SA}_{\theta^{(lm)}},Z^{(l-1)}\right)\right)\right),$ $$

for some $R>0$.
We define a class of Transformers with certain parameters as $\Theta_{d,L,M,d^{\prime},B_{\texttt{TF}}}:=\{\theta=(\theta^{(lm)},\theta^{(la)})_{l\in[L],m\in[M]}:\|\theta\|_{\texttt{TF}}\leq B_{\texttt{TF}}\}$, where $M$ is the number of heads of self-attention,

$$ $\displaystyle\|\theta\|_{\texttt{TF}}:=\max_{l\in[L]}\left\{\|\theta^{(la)}\|_{\texttt{M-(L)SA}}+\|\theta^{(lm)}\|_{\texttt{MLP}}\right\},$ (9) $$

and $B_{\texttt{TF}}>0$ is some constant. When it is clear from the context, we may omit the subscripts and write it as $\Theta$ for simplicity. We assume $R$ to be sufficiently large such that clip does not take effect on any of our approximation results.

### A.2 In-Context Learning

In-context learning is an emergent behavior of LLMs (Brown et al., ), which means that these models can adapt and learn from a limited number of examples provided within their immediate input context. In in-context learning, the prompt is usually constituted by a length of $T$ in-context (independent) examples $(x_{t},y_{t})_{t\in[T]}$ and $(T+1)$-th input $x_{T+1}$, so the $\text{LLM}((z_{t})_{t\in[T]},x_{T+1})$ provides the inference of $y_{T+1}$, where $z_{t}=(x_{t},y_{t})$.

### A.3 Online Learning Algorithms

##### Follow-the-regularized-leader (FTRL).

The Follow-the-Regularized-Leader (FTRL) algorithm (Shalev-Shwartz, ) is an iterative method that updates policy based on the observed data and a regularization term. The idea is to choose the next policy that minimizes the sum of the past losses and a regularization term.

Mathematically, given a sequence of loss vectors ${\ell}_{1},{\ell}_{2},\dots,{\ell}_{t}$, the FTRL algorithm updates the policy $\pi$ at each time step $t$ as follows:

$$ $\pi_{t+1}=\arg\min_{\pi\in\Pi}\left(\sum_{i=1}^{t}\langle{\ell}_{i},\pi\rangle+R(\pi)\right),$ $$

where $R(\pi)$ is a regularization term. The regularization term $R(\pi)$ is introduced to prevent overfitting and can be any function that penalizes the complexity of the model. A function $R(\pi)$ is said to be $\lambda$-strongly convex with respect to a norm $\|\cdot\|$ if for all $\pi,\pi^{\prime}\in\Pi$:

$$ $R(\pi)\geq R(\pi^{\prime})+\langle\nabla R(\pi^{\prime}),\pi-\pi^{\prime}\rangle+\frac{\lambda}{2}\|\pi-\pi^{\prime}\|_{2}^{2}.$ $$

A key property that ensures the convergence and stability of the FTRL algorithm is the strong convexity of the regularization term $R(\pi)$. Strong convexity of $R(\pi)$ ensures that the optimization problem in FTRL has a unique solution. The FTRL algorithm’s flexibility allows it to encompass a wide range of online learning algorithms, from gradient-based methods like online gradient descent to decision-making algorithms like Hedge (Freund and Schapire, ).

##### Connection to online gradient descent (OGD).

The Online Gradient Descent (OGD) (Cesa-Bianchi et al., ) algorithm is a special case of the FTRL algorithm when the regularization term is the $L_{2}$-norm square, i.e., $R(\pi)=\frac{1}{2}\|\pi\|_{2}^{2}$ and $\Pi=\mathbb{R}^{d}$. In OGD, at each time step $t$, the policy $\pi$ is updated using the gradient of the loss function:

$$ $\pi_{t+1}=\pi_{t}-{\ell}_{t}.$ $$

Therefore, the connection between FTRL and OGD can be seen by observing that the update rule for FTRL with $L_{2}$ regularization can be derived from the OGD update rule.

##### Connection to the Hedge algorithm.

The Hedge algorithm (Freund and Schapire, )
(also referred to as the Multiplicative Weight Update algorithm (Arora et al., )) is an online learning algorithm designed for problems where the learner has to choose from a set of actions (denoted as $\mathcal{A}$) at each time step and suffers a loss based on the chosen action. The FTRL framework can be used to derive the Hedge algorithm by considering an entropy regularization term. Specifically, the regularization term is the negative entropy $R(\pi)=\sum_{j\in[d]}\pi_{j}\log\pi_{j}$ (where $d$ is the dimension of policy $\pi$), then the FTRL update rule yields the Hedge algorithm as

$$ $\displaystyle\pi_{(t+1)j}=\pi_{tj}\frac{\exp(-{\ell}_{tj}\pi_{tj})}{\sum_{i\in[d]}\exp(-{\ell}_{ti}\pi_{ti})}$ $$

for $j\in[d]$.

##### Follow-the-perturbed-leader (FTPL).

Given a sequence of loss vectors ${\ell}_{1},{\ell}_{2},\dots,{\ell}_{t-1}$, the Follow-the-Perturbed-Leader algorithm (Kalai and Vempala, ) updates the policy $\pi$ at each time step $t$ by incorporating a perturbation vector $\epsilon_{t}$. This perturbation is *sampled* from a pre-defined distribution. The policy $\pi_{t}$ for the next time step is chosen by solving the following optimization problem:

$$ $\displaystyle\pi_{t}=\mathbb{E}\left[\arg\min_{\pi\in\Pi}\langle\epsilon_{t},\pi\rangle+\sum_{i=1}^{t-1}\langle{\ell}_{i},\pi\rangle\right].$ (10) $$

Here $\epsilon_{t}$ introduces randomness to the decision-making.

##### Relationship between FTRL and FTPL.

The FTRL and FTPL algorithms are deeply related. For example, FTPL with perturbations of Gumbel distribution and FTRL with Entropy Regularization (i.e., Hedge) are equivalent. In general, for the FTPL algorithm with any perturbation distribution, one can always find an FTRL algorithm with a particular regularization such that their update rule is equivalent. However, this relationship does not hold vice versa. For example, Hofbauer and Sandholm () shows that for FTRL with log barrier regularization, there does not exist an equivalent perturbation distribution for FTPL.

##### Restarting techniques for non-stationary online learning.

For non-stationary online learning problems, one common technique is *restarting*: one restarts the standard online learning algorithm periodically (Besbes et al., ) (see also e.g., Wei and Luo (); Mao et al. ()). After each restarting operation, the algorithm will ignore the previous history and execute as if it is the beginning of the interaction with the environment. Since the variation of the loss sequences is bounded, loss sequences between two consecutive restarting operations can be regarded as being *almost stationary*, which makes achieving an overall sublinear dynamic regret guarantee possible.

### A.4 Why Focusing on Linear Loss Function?

We note that focusing on the linear loss function $f_{t}(\pi):=\langle{\ell}_{t},\pi\rangle$ does not lose much of generality. Specifically,
for the general convex loss function $(f_{t})_{t\in[T]}$, we have $f_{t}(\pi_{\mathscr{A},t})-f_{t}(\pi)\leq\langle\nabla f_{t}(\pi_{\mathscr{A},t}),\pi_{\mathscr{A},t}-\pi\rangle$ for any $\pi\in\Pi$, which indicates

$$ $\displaystyle\text{Regret}_{\mathscr{A}}\left((f_{t})_{t\in[T]}\right)\leq\sum_{t=1}^{T}\mathbb{E}[\langle\nabla f_{t}(\pi_{\mathscr{A},t}),\pi_{\mathscr{A},t}\rangle]-\inf_{\pi\in\Pi}\sum_{t=1}^{T}\mathbb{E}[\langle\nabla f_{t}(\pi_{\mathscr{A},t}),\pi\rangle].$ $$

Therefore, one can regard the loss vector $({\ell}_{t})_{t\in[T]}$ as ${\ell}_{t}:=\nabla f_{t}(\pi_{\mathscr{A},t})$ for $t\in[T]$, and control the actual regret by studying the linear loss function (Hazan, ). The same argument on the general convex $f_{t}$ can be applied to the dynamic-regret metric as well. In sum, an algorithm designed for online *linear* optimization can be adapted to solve online *convex* optimization, with the understanding that the instance received at round $t$ corresponds to the gradient of the convex function evaluated at the policy at that round.

### A.5 Six Representative General-Sum Games

In game theory, there are six representative two-player general-sum games (Robinson and Goforth, ). Firstly, consider the win-win game represented by matrices $A=\begin{pmatrix}1&4\\
1&2\end{pmatrix}$ and $B=\begin{pmatrix}1&4\\
1&2\end{pmatrix}$ for players A and B, respectively. This setup fosters a cooperative dynamic, as both players receive identical payoffs, encouraging strategies that benefit both parties equally.

In contrast, the prisoner’s dilemma, depicted by payoff matrices $A=\begin{pmatrix}1&3\\
2&4\end{pmatrix}$ and $B=\begin{pmatrix}4&3\\
2&1\end{pmatrix}$, illustrates the conflict between individual and collective rationality, where players are tempted to pursue individual gain at the collective’s expense, often resulting in suboptimal outcomes for both.

In the unfair game, represented by $A=\begin{pmatrix}2&1\\
3&4\end{pmatrix}$ and $B=\begin{pmatrix}4&3\\
1&2\end{pmatrix}$, the asymmetry in the payoff structure places one player at a disadvantage, regardless of the chosen strategy. This imbalance often reflects real-world scenarios where power or information asymmetry affects decision-making.

The cyclic game, with matrices $A=\begin{pmatrix}3&1\\
2&4\end{pmatrix}$ and $B=\begin{pmatrix}3&4\\
2&1\end{pmatrix}$, presents a scenario where no stable equilibrium exists. The best strategy for each player changes in response to the other’s actions, leading to a continuous cycle of strategy adaptation without a clear resolution.

The biased game, depicted by $A=\begin{pmatrix}3&2\\
1&4\end{pmatrix}$ and $B=\begin{pmatrix}4&2\\
1&3\end{pmatrix}$, inherently favors one player, often reflecting situations where external factors or inherent advantages influence outcomes, leading to consistently unequal payoffs.

Finally, the second-best game, with payoff matrices $A=\begin{pmatrix}1&2\\
3&4\end{pmatrix}$ and $B=\begin{pmatrix}1&4\\
3&2\end{pmatrix}$, encapsulates scenarios where players settle for less-than-optimal outcomes due to constraints like risk aversion or limited options. This often results in players choosing safer, albeit less rewarding, strategies.

Each of these games exemplifies distinct aspects of strategic decision-making and interactions. From cooperative to competitive and fair to biased scenarios, these matrices provide a rich landscape for exploring the nuances of decision-making behavior in game theory.

## Appendix B Deferred Results and Proofs in Section 3

### B.1 Ablation Study on Prompts

##### Ablation study on online learning.

To systematically understand the effects of our prompt on the final performance of the LLM agent, we create three different variants of our prompt and report the regret by using different prompts in [Figure 12](#A2.F12). Specifically, for Ablation1, we remove examples to illustrate the game rules. For Ablation2, we remove the number of iterations. For Ablation3, we incorporate some *hints* for the LLM for decision-making, including the hints to suggest it to pay attention to the loss history, to behave more greedily at the end of an episode, and also to explain the reason of its decision step-by-step. The latter hint is a popular technique in prompt engineering known as the *Chain-of-Thought* prompting (Wei et al., ). Finally, we recall that $d$ is the number of actions in all prompts.

Figure: Figure 12: Ablation study on our prompt design.
Refer to caption: /html/2403.16843/assets/x20.png

Original prompt

You are solving a decision-making problem for 25 rounds.

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy, it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components should equal 1.

After that, you will be shown the reward vector for choosing each action.

Remember that this reward vector is decided by the external system and can be potentially

different for different rounds.

It is not decided by what policies you have chosen. The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

For example, a reward vector of [0.8, 3.2] means reward for action_0 is 0.8

and the reward for action_1 is 3.2.

Then your reward for this round will be calculated according to the reward of each action and

your probability of choosing each action.

For example, if you choose the policy [0.2, 0.8] and get the reward vector [1, 2],

then your expected reward is 0.2*1 + 0.8*2=1.8

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as ’Policy: [your $d$-dimensional policy]’.

Ablation1: no examples

You are solving a decision-making problem for 25 rounds.

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy, it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components should equal 1.

After that, you will be shown the reward vector for choosing each action.

Remember that this reward vector is decided by the external system and can be potentially

different for different rounds.

It is not decided by what policies you have chosen. The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

Then your reward for this round will be calculated according to the reward of each action and

your probability of choosing each action.

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as ’Policy: [your $d$-dimensional policy]’.

Ablation2: no round information

You are solving a decision-making problem.

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy, it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components should equal 1.

After that, you will be shown the reward vector for choosing each action.

Remember that this reward vector is decided by the external system and can be potentially

different for different rounds.

It is not decided by what policies you have chosen. The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

For example, a reward vector of [0.8, 3.2] means reward for action_0 is 0.8

and the reward for action_1 is 3.2.

Then your reward for this round will be calculated according to the reward of each action and

your probability of choosing each action.

For example, if you choose the policy [0.2, 0.8] and get the reward vector [1, 2],

then your expected reward is 0.2*1 + 0.8*2=1.8

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as ’Policy: [your $d$-dimensional policy]’.

Ablation3: adding hints

You are solving a decision-making problem for 25 rounds.

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy, it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components should equal 1.

After that, you will be shown the reward vector for choosing each action.

Remember that this reward vector is decided by the external system and can be potentially

different for different rounds.

It is not decided by what policies you have chosen. The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

For example, a reward vector of [0.8, 3.2] means reward for action_0 is 0.8

and the reward for action_1 is 3.2.

Then your reward for this round will be calculated according to the reward of each action and

your probability of choosing each action.

For example, if you choose the policy [0.2, 0.8] and get the reward vector [1, 2],

then your expected reward is 0.2*1 + 0.8*2=1.8

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as ’Policy: [your $d$-dimensional policy]’.

Let’s think step by step. Explicitly examining history is important.

Please explain how you chose the policy by guessing what reward you might receive

for each action according to the history.

You should explore for first several rounds and behave greedily for later rounds, for example,

choosing one action with probability more than 0.99.

Please also explain whether you are behaving more greedily and less greedily

by explicitly considering the policy you just used for last round.

We can see in [Figure 12](#A2.F12) that the performances of LLM agents are consistent under different variants of the prompts.

##### Ablation study on repeated games.

For the game setting, we also investigate whether explicitly informing LLM agents that they are ‘‘playing a repeated matrix game with some other opponents’’ would affect the performance. Therefore, we evaluate three different prompts by informing LLM agents that they are playing a matrix game, solving multi-arm bandit, or solving general decision-making problems, in the first line of the prompt. We show the performance of such three prompts in [Figure 13](#A2.F13), where it is seen that LLM agents’ performance on repeated games is consistent among these variants of the prompts.

Figure: Figure 13: Regret of GPT-4 for repeated games under $3$ different prompt ablations. Its performance is consistent among three different prompts.
Refer to caption: /html/2403.16843/assets/x21.png

### B.2 Proof for Proposition 1

###### Proof.

Under the null hypothesis $H_{0}$, the probability $p$ that $\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t]}\right)/t-\text{Regret}_{\mathscr{A}}\left((f_{\tau})_{\tau\in[t+1]}\right)/(t+1)>0$ is less than $\frac{1}{2}$. Therefore, if we consider the event $\mathcal{E}(s,T)$, we have

$$ $\displaystyle\mathbb{P}_{H_{0}}(\mathcal{E}(s,T))=\sum_{k=s}^{T-1}p^{s}(1-p)^{T-1-s}\begin{pmatrix}T-1\\ k\end{pmatrix}\leq\frac{1}{2^{T-1}}\sum_{k=s}^{T-1}\begin{pmatrix}T-1\\ k\end{pmatrix}$ $$

since $s\geq\frac{T-1}{2}$.
∎

###### Proof.

### B.3 Results of GPT-4 Turbo

When preparing our draft, we are aware of a more advanced version of GPT-4, GPT-4 Turbo. Although GPT-4 has successfully exhibited no-regret behaviors in most problems, we also evaluate and report the performance of its more advanced version here. Specifically, we compare GPT-4 and GPT-4 Turbo for the repeated game scenario in [Figure 14](#A2.F14), where we validate that GPT-4 Turbo performs either comparably with or better than GPT-4.

Figure: Figure 14: Comparisons of GPT-4 Turbo with GPT-4 and GPT-3.5 Turbo. Due to the symmetry of agents in randomly generated games, we report the regret of one agent for ease of presentation. We can see that GPT-4 Turbo performs on par with or better than GPT-4.
Refer to caption: /html/2403.16843/assets/x22.png

### B.4 LLM Agents’ Explanation on Their Output Policies

We here present two examples of LLMs’ explanations of their output policies, showing how they *reason* in their decision-making.

Output Example 1

Looking at the history of the reward vectors, it is clear to see that the reward for choosing

action 0 is consistently higher than that for choosing action 1. This pattern is evident across

all 5 rounds of the game.

In the first round, the difference in rewards is 3.52, this remained relatively stable with

the difference in the last round being 3.52 as well. It’s indicative that action 0 consistently

brings a higher return than action 1.

Therefore, I’ll lean more towards selecting action 0 in my policy.

However, acting deterministically, that is always choosing action 0, would introduce the risk

of becoming too predictable and letting the opponent exploit this predictability

So, in order to maintain some unpredictability while still capitalizing on the higher average

return of action 0, I’ll set my policy as follows: select action 0 with a probability of 0.8

and action 1 with a probability of 0.2.

This way, the majority of the time I’m choosing the action that gives a higher reward, but I’m

also occasionally choosing the other action to keep the opponent guessing, and avoid

any predictable patterns that can be exploited

So, my policy is as follows: Policy: [0.8, 0.2]

Output Example 2

The total reward from action_1 is 130.92 and the average reward is around 10.07.

On the other hand, the total reward from action_0 is 33.418 and the average reward is around 8.354.

Although action_0 has a less average reward, its highest reward is 26.977,

which is considerably high and shows its potential.

For this round, I want to continue to explore action_0 but also significantly exploit action 1

as it reliably gives higher reward.

So, I’d like to decrease the probability of selecting action_0 compared to last time

but still keep it in the policy as a possible option.

The policy for the previous round was not provided but assuming it was more balanced,

I’m now going to lean more towards action_1.

So, the policy will be: [0.3, 0.7]. Here, I’m behaving more greedily compared to earlier rounds,

as we are choosing the action which has been giving us higher rewards with more probability.

As observed from the output, the LLM agent tends to use the history of the reward vectors by looking at the “sum/average of the accumulated rewards”, and tends to introduce randomization in decision-making, in order to be “less predictable and exploitable”. These are several key components in achieving no-regret in online learning and games (Cesa-Bianchi and Lugosi, ), explaining the empirical evidence we had in [Section 3](#S3).

## Appendix C Deferred Results and Proofs in Section 4

### C.1 Deferred Proof of Observation 1

###### Proof of 1 .

For given $(\ell_{t})_{t\in[T]}$,

$$ $\displaystyle\sum_{t=1}^{T}\int_{\pi_{t}\in\Pi}P_{\text{LLM}}(\pi_{t}\mid(\ell_{i})_{i\in[t-1]})\langle\ell_{t},\pi_{t}\rangle d\pi_{t}\leq\sum_{t=1}^{T}\int_{\pi_{t}\in\Pi}\left(P_{\text{data}}(\pi_{t}\mid(\ell_{i})_{i\in[t-1]})+\epsilon\right)\langle\ell_{t},\pi_{t}\rangle d\pi_{t}$ $$

holds, where we use the convention of $P_{\text{LLM}}(\pi_{t}\mid(\ell_{0})):=P_{\text{LLM}}(\pi_{t})$ and $P_{\text{data}}(\pi_{t}\mid(\ell_{0})):=P_{\text{data}}(\pi_{t})$. Hence,

$$ $\displaystyle\text{Regret}_{\text{LLM}}((\ell_{t})_{t\in[T]})$ $\displaystyle=\sum_{t=1}^{T}\int_{\pi_{t}\in\Pi}P_{\text{LLM}}(\pi_{t}\mid(\ell_{i})_{i\in[t-1]})\langle\ell_{t},\pi_{t}\rangle d\pi_{t}-\inf_{\pi\in\Pi}\sum_{t=1}^{T}\langle\ell_{t},\pi\rangle$ $\displaystyle\leq\sum_{t=1}^{T}\int_{\pi_{t}\in\Pi}\left(P_{\text{data}}(\pi_{t}\mid(\ell_{i})_{i\in[t-1]})+\epsilon\right)\langle\ell_{t},\pi_{t}\rangle d\pi_{t}-\inf_{\pi\in\Pi}\sum_{t=1}^{T}\langle\ell_{t},\pi\rangle$ $\displaystyle=\sum_{t=1}^{T}\int_{\pi_{t}\in\Pi}\left(P_{\text{data}}(\pi_{t}\mid(\ell_{i})_{i\in[t-1]})\right)\langle\ell_{t},\pi_{t}\rangle d\pi_{t}-\inf_{\pi\in\Pi}\sum_{t=1}^{T}\langle\ell_{t},\pi\rangle+\sum_{t=1}^{T}\int_{\pi_{t}\in\Pi}\langle\ell_{t},\epsilon\pi_{t}\rangle d\pi_{t}$ $\displaystyle\leq\text{Regret}_{\text{data}}((\ell_{t})_{t\in[T]})+\epsilon\|\ell\|_{p}\|\pi\|_{q}T$ $$

where $\frac{1}{p}+\frac{1}{q}=1$ and $p,q\geq 1$. Similarly, we can establish the lower bound for $\text{Regret}_{\text{LLM}}((\ell_{t})_{t\in[T]})$. To prove the result for the dynamic-regret case, we can simply change the term $\inf_{\pi\in\Pi}\sum_{t=1}^{T}\langle\ell_{t},\pi\rangle$ in the above derivation to $\sum_{t=1}^{T}\inf_{\pi\in\Pi}\langle\ell_{t},\pi\rangle$.
∎

###### Proof of 1 .

### C.2 Deferred Proof of Lemma 1

See [1](#Thmlemma1)

###### Proof.

Note that

$$ $\displaystyle\mathbb{P}(z{\,|\,}x_{1:N_{t}})=\int_{\ell_{1:t}}\mathbb{P}(z,\ell_{1:t}{\,|\,}x_{1:N_{t}})d{\ell_{1:t}}=\int_{\ell_{1:t}}\mathbb{P}(\ell_{1:t}{\,|\,}x_{1:N_{t}})\mathbb{P}(z{\,|\,}x_{1:N_{t}},\ell_{1:t})d\ell_{1:t}.$ $$

For $\mathbb{P}(\ell_{1:t}{\,|\,}x_{1:N_{t}})$, since we have assumed the existence of function $f$ to decode $\ell_{1:t}$ from $x_{1:N_{t}}$, it holds that

$$ $\displaystyle\mathbb{P}(\ell_{1:t}{\,|\,}x_{1:N_{t}})=\prod_{i=1}^{t}\delta\left(\ell_{i}-f(x_{N_{i-1}+1:N_{i}})\right),$ $$

where we use $\delta$ to denote the $d$-dimensional Dirac-delta function.
For $\mathbb{P}(z{\,|\,}x_{1:N_{t}},\ell_{1:t})$, by [1](#Thmassumption1), it holds that

$$ $\displaystyle\mathbb{P}(z,x_{1:N_{t}},\ell_{1:t})=\mathbb{P}(z,\ell_{1:t})\mathbb{P}(x_{1:N_{t}}{\,|\,}\ell_{1:t}),$ $$

which leads to $\mathbb{P}(x_{1:N_{t}}{\,|\,}\ell_{1:t})=\mathbb{P}(x_{1:N_{t}}{\,|\,}\ell_{1:t},z)$ by Bayes rule. This implies that the random variable $x_{1:N_{t}}$ and $z$ are independent conditioned on $\ell_{1:t}$. Therefore, it holds that $\mathbb{P}(z{\,|\,}x_{1:N_{t}},\ell_{1:t})=\mathbb{P}(z{\,|\,}\ell_{1:t})$. Finally, we can compute

$$ $\displaystyle\mathbb{P}(z{\,|\,}x_{1:N_{t}})$ $\displaystyle=\int_{\ell_{1:t}}\mathbb{P}(z,\ell_{1:t}{\,|\,}x_{1:N_{t}})d{\ell_{1:t}}=\int_{\ell_{1:t}}\prod_{i=1}^{t}\delta(\ell_{i}-f(x_{N_{i-1}+1:N_{i}}))\mathbb{P}(z{\,|\,}\ell_{1:t})d\ell_{1:t}$ $\displaystyle=\mathbb{P}\left(z{\,|\,}\left(\ell_{i}=f(x_{N_{i-1}+1:N_{i}})\right)_{i\in[t]}\right).$ $$

Based on this, we conclude that

$$ $\displaystyle\mathbb{P}(g(x_{N_{t}+1:N_{t+1}}){\,|\,}x_{1:N_{t}})$ $\displaystyle=\int_{z}\mathbb{P}(g(x_{N_{t}+1:N_{t+1}}){\,|\,}z,x_{1:N_{t}})\mathbb{P}(z{\,|\,}x_{1:N_{t}})dz$ $\displaystyle=\int_{z}\mathbb{P}(g(x_{N_{t}+1:N_{t+1}}){\,|\,}z)\mathbb{P}(z{\,|\,}\{\ell_{i}=f(x_{N_{i-1}+1:N_{i}})\}_{i\in[t]})dz$ $\displaystyle=\mathbb{P}\left(g(x_{N_{t}+1:N_{t+1}}){\,|\,}\left(\ell_{i}=f(x_{N_{i-1}+1:N_{i}})\right)_{i\in[t]}\right)$ $$

where the first equality is by the independence between $x_{N_{t}+1:N_{t+1}}$ and $x_{1:N_{t}}$ conditioned on $z$, due to [1](#Thmassumption1). Therefore, it suffices to consider the probability of $\mathbb{P}(a{\,|\,}\ell_{1:t})$ only, in order to analyze $\mathbb{P}(g(x_{N_{t}+1:N_{t+1}}){\,|\,}x_{1:N_{t}})$, where we recall the definition that $a=g(x_{N_{t}+1:N_{t+1}})$.
Since $z\sim\mathcal{N}(\boldsymbol{0}_{d},\sigma^{2}I)$, and ${\ell}_{i}\mid z\sim\mathcal{N}(z,\sigma^{2}I)$, we have

$$ $\displaystyle z\mid\ell_{1:t}\sim\mathcal{N}\left(\frac{1}{t+1}\sum_{i\in[t]}{\ell}_{i},\frac{\sigma^{2}}{t+1}I\right),$ (11) $$

by the posterior distribution of Gaussian distribution. Now we conclude that

$$ $\displaystyle\mathbb{P}(a{\,|\,}\ell_{1:t})$ $\displaystyle=\int_{z}\mathbb{P}(a{\,|\,}z,\ell_{1:t})\mathbb{P}(z{\,|\,}\ell_{1:t})dz=\int_{z}\mathbb{P}(a{\,|\,}z)\mathbb{P}(z{\,|\,}\ell_{1:t})dz$ $\displaystyle=\int_{z}\mathbbm{1}(a\in\operatorname*{arg\,min}_{a^{\prime}\in\mathcal{A}}z_{a^{\prime}})\mathbb{P}(z{\,|\,}\ell_{1:t})dz=\int_{z}\mathbbm{1}\left(a\in\operatorname*{arg\,min}_{a^{\prime}\in\mathcal{A}}\left(\frac{\sigma}{\sqrt{t+1}}\epsilon+\frac{1}{t+1}\sum_{i\in[t]}\ell_{i}\right)_{a^{\prime}}\right)\mathbb{P}(\epsilon)d\epsilon$ $\displaystyle=\int_{z}\mathbbm{1}\left(a\in\operatorname*{arg\,min}_{a^{\prime}\in\mathcal{A}}\left(\sigma\sqrt{t+1}\epsilon+\sum_{i\in[t]}\ell_{i}\right)_{a^{\prime}}\right)\mathbb{P}(\epsilon)d\epsilon=\mathbb{P}\left(a\in\operatorname*{arg\,min}_{a^{\prime}\in\mathcal{A}}\left(\sigma\sqrt{t+1}\epsilon+\sum_{i\in[t]}\ell_{i}\right)_{a^{\prime}}\right)$ $\displaystyle=P_{quantal}^{\sigma\sqrt{t+1}}(a{\,|\,}\ell_{1:t}),$ $$

where $\mathbb{P}(\epsilon)=\mathcal{N}(\boldsymbol{0}_{d},I)$. This completes the proof.
∎

###### Proof.

### C.3 Relationship between FTPL and Definition 2

###### Fact 1 .

Performing generalized quantal response of [Definition 2](#Thmdefinition2) at every iteration $t\in[T]$ w.r.t. history loss vectors $\ell_{1:t-1}$ is essentially executing an FTPL algorithm.

###### Proof.

Before we move to the proof, we will define the random variable which has distribution $P_{\text{noise}}$ as $Z_{\text{noise}}$. Note that at round $t\geq 2$ (as the policy at round $t=1$ is fixed), we have

$$ $\displaystyle P_{\text{quantal}}^{\eta_{t-1}}(a{\,|\,}\ell_{1:t-1}):=\mathbb{P}\left(a\in\operatorname*{arg\,min}_{a^{\prime}\in\mathcal{A}}~{}\left(\sum_{i=1}^{t-1}{\ell}_{i}+\eta_{t-1}\epsilon\right)(a^{\prime})\right)$ (12) $$

which is exactly the case when $\epsilon_{t}$ in [Equation 10](#A1.E10)
satisfies $\epsilon_{t}\overset{d}{=}\eta_{t-1}\epsilon$.
∎

###### Fact 1 .

###### Proof.

### C.4 Deferred Proof of Theorem 1

See [1](#Thmtheorem1)

###### Proof.

Note that

$$ $\displaystyle\max_{\left\{q_{j}\in\left\{\mathcal{V}^{j-1}\rightarrow\Delta(\mathcal{V})\right\}\right\}_{j\in[N_{t+1}]}}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log q_{j}\left(x_{j}{\,|\,}x_{1:j-1}\right)$ $\displaystyle=\max_{q\in\Delta(\mathcal{V}^{N_{t+1}})}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\log q(x_{1:N_{t+1}})$ $\displaystyle=\max_{q\in\Delta(\mathcal{V}^{N_{t+1}})}-\text{KL}(P_{t}^{text}{\,|\,}{\,|\,}q)+\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}[P_{t}^{text}(x_{1:N_{t+1}})],$ $$

where $\text{KL}(q{\,|\,}{\,|\,}p)$ denotes the Kullback–Leibler divergence between two distributions $p,q$. Now we define $\text{LLM}_{\theta}(x_{1:N_{t+1}})=\prod_{t=1}^{N_{t+1}}\text{LLM}_{\theta}(x_{j}{\,|\,}x_{1:j-1})$.
It is easy to verify that $\text{LLM}_{\theta}(x_{1:N_{t+1}})\in\Delta(\mathcal{V}^{N_{t+1}})$, i.e., it also defines a valid joint distribution over tokens. Therefore, we have

$$ $\displaystyle\max_{\theta\in\Theta}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log\text{LLM}_{\theta}\left(x_{j}{\,|\,}x_{1:j-1}\right)=\max_{\theta\in\Theta}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\log\text{LLM}_{\theta}(x_{1:N_{t+1}}).$ $$

Now, due to our assumption that

$$ $\displaystyle\max_{\theta\in\Theta}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log\text{LLM}_{\theta}\left(x_{j}{\,|\,}x_{1:j-1}\right)=\max_{\left\{q_{j}\in\left\{\mathcal{V}^{j-1}\rightarrow\Delta(\mathcal{V})\right\}\right\}_{j\in[N_{t+1}]}}\mathbb{E}_{x_{1:N_{t+1}}\sim P^{text}_{t}}\sum_{j=1}^{N_{t+1}}\log q_{j}\left(x_{j}{\,|\,}x_{1:j-1}\right),$ $$

we conclude that

$$ $\displaystyle\min_{\theta\in\Theta}\text{KL}(P_{t}^{text}{\,|\,}{\,|\,}\text{LLM}_{\theta})=\min_{q\in\Delta(\mathcal{V}^{N_{t+1}})}\text{KL}(P_{t}^{text}{\,|\,}{\,|\,}q)=0,$ $$

which implies that $\text{LLM}_{\theta^{\star}}=P_{t}^{text}$. Correspondingly, if we define $\text{LLM}_{\theta^{\star}}(x_{N_{t}+1:N_{t+1}}{\,|\,}x_{1:N_{t}})$ to be the distribution induced by the joint distribution $\text{LLM}_{\theta^{\star}}(x_{1:N_{t+1}})$, it holds that

$$ $\displaystyle\text{LLM}_{\theta^{\star}}(x_{N_{t}+1:N_{t+1}}{\,|\,}x_{1:N_{t}})=\mathbb{P}(x_{N_{t}+1:N_{t+1}}{\,|\,}x_{1:N_{t}}).$ $$

In other words, intuitively, $\text{LLM}_{\theta^{\star}}$ has learned the corresponding *pre-training* distribution perfectly. Note that this has been a common assumption in the Bayesian perspective of ICL (Xie et al., ; Lee et al., ; Zhang et al., ). Therefore, to analyze the actions taken by $\text{LLM}_{\theta^{\star}}$, it suffices to consider $\mathbb{P}(g(x_{N_{t}+1:N_{t+1}}){\,|\,}x_{1:N_{t}})$, which is equal to $P_{quantal}^{\sigma\sqrt{t+1}}\left(g(x_{N_{t}+1:N_{t+1}}){\,\big{|}\,}\left\{f(x_{N_{i-1}+1:N_{i}})\right\}_{i\in[t]}\right)$ by [Lemma 1](#Thmlemma1). Therefore, we proved that $\text{LLM}_{\theta^{\star}}$ is essentially mimicking the well-known no-regret algorithm, FTPL with perturbation distribution as $\mathcal{N}(\boldsymbol{0}_{d},\sigma^{2}tI)$ for round $t\in[T]$, according to [Equation 12](#A3.E12) of [1](#Thmfact1), for which we can establish the corresponding regret guarantee for each case:

(1) Combining the above result with [Lemma 2](#Thmlemma2), we can derive the regret bound for online learning with full-information feedback.

(2) Combining the above result with [Lemma 2](#Thmlemma2) and [Lemma 4](#Thmlemma4), we get that

$$ $\displaystyle\text{D-Regret}_{\text{LLM}{{}_{\theta^{\star}}}}(({\ell}_{i})_{i\in[T]})\leq\min_{\Delta_{T}\in[T]}\frac{2T}{\Delta_{T}}C\sqrt{\Delta_{T}\log d}+2\Delta_{T}V_{T},$ $$

for some constant $C$. We firstly consider the following problem

$$ $\displaystyle\min_{u>0}\frac{2T}{u}C\sqrt{u\log d}+2uV_{T},$ $$

where the optimal solution is $u^{\star}=\left(\frac{C^{2}T^{2}\log d}{4V_{T}^{2}}\right)^{1/3}$. Therefore, if we have $u^{\star}\in[1,T]$, we can choose $\Delta_{T}=\lceil u^{\star}\rceil$, which results in a regret bound of

$$ $\displaystyle\text{D-Regret}_{\text{LLM}{{}_{\theta^{\star}}}}(({\ell}_{i})_{i\in[T]})\leq\frac{2T}{\sqrt{u^{\star}}}C\sqrt{\log d}+4u^{\star}V_{T}=\mathcal{O}\left((\log d\ V_{T})^{1/3}T^{2/3}\right).$ $$

Now we check the conditions for $u^{\star}\in[1,T]$. It is direct to see that since $V_{T}\leq T$, $u^{\star}\geq 1$ holds as long as $d$ is sufficiently large. To ensure $u^{\star}\leq T$, we get the condition $V_{T}\geq C\sqrt{\frac{\log d}{4T}}$, which holds as long as $T$ is large enough.

(3) Combining the above result with [Lemma 3](#Thmlemma3), we can prove a regret guarantee for online learning with bandit feedback.

(4) Combining this result with [Lemma 3](#Thmlemma3) and [Lemma 4](#Thmlemma4), it holds that

$$ $\displaystyle\mathbb{E}[\text{D-Regret}_{\text{LLM}{{}_{\theta^{\star}}}}(({\ell}_{i})_{i\in[T]})]\leq\min_{\Delta_{T}\in[T]}\frac{2T}{\Delta_{T}}C(\log d)^{\frac{1}{2}}d\Delta_{T}^{\frac{1}{2}+\frac{1}{\log T}}\log\Delta_{T}+2\Delta_{T}V_{T},$ $$

for some constant $C$. By adopting a similar analysis as that of (2), we choose $u^{\star}=\left(\frac{C^{\prime}T^{2}d^{2}}{V_{T}^{2}}\right)^{1/3}$ for some constant $C^{\prime}$. If $u^{\star}\in[1,T]$, we choose $\Delta_{T}=\lceil u^{\star}\rceil$ and derive the following regret:

$$ $\displaystyle\mathbb{E}[\text{D-Regret}_{\text{LLM}{{}_{\theta^{\star}}}}(({\ell}_{i})_{i\in[T]})]\leq\mathcal{O}\left((T^{2}d^{2}V_{T})^{1/3}(\log d)^{1/2}T^{1/\log T}\log T\right).$ $$

Now we check the condition of $u^{\star}\in[1,T]$. Note that since $V_{T}\leq T$, $u^{\star}\geq 1$ holds as long as $d$ is sufficiently large. For $u^{\star}\leq T$, we have $V_{T}\geq\sqrt{\frac{C^{\prime}d^{2}}{T}}$, which holds as long as $T$ is large enough.

Now, we present [Lemma 2](#Thmlemma2) - [Lemma 4](#Thmlemma4). Before proceeding, we assume $\|\ell_{t}\|_{\infty}\leq B=1$ for simplicity of presentations hereafter. The results and proof are not affected by the constant bound $B$.

###### Lemma 2 (Regret guarantee of FTPL with full-information feedback) .

Suppose the noise distribution of FTPL satisfies that $\epsilon_{t}\sim\mathcal{N}(\boldsymbol{0}_{d},\zeta_{t}^{2}I)$ in [Equation 10](#A1.E10) and $\zeta_{t}=\sigma\sqrt{t}$, then for online learning with full-information feedback,

$$ $\displaystyle\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})\leq 4\left(\sigma+\frac{1}{\sigma}\right)\sqrt{T\log d}=\mathcal{O}(\sqrt{T\log d}).$ $$

###### Proof.

By Theorem 8 of Abernethy et al. (), we have

$$ $\displaystyle\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})\leq\sqrt{2\log d}\left(\eta_{T}+\sum_{t=1}^{T}\frac{1}{\eta_{t}}\|\ell_{t}\|_{\infty}^{2}\right).$ $$

Therefore, plugging $\zeta_{t}=\sigma\sqrt{t}$ and $\|\ell_{t}\|_{\infty}^{2}\leq 1$ provides

$$ $\displaystyle\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})\leq\sqrt{2\log d}\left(\sigma\sqrt{T}+\sum_{t=1}^{T}\frac{1}{\sigma\sqrt{t}}\right)\leq 4\left(\sigma+\frac{1}{\sigma}\right)\sqrt{T\log d},$ $$

completing the proof.
∎

###### Lemma 3 (Regret guarantee of FTPL with bandit feedback) .

Suppose the noise distribution of FTPL satisfies that $\epsilon_{t}\sim\mathcal{N}(\boldsymbol{0}_{d},\zeta_{t}^{2}I)$ in [Equation 10](#A1.E10) and $\zeta_{t}=\sigma\sqrt{t}$, then for online learning with bandit feedback,

$$ $\displaystyle\mathbb{E}[\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})]\leq\mathcal{O}((\log d)^{\frac{1}{2}}dT^{\frac{1}{2}+\frac{1}{\log T}}\log T).$ $$

###### Proof.

The proof of the bandit problem is more complex. We first define the following notation. We denote $G_{t}=\sum_{t^{\prime}=1}^{t}-\ell_{t^{\prime}}$, $\widehat{G}_{t}=\sum_{t^{\prime}=1}^{t}-\widehat{\ell}_{t^{\prime}}$, $\Phi(G)=\max_{\pi}\langle\pi,G\rangle$, $\Phi_{t}(G)=\mathbb{E}_{\epsilon\sim\mathcal{N}(\boldsymbol{0}_{d},I)}\Phi(G+\zeta_{t}\epsilon)$, and $D_{\Phi_{t}}$ to be the Bregman divergence with respect to $\Phi_{t}$, where we recall the construction of the empirical estimator $\widehat{\ell}_{t^{\prime}}$ of $\ell_{t^{\prime}}$ in [Section 3.2](#S3.SS2). By Li and Tewari (), $\pi_{t}=\nabla\Phi_{t}(\widehat{G}_{t})$.
Now due to the convexity of $\Phi$,

$$ $\displaystyle\Phi(G_{T})=\Phi(\mathbb{E}[\widehat{G}_{T}])\leq\mathbb{E}[\Phi(\widehat{G}_{T})].$ $$

Therefore,

$$ $\displaystyle\mathbb{E}[\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})]$ $\displaystyle=\Phi(G_{T})-\mathbb{E}\left[\sum_{t=1}^{T}\langle\pi_{t},-\widehat{\ell}_{t}\rangle\right]\leq\mathbb{E}\left[\Phi(\widehat{G}_{T})-\sum_{t=1}^{T}\langle\pi_{t},-\widehat{\ell}_{t}\rangle\right].$ $$

By recalling the definition of the Bregman divergence, we have

$$ $\displaystyle-\sum_{t=1}^{T}\langle\pi_{t},-\widehat{\ell}_{t}\rangle$ $\displaystyle=-\sum_{t=1}^{T}\langle\nabla\Phi_{t}(\widehat{G}_{t}),-\widehat{\ell}_{t}\rangle=-\sum_{t=1}^{T}\langle\nabla\Phi_{t}(\widehat{G}_{t}),\widehat{G}_{t}-\widehat{G}_{t-1}\rangle$ $\displaystyle=\sum_{t=1}^{T}D_{\Phi_{t}}(\widehat{G}_{t},\widehat{G}_{t-1})+\Phi_{t}(\widehat{G}_{t-1})-\Phi_{t}(\widehat{G}_{t}).$ $$

Therefore,

$$ $\displaystyle\mathbb{E}\left[\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})\right]$ $\displaystyle\leq\underbrace{\mathbb{E}\left[\sum_{t=1}^{T}D_{\Phi_{t}}(\widehat{G}_{t},\widehat{G}_{t-1})\right]}_{(i)}+\underbrace{\mathbb{E}\left[\sum_{t=1}^{T}\Phi_{t}(\widehat{G}_{t-1})-\Phi_{t-1}(\widehat{G}_{t-1})\right]}_{(ii)}+\underbrace{\mathbb{E}\left[\Phi(\widehat{G}_{T})-\Phi_{T}(\widehat{G}_{T})\right]}_{(iii)}.$ $$

$(iii)\leq 0$ due to the convexity of $\Phi$. For $(ii)$, we use Lemma 10 of Abernethy et al. () to obtain

$$ $\mathbb{E}\left[\sum_{t=1}^{T}\Phi_{t}(\widehat{G}_{t-1})-\Phi_{t-1}(\widehat{G}_{t-1})\right]\leq\zeta_{T}\mathbb{E}_{\epsilon}[\Phi(\epsilon)]\leq\mathcal{O}(\sqrt{2T\log d}).$ $$

For $(i)$, by Theorem 8 of Li and Tewari (), for any $\alpha\in(0,1)$, the following holds:

$$ $\displaystyle\mathbb{E}\left[\sum_{t=1}^{T}D_{\Phi_{t}}(\widehat{G}_{t},\widehat{G}_{t-1})\right]\leq\sum_{t=1}^{T}\zeta_{t}^{\alpha-1}\frac{4d}{\alpha(1-\alpha)}\leq\frac{4d}{\alpha(1-\alpha)}\mathcal{O}(T^{\frac{1+\alpha}{2}}).$ $$

By tuning $\alpha=\frac{2}{\log T}$, we proved that $\mathbb{E}[\text{Regret}_{\text{FTPL}}(({\ell}_{i})_{i\in[T]})]\leq\mathcal{O}((\log d)^{\frac{1}{2}}dT^{\frac{1}{2}+\frac{1}{\log T}}\log T)$.
∎

###### Lemma 4 .

Denote the variation of loss vectors as $L_{T}=\sum_{t=1}^{T-1}\|\ell_{t+1}-\ell_{t}\|_{\infty}$. Suppose there exists an algorithm $\mathscr{A}$ for online learning with full-information feedback with regret guarantee that $\text{Regret}_{\mathscr{A}}((\ell_{i})_{i\in[T]})\leq f(T,d)$ for some function $f$, where $T$ denotes the horizon and $d$ denotes the policy dimension. Then, there exists another algorithm $\mathscr{A}^{\prime}$ that can achieve

$$ $\displaystyle\text{D-Regret}_{\mathscr{A}^{\prime}}(({\ell}_{i})_{i\in[T]})\leq\min_{\Delta_{T}\in[T]}\left(\frac{T}{\Delta_{T}}+1\right)f(\Delta_{T},d)+2\Delta_{T}L_{T}.$ $$

Similarly, suppose there exists an algorithm $\mathscr{B}$ for online learning with bandit feedback with regret guarantee that $\mathbb{E}\left[\text{Regret}_{\mathscr{B}}((\ell_{i})_{i\in[T]})\right]\leq g(T,d)$ for some function $g$; then there exists another algorithm $\mathscr{B}^{\prime}$ that can achieve

$$ $\displaystyle\mathbb{E}[\text{D-Regret}_{\mathscr{B}^{\prime}}(({\ell}_{i})_{i\in[T]})]\leq\min_{\Delta_{T}\in[T]}\left(\frac{T}{\Delta_{T}}+1\right)g(\Delta_{T},d)+2\Delta_{T}L_{T}.$ $$

###### Proof.

We denote $\mathscr{A}^{\prime}$ as the algorithm that restarts $\mathscr{A}$ every $\Delta_{T}$ iterations. We break the time index $[T]$ into $m$ batches ${\mathcal{T}}_{1:m}$ of size $\Delta_{T}$ (except for, possibly the last batch). Denote $\ell_{i}^{\star}:=\min_{j\in[d]}\ell_{ij}$. By Equation (6) of Besbes et al. (), it holds that for each $k\in[m]$

$$ $\displaystyle\min_{j\in[d]}\left(\sum_{t\in{\mathcal{T}}_{k}}\ell_{t}\right)_{j}-\sum_{t\in{\mathcal{T}}_{k}}\ell^{\star}_{t}\leq 2\Delta_{T}L_{k},$ $$

where we define $L_{k}=\sum_{t\in{\mathcal{T}}_{k}}\|\ell_{t+1}-\ell_{t}\|_{\infty}$.
Therefore, we have

$$ $\displaystyle\text{D-Regret}_{\mathscr{A}^{\prime}}(({\ell}_{i})_{i\in[T]})$ $\displaystyle\leq\min_{j\in[d]}\left(\sum_{t\in[T]}\ell_{t}\right)_{j}-\sum_{t\in[T]}\ell^{\star}_{t}+\sum_{k\in[m]}\text{Regret}_{\mathscr{A}}((\ell_{i})_{i\in[{\mathcal{T}}_{k}]})$ (13) $\displaystyle\leq 2\Delta_{T}(\sum_{k\in[m]}L_{k})+(T/\Delta_{T}+1)g(\Delta_{T},d).$ $$

By Equation (4) of Besbes et al. () that $\sum_{k\in[m]}L_{k}\leq L_{T}$ and this inequality holds for any $\Delta_{T}\in[T]$, we proved $\text{D-Regret}_{\mathscr{A}^{\prime}}(({\ell}_{i})_{i\in[T]})\leq\min_{\Delta_{T}\in[T]}\left(\frac{T}{\Delta_{T}}+1\right)f(\Delta_{T},d)+2\Delta_{T}L_{T}$.

Similarly, if we take the expectation for [Equation 13](#A3.E13), it holds that

$$ $\displaystyle\mathbb{E}[\text{D-Regret}_{\mathscr{B}^{\prime}}(({\ell}_{i})_{i\in[T]})]$ $\displaystyle\leq\min_{j\in[d]}\left(\sum_{t\in[T]}\ell_{t}\right)_{j}-\sum_{t\in[T]}\ell^{\star}_{t}+\sum_{k\in[m]}\mathbb{E}[\text{Regret}_{\mathscr{B}}((\ell_{i})_{i\in[{\mathcal{T}}_{k}]})]$ $\displaystyle\leq\min_{\Delta_{T}\in[T]}\left(\frac{T}{\Delta_{T}}+1\right)g(\Delta_{T},d)+2\Delta_{T}L_{T},$ $$

thus completing the proof.
∎

Combining the results above completes the proof for
[Theorem 1](#Thmtheorem1).
∎

###### Proof.

###### Lemma 2 (Regret guarantee of FTPL with full-information feedback) .

###### Proof.

###### Lemma 3 (Regret guarantee of FTPL with bandit feedback) .

###### Proof.

###### Lemma 4 .

###### Proof.

### C.5 Extending Theorem 1 with Relaxed Assumptions

#### C.5.1 Relaxation under More General Data Distributions

We first remark on the possibility of relaxing the Gaussian assumptions on the data distributions.

###### Remark 3 (Relaxing the Gaussian distribution assumption) .

In the proof of [Lemma 1](#Thmlemma1), to obtain the result that the action is a quantal response w.r.t. $\ell_{1:T}$, one does not necessarily require *both* the prior distribution of $z$ and the conditional distribution of
${\ell}_{i}$ given $z$ to be Gaussian. Instead, for *any* joint distribution $\mathbb{P}(z,\ell_{1:T})$, as long as its posterior distribution satisfies [Equation 11](#A3.E11), it would suffice. It is a combined effect of both the prior and the conditional distributions.

More formally, we can extend [Theorem 1](#Thmtheorem1) to the case with a much more general prior task distribution than the Gaussian one, where the key is that [Equation 11](#A3.E11) only needs to hold approximately.

###### Theorem 5 .

In [Theorem 1](#Thmtheorem1), we can relax the assumption on $\mathbb{P}(z)$ to one where we only require $\mathbb{P}(z)$ to be i.i.d for each coordinate of $z$ and $0<\mathbb{P}(z_{j})<\infty$, $|\nabla\mathbb{P}(z_{j})|<\infty$ for any $j\in[d]$, $z_{j}\in\mathbb{R}$, and the bounds for (1) and (2) of [Theorem 1](#Thmtheorem1) still hold, with only a degradation of $\mathcal{O}(d^{2}\log T)$.

The key idea of the proof is that when $t$ is large enough, the prior distribution does not affect the posterior distribution, which is also referred to as the *Bernstein–von Mises theorem* (Van der Vaart, ).

###### Proof.

Since we extend [Theorem 1](#Thmtheorem1) to settings with general task prior distribution only requiring the coordinates to be i.i.d, from now on, we consider the $j$-th coordinate only. To begin with, fix $t\in[T]$, we define the log-likelihood of the posterior as

$$ $\displaystyle L_{t}(z_{j}):=\log\prod_{i=1}^{t}\frac{1}{\sigma^{d}(2\pi)^{d/2}}e^{-\frac{1}{2\sigma^{2}}(\ell_{ij}-z_{j})^{2}}=-t\log\sigma-\frac{t}{2}\log 2\pi-\sum_{i=1}^{t}\frac{1}{2\sigma^{2}}(\ell_{ij}-z_{j})^{2}.$ $$

Then, the MLE estimator $\widehat{z}_{j,t}$ is defined as

$$ $\displaystyle\widehat{z}_{j,t}:=\arg\max_{z_{j}\in\mathbb{R}}L_{t}(z_{j})=\frac{1}{t}\sum_{i=1}^{t}\ell_{ij}.$ $$

We also define $\widehat{J}_{t}:\mathbb{R}\to\mathbb{R}$ as:

$$ $\displaystyle\widehat{J}_{t}(z_{j}):=-\frac{\nabla^{2}L_{t}(z_{j})}{t}=\frac{1}{\sigma^{2}}.$ $$

For Assumption 1 of Kasprzak et al. () to hold, any $\delta>0$, $M_{2}>0$ suffices.

For Assumption 2 of Kasprzak et al. () to hold, we can choose $\widehat{M}_{1}=\max_{z_{j}\in[-\delta,1+\delta]}\frac{1}{\mathbb{P}(z_{j})}$

For Assumption 7 of Kasprzak et al. () to hold, we choose $\delta$ to be $\sigma$.

For Assumption 8 of Kasprzak et al. () to hold, one can choose $M_{2}=\frac{\sigma}{2}$.

For Assumption 9 of Kasprzak et al. () to hold, we have

$$ $\displaystyle\kappa\leq-\sup_{(z_{j}-\widehat{z}_{j})^{2}\geq\delta}\frac{L_{t}(z_{j})-L_{t}(\widehat{z}_{j,t})}{t}=-\frac{1}{2\sigma^{2}t}\sup_{(z_{j}-\widehat{z}_{j,t})^{2}\geq\delta}\sum_{i=1}^{t}(\ell_{ij}-\widehat{z}_{j,t})^{2}-(\ell_{ij}-z_{j})^{2}=\frac{1}{4\sigma}.$ $$

For Assumption 10 of Kasprzak et al. () to hold, we choose $M_{1}=\sup_{z_{j}\in[-\delta,1+\delta]}\left|\frac{\nabla\mathbb{P}(z_{j})}{\mathbb{P}(z_{j})}\right|$, $\widetilde{M}_{1}=\sup_{z_{j}\in[-\delta,1+\delta]}|\mathbb{P}(z_{j})|$ since we have assumed that $0<\mathbb{P}(z_{j})<\infty$, $|\nabla\mathbb{P}(z_{j})|<\infty$.

By Theorem 6.1 of Kasprzak et al. (), we have

$$ $\displaystyle\int_{z_{j}}|$ $\displaystyle\mathbb{P}(z_{j}/\sqrt{t}+\widehat{z}_{j}{\,|\,}(\ell_{ij})_{i\in[t]})-Ce^{-\frac{1}{2\sigma^{2}}z_{j}^{2}}|dz_{j}$ $\displaystyle=\sqrt{t}\int_{z_{j}}|\mathbb{P}(z_{j}{\,|\,}(\ell_{ij})_{i\in[t]})-\mathcal{N}(\widehat{z}_{j},\frac{\sigma^{2}}{t})|dz_{j}\leq D_{1}t^{-1/2}+D_{2}t^{1/2}e^{-t\kappa}+2\widehat{\mathcal{D}}(t,\delta),$ $$

where $C$ is the normalization constant and

$$ $\displaystyle D_{1}$ $\displaystyle=\frac{\sqrt{\widetilde{M}_{1}\widehat{M}_{1}}}{\sigma}\left(\frac{\sqrt{3}\sigma^{2}}{2\left(1-\sqrt{\widehat{\mathcal{D}}(t,\delta)}\right)}M_{2}+M_{1}\right)$ $\displaystyle D_{2}$ $\displaystyle=\frac{2\widehat{M_{1}}\widehat{J}_{t}^{p}(\widehat{z}_{j},\delta)}{(2\pi)^{1/2}(1-\widehat{\mathcal{D}}^{p}(t,\delta))}$ $\displaystyle\widehat{\mathcal{D}}(t,\delta)$ $\displaystyle=e^{-\frac{1}{2}(\sqrt{t}-1)^{2}}$ $\displaystyle\widehat{J}_{t}^{p}(\widehat{z}_{j},\delta)$ $\displaystyle=\frac{1}{\sigma^{2}}+\frac{\delta M_{2}}{3}.$ $$

Therefore, we conclude that the TV distance between $z$ (conditioned on $(\ell_{i})_{i\in[t]}$) and $\mathcal{N}\left(\widehat{z},\frac{\sigma^{2}}{t}\right)$ satisfies that

$$ $\displaystyle\int_{z}\bigg{|}\mathbb{P}(z{\,|\,}(\ell_{i})_{i\in[t]})-\mathcal{N}\left(\widehat{z},\frac{\sigma^{2}}{t}\right)\bigg{|}dz\leq\sum_{j=1}^{d}\int_{z_{j}}\bigg{|}\mathbb{P}(z_{j}{\,|\,}(\ell_{ij})_{i\in[t]})-\mathcal{N}\left(\widehat{z}_{j},\frac{\sigma^{2}}{t}\right)\bigg{|}dz_{j}\leq\mathcal{O}(d/t),$ $$

due to the independence of $(z_{j})_{j\in[d]}$ conditioned on $\ell_{1:t}$. Now we denote algorithm $\widehat{\text{FTPL}}$ to be the FTPL algorithm w.r.t. the noise distribution $\mathbb{P}(z{\,|\,}(\ell_{i})_{i\in[t]})$, and FTPL to be the algorithm w.r.t. the noise distribution $\mathcal{N}(\widehat{z},\frac{\sigma^{2}}{t})$.
Therefore, we have

$$ $\displaystyle\big{|}\text{Regret}_{\text{FTPL}}((\ell)_{i\in[T]})-\text{Regret}_{\widehat{\text{FTPL}}}((\ell)_{i\in[T]})\big{|}\leq\sum_{t=1}^{T}d\|\pi_{t}-\widehat{\pi}_{t}\|_{\infty}$ $\displaystyle\quad\leq d\sum_{t=1}^{T}\int_{z}\big{|}\mathbb{P}(z{\,|\,}(\ell_{i})_{i\in[t]})-\mathcal{N}(\widehat{z},\frac{\sigma^{2}}{t})\big{|}dz=\mathcal{O}(d^{2}\log T).$ $$

In other words, using $\mathbb{P}(z{\,|\,}(\ell_{i})_{i\in[t]})$ as the noise distribution only increases the regret by $\mathcal{O}(d^{2}\log T)$. Similarly, it is easy to see that

$$ $\displaystyle\big{|}\text{D-Regret}_{\text{FTPL}}((\ell)_{i\in[T]})-\text{D-Regret}_{\widehat{\text{FTPL}}}((\ell)_{i\in[T]})\big{|}\leq\mathcal{O}(d^{2}\log T),$ $$

which completes the proof.
∎

###### Remark 3 (Relaxing the Gaussian distribution assumption) .

###### Theorem 5 .

###### Proof.

#### C.5.2 Relaxation under Decision-Irrelevant Pre-Training Data

We then remark on the possible relaxation when the training data may not all come from decision-making tasks.

###### Remark 4 (Pre-training with relaxed data assumptions) .

Note that the pre-training (text) data are so far assumed to be related to *decision-making* problems (though not necessarily *sequential ones*), see [1](#Thmassumption1) and [Example 1](#Thmexample1) for instance. It can also be generalized to the text datasets involving *Question-Answering* (Q-A), a typical task in natural language processing, where the *true/fact* answer, sampled answers from different human users (with possibly wrong or biased answers), correspond to the latent $z$ (and associated maximizer $a$) and $\ell_{1:t}$, respectively. Moreover, in practice, the pre-training data may also involve *non-decision-making/Q-A* texts, given the diversity of the datasets. For such scenarios, we will make the assumptions on the data distribution *conditioned on the prompt for decision-making*. Specifically, when interacting with the LLM, human users will provide prompts (see e.g., our [Figure 3](#S3.F3)), to induce it to make decisions. This will query the *conditional* distribution of

$$ $\mathbb{P}\left(g(x_{N_{t}+1:N_{t+1}}){\,\big{|}\,}x_{1:N_{t}},\text{decision-making prompt}\right)$ $$

to generate the control action. Correspondingly, [1](#Thmassumption1) will thus only need to be made on

$$ $\mathbb{P}\left(z,\ell_{1:t},x_{1:N_{t+1}},\text{decision-making prompt}\right),$ $$

while we do not need to make such assumptions on other prompts, e.g., corpora that are not related to decision-making.

###### Remark 4 (Pre-training with relaxed data assumptions) .

## Appendix D Deferred Results and Proofs in Section 5

### D.1 Basic Lemmas

###### Lemma 5 (Double iterated limit) .

For a sequence $(a_{mn})_{m,n\in\mathbb{N}^{+}}$, suppose that $\lim_{m,n\to\infty}a_{mn}=L$. Then the following are equivalent:

- •
For each $m$, $\lim_{n\to\infty}a_{mn}$ exists;
- •
$\lim_{m\to\infty}\lim_{n\to\infty}a_{mn}=L$.

###### Lemma 6 (Hoeffding’s inequality) .

Let $X_{1},X_{2},\ldots,X_{n}$ be independent random variables bounded by the intervals $[a_{i},b_{i}]$, respectively. Define $\bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_{i}$ and let $\mu=\mathbb{E}[\bar{X}]$ be the expected value of $\bar{X}$. Then, for any $t>0$,

$$ $\mathbb{P}(|\bar{X}-\mu|\geq t)\leq 2\exp\left(-\frac{2n^{2}t^{2}}{\sum_{i=1}^{n}(b_{i}-a_{i})^{2}}\right).$ $$

###### Lemma 7 (Uniform convergence $\Longrightarrow$ Interchanging limit and infimum) .

If $(f_{n}:X\to\mathbb{R})_{n\in\mathbb{N}^{+}}$ is a sequence of continuous functions that uniformly converge to a function $f:X\to\mathbb{R}$ on the domain $X$, then $\lim_{n\to\infty}\inf_{x\in X}f_{n}(x)=\inf_{x\in X}f(x)$ holds.

###### Lemma 5 (Double iterated limit) .

###### Lemma 6 (Hoeffding’s inequality) .

###### Lemma 7 (Uniform convergence ⟹ ⟹ \Longrightarrow Interchanging limit and infimum) .

### D.2 Deferred Proof for the Arguments in Section 5.1

In this section, we prove some properties of $\mathcal{L}(\theta,k,N)$ under certain regularity conditions of $f,h$. Throughout this subsection, we will assume the following condition holds.

###### Condition 1 .

For $h:\mathbb{R}\to\mathbb{R}^{+}$ and $f:\mathbb{R}\times\mathbb{N}^{+}\to\mathbb{R}^{+}$, suppose $h(\cdot)$ and $f(\cdot,k)$ are both continuous and non-decreasing functions for any $k\in\mathbb{N}^{+}$. The derivative $h^{\prime}:\mathbb{R}\to\mathbb{R}$ is also a continuous function. Moreover, $f$ satisfies that $\log f(R_{1},k_{1})-\log f(R_{1},k_{2})\geq\log f(R_{2},k_{1})-\log f(R_{2},k_{2})$ for $R_{1}\geq R_{2}$ and $k_{1}\geq k_{2}$, i.e., $\log f$ is supermodular. Lastly, $f$ is a function such that $\lim_{k\to\infty}\frac{f(R_{1},k)}{f(R_{2},k)}=\infty\cdot\mathbbm{1}(R_{1}>R_{2})+\mathbbm{1}(R_{1}=R_{2})$, with the convention of $\infty\cdot 0=0$. Lastly, $(\ell_{t}^{(j)})_{t\in[T],j\in[N]}$ are continuous random variables supported on $[-B,B]^{T\times N}$.

###### Claim 1 (Iterated limit of $\mathcal{L}(\theta,k,N)$ is the same as double limit of $\mathcal{L}(\theta,k,N)$ ) .

It holds that:

$$ $\displaystyle\lim_{N\to\infty}\lim_{k\to\infty}\mathcal{L}(\theta,k,N)=\lim_{N,k\to\infty}\mathcal{L}(\theta,k,N)=\lim_{k\to\infty}\lim_{N\to\infty}\mathcal{L}(\theta,k,N)=h\left(\max_{\ell_{1},\dots,\ell_{T}}\emph{Regret}_{\emph{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right).$ $$

###### Proof.

Step 1. Proving $\lim_{N\to\infty}\lim_{k\to\infty}\mathcal{L}(\theta,k,N)=h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)$.

Firstly, as both $h$ and $f$ are non-negative ([1](#Thmcondition1)), and $\mathbb{E}_{(\ell_{t}^{(j)})_{t\in[T],j\in[N]}}\left[h(\max_{j\in[N]}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))\right]$ exists, we have by dominated convergence theorem that

$$ $\displaystyle\lim_{k\to\infty}\mathcal{L}(\theta,k,N)$ $\displaystyle=\mathbb{E}\lim_{k\to\infty}\left[\frac{\sum_{j\in[N]}h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}{\sum_{j\in[N]}f(R_{\text{LLM}_{\theta}}((\ell_{i}^{(j)})_{t\in[T]}),k)}\right]$ $\displaystyle=\mathbb{E}_{(\ell_{t}^{(j)})_{t\in[T],j\in[N]}}\left[h(\max_{j\in[N]}R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))\right]$ $$

where $R_{\text{LLM}_{\theta}}$ denotes an abbreviation of $\text{Regret}_{\text{LLM}_{\theta}}$.
By (Ahsanullah et al., , Chapter 11),
we have $h(\max_{j\in[N]}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))\overset{p}{\to}h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$ when $N\to\infty$. Hence, we have
$\lim_{N\to\infty}\lim_{k\to\infty}\mathcal{L}(\theta,k,N)=h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$ holds.

Step 2. Proving $\lim_{N,k\to\infty}\mathcal{L}(\theta,k,N)=h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$.

Now, we will calculate $\lim_{N,k\to\infty}\mathcal{L}(\theta,k,N)$.

###### Lemma 8 .

For any $0<\epsilon<1$, it follows that

$$ $\lim_{N,k\to\infty}\frac{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})>1-\epsilon/2)}=0$ $$

and

$$ $\lim_{N,k\to\infty}\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon/2)}=0$ $$

hold with probability $1$, where $X_{i}$’s are i.i.d. random variables, $\operatorname{esssup}(H(X_{i}))=1$, and $H:\mathbb{R}\to\mathbb{R}^{+}$ is a continuous non-decreasing function.

###### Proof of Lemma 8 .

Since $f(\cdot,k),H$ are non-negative and non-decreasing functions, we have

$$ $\displaystyle\frac{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})>1-\epsilon/2)}\leq\frac{(1-\epsilon)f(H^{-1}(1-\epsilon),k)|\{i\in[N]\mid(H(X_{i})<1-\epsilon)\}|}{(1-\epsilon/2)f(H^{-1}(1-\epsilon/2),k)|\{i\in[N]\mid(H(X_{i})>1-\epsilon/2)\}|}$ $$

and we know that

$$ $\displaystyle\frac{|\{i\in[N]\mid(H(X_{i})<1-\epsilon)\}|}{|\{i\in[N]\mid(H(X_{i})>1-\epsilon/2)\}|}\overset{a.s.}{\to}\frac{F(1-\epsilon)}{1-F(1-\epsilon/2)}$ $$

as $N\to\infty$, where $F$ is the cumulative distribution function of random variable $H(X)$. Therefore, we have

$$ $\displaystyle 0\leq\lim_{N,k\to\infty}\frac{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})>1-\epsilon/2)}$ $\displaystyle\leq\lim_{N,k\to\infty}\frac{(1-\epsilon)f(H^{-1}(1-\epsilon),k))|\{i\in[N]\mid(H(X_{i})<1-\epsilon)\}|}{(1-\epsilon/2)f(H^{-1}(1-\epsilon/2),k))|\{i\in[N]\mid(H(X_{i})>1-\epsilon/2)\}|}$ $\displaystyle\underset{a.s.}{\leq}\lim_{N,k\to\infty}\frac{(1-\epsilon)f(H^{-1}(1-\epsilon),k))}{(1-\epsilon/2)f(H^{-1}(1-\epsilon/2),k))}\frac{F(1-\epsilon)}{1-F(1-\epsilon/2)}=0.$ $$

By a similar argument, we have

$$ $\lim_{N,k\to\infty}\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon/2)}=0$ $$

with probability 1.
∎

One key idea in the proof above is the use of some *truncation* level $\epsilon$ for $H(X)$ with $\text{esssup}(H(X))=1$.
By [Lemma 8](#Thmlemma8), we have

$$ $\lim_{N,k\to\infty}\frac{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})\mathbbm{1}(H(X_{i})>1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)H(X_{i})}=\lim_{N,k\to\infty}\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)}=1,$ $$

since

$$ $\displaystyle 0\leq\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)}\leq\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon/2)}$ $$

holds with probability 1.
Therefore, for any $0<\epsilon<1$, we have

$$ $\displaystyle\lim_{N,k\to\infty}\mathcal{L}(\theta,k,N)=\mathbb{E}\lim_{N,k\to\infty}\left[\frac{\sum_{j\in[N]}h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}{\sum_{j\in[N]}f(R_{\text{LLM}_{\theta}}((\ell_{i}^{(j)})_{t\in[T]}),k)}\right]$ $\displaystyle=h\left(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)$ $\displaystyle\,\,\quad\times\mathbb{E}\lim_{N,k\to\infty}\left[\frac{\sum_{j\in[N]}\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)\mathbbm{1}(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}>1-\epsilon)}{\sum_{j\in[N]}f(R_{\text{LLM}_{\theta}}((\ell_{i}^{(j)})_{t\in[T]}),k)\mathbbm{1}(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}>1-\epsilon)}\right]$ $\displaystyle\geq(1-\epsilon)h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$ $$

which implies $\lim_{N,k\to\infty}\mathcal{L}(\theta,k,N)=h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$
since

$$ $\mathcal{L}(\theta,k,N)\leq h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)$ $$

by definition of $\mathcal{L}$, the
fact that $h$ is non-decreasing, and by setting $\epsilon\to 0$ to obtain

$$ $\mathcal{L}(\theta,k,N)\geq h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right).$ $$

Here, we used the fact that $(\ell_{t})_{t\in[T]}$ has a continuous distribution, $\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})$ is a continuous function, and the non-decreasing property and continuity of $h$ ([1](#Thmcondition1)), which lead to:

$$ $\displaystyle\text{esssup}\left(h\left(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)\right)=\max_{\ell_{1},\dots,\ell_{T}}h\left(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)=h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right).$ (14) $$

[Equation 14](#A4.E14) will be used frequently in the overall proof in [Section D.2](#A4.SS2).

Step 3. Proving $\lim_{k\to\infty}\lim_{N\to\infty}\mathcal{L}(\theta,k,N)=h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)$.

Lastly, if $N\to\infty$, similarly by dominated convergence theorem we have

$$ $\displaystyle\lim_{N\to\infty}\mathcal{L}(\theta,k,N)$ $\displaystyle=\mathbb{E}\lim_{N\to\infty}\left[\frac{\sum_{j\in[N]}h\left(R_{\text{LLM}_{\theta}}\left((\ell_{t}^{(j)})_{t\in[T]}\right)\right)f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}{\sum_{j\in[N]}f\left(R_{\text{LLM}_{\theta}}\left((\ell_{i}^{(j)})_{t\in[T]}\right),k\right)}\right]$ $\displaystyle=\frac{\mathbb{E}\left[h\left(R_{\text{LLM}_{\theta}}\left((\ell_{t}^{(j)})_{t\in[T]}\right)\right)f\left(R_{\text{LLM}_{\theta}}\left((\ell_{t}^{(j)})_{t\in[T]}\right),k\right)\right]}{\mathbb{E}\left[f\left(R_{\text{LLM}_{\theta}}\left((\ell_{i}^{(j)})_{t\in[T]}\right),k\right)\right]}.$ $$

Thus, $\lim_{N\to\infty}\mathcal{L}(\theta,k,N)$ always exists for every $k$. Now, we use the known property of double iterated limit ([Lemma 5](#Thmlemma5)), and obtain that $\lim_{k\to\infty}\lim_{N\to\infty}\mathcal{L}(\theta,k,N)=h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$.
∎

###### Claim 2 (Uniform convergence of $\mathcal{L}(\theta,k,N)$ (with respect to $k$ and $N$ )) .

$\mathcal{L}(\theta,k,N)$ uniformly converges to $h(\max_{\ell_{1},\dots,\ell_{T}}\emph{Regret}_{\emph{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$ on the domain $\Theta$.

###### Proof.

We will provide a similar analysis as [Lemma 8](#Thmlemma8) as follows:

###### Lemma 9 .

For any $0<\epsilon<1$, $0<\delta<1$, and $k\in\mathbb{N}^{+}$, we have

$$ $\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon)}=\widetilde{\mathcal{O}}\left(A(k,H,\epsilon)\left(\frac{1}{1-F_{H,X}(1-\epsilon/2)}+\frac{1}{\sqrt{N}}\right)\right)$ $$

with probability at least $1-\delta$, where $X_{i}$’s are i.i.d. random variables, $\operatorname{esssup}(H(X_{i}))=1$, $H:\mathbb{R}\to\mathbb{R}^{+}$ is a continuous non-decreasing function, $A(k,t,\epsilon):=\frac{(1-\epsilon)f((t/\operatorname{esssup}(t(X)))^{-1}(1-\epsilon),k)}{(1-\epsilon/2)f((t/\operatorname{esssup}(t(X)))^{-1}(1-\epsilon/2),k)}$, for any non-decreasing function $t:\mathbb{R}\to\mathbb{R}^{+}$, and $F_{t,X}$ is a cumulative distribution function of random variable $t(X)/\text{esssup}(t(X))$.

###### Proof of Lemma 9 .

With the same argument as the proof of [Lemma 8](#Thmlemma8), we have

$$ $\displaystyle\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon/2)}\leq\frac{f(H^{-1}(1-\epsilon),k)|\{i\in[N]\mid(H(X_{i})<1-\epsilon)\}|}{f(H^{-1}(1-\epsilon/2),k)|\{i\in[N]\mid(H(X_{i})>1-\epsilon/2)\}|}.$ $$

It holds that $\frac{1}{N}|\{i\in[N]\mid(H(X_{i})<1-\epsilon)\}|=F_{H,X}(1-\epsilon)+\widetilde{\mathcal{O}}(1/\sqrt{N})$ with probability at least $1-\delta/2$ due to Hoeffding’s inequality ([Lemma 6](#Thmlemma6)). Similarly, we have $\frac{1}{N}|\{i\in[N]\mid(H(X_{i})>1-\epsilon/2)\}|=1-F_{H,X}(1-\epsilon/2)+\widetilde{\mathcal{O}}(1/\sqrt{N})$ with probability at least $1-\delta/2$. Therefore,

$$ $\displaystyle\frac{|\{i\in[N]\mid(H(X_{i})<1-\epsilon)\}|}{|\{i\in[N]\mid(H(X_{i})>1-\epsilon/2)\}|}=\frac{F_{H,X}(1-\epsilon)}{1-F_{H,X}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N})\leq\frac{1}{1-F_{H,X}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N}),$ $$

with probability at least $1-\delta$. Finally, we have

$$ $\displaystyle\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon)}<\frac{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})<1-\epsilon)}{\sum_{i=1}^{N}f(X_{i},k)\mathbbm{1}(H(X_{i})>1-\epsilon/2)}$ $\displaystyle\leq A(k,H,\epsilon)\left(\frac{1}{1-F_{H,X}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\frac{1}{\sqrt{N}})\right).$ $$

∎

Note that $\lim_{k\to\infty}A(k,H,\epsilon)=0$, since $\lim_{k\to\infty}\frac{f(R_{1},k)}{f(R_{2},k)}=\infty\cdot\mathbbm{1}(R_{1}>R_{2})+\mathbbm{1}(R_{1}=R_{2})$.
By [Lemma 9](#Thmlemma9) with $H(R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))=\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}$, we have

$$ $\displaystyle\frac{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)\mathbbm{1}\left(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}\geq 1-\epsilon\right)}{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)}$ $\displaystyle\quad=\frac{1}{1+\frac{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)\mathbbm{1}\left(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}<1-\epsilon\right)}{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)\mathbbm{1}\left(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}\geq 1-\epsilon\right)}}\geq\frac{1}{1+A(k,H,\epsilon)(\frac{1}{1-F_{H,R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))},$ $$

where we recall the shorthand notation of $R_{\text{LLM}_{\theta}}=\text{Regret}_{\text{LLM}_{\theta}}$. Note that $A(k,H,\epsilon)=A(k,h,\epsilon)$ and $F_{H,R_{\text{LLM}_{\theta}}}=F_{h,R_{\text{LLM}_{\theta}}}$ hold by the definitions of $F_{t,X}$ and $A(k,t,\epsilon)$ in [Lemma 9](#Thmlemma9).
Therefore,

$$ $\displaystyle 1$ $\displaystyle\geq\frac{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}}{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)}$ $\displaystyle\geq\frac{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}\mathbbm{1}(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}\geq 1-\epsilon)}{\sum_{i=1}^{N}f(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}),k)\mathbbm{1}(\frac{h(R_{\text{LLM}_{\theta}}((\ell_{t}^{(i)})_{t\in[T]}))}{h(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))}\geq 1-\epsilon)}$ $\displaystyle\qquad\times\frac{1}{1+A(k,h,\epsilon)(\frac{1}{1-F_{h,R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))}$ $\displaystyle\geq\frac{1-\epsilon}{1+A(k,h,\epsilon)(\frac{1}{1-F_{h,R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))}$ $$

with probability at least $1-\delta$.

Now, for any $\epsilon>0$ and $\delta>0$, we have

$$ $\displaystyle 0$ $\displaystyle\leq h\left(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)-\mathcal{L}(\theta,k,N)$ $\displaystyle\leq h\left(\max_{\ell_{1},\dots,\ell_{T}}R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)\left(1-\frac{(1-\delta)(1-\epsilon)}{1+A(k,h,\epsilon)(\frac{1}{1-F_{h,R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))}\right).$ $$

Note that

$$ $1-F_{h,R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})}(1-\epsilon/2)=\mathbb{P}\left(h\left(\text{Regret}_{\text{LLM}_{\theta}}\left((\ell_{t})_{t\in[T]}\right)\right)>(1-\epsilon/2)h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}\left((\ell_{t})_{t\in[T]}\right)\right)\right)$ $$

is a continuous function of $\theta$, since we assume $\text{LLM}_{\theta}$ is a continuous function of $\theta$, $(\ell_{t})_{t\in[T]}$ has a continuous distribution, and $\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})$ is a continuous function of $\text{LLM}_{\theta}$ and $(\ell_{t})_{t\in[T]}$. Since we consider a compact $\Theta$ (as several recent works on analyzing Transformers (Bai et al., ; Lin et al., )), we have $p(\epsilon):=\min_{\theta\in\Theta}1-F_{h,R_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})}(1-\epsilon/2)>0$. Therefore,

$$ $\displaystyle\left(1-\frac{(1-\delta)(1-\epsilon)}{1+A(k,h,\epsilon)(\frac{1}{1-F_{h,R_{\text{LLM}_{\theta}}}(1-\epsilon/2)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))}\right)\leq\left(1-\frac{(1-\delta)(1-\epsilon)}{1+A(k,h,\epsilon)(\frac{1}{p(\epsilon)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))}\right),$ (15) $$

and we know that $\lim_{N,k\to\infty}1+A(k,h,\epsilon)(\frac{1}{p(\epsilon)}+\widetilde{\mathcal{O}}(\sqrt{1/N}))=1$, which is not dependent on $\theta$. Thus, we can conclude that $\lim_{N,k\to\infty}\sup_{\theta\in\Theta}|h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))-\mathcal{L}(\theta,k,N)|=0$, as we can choose arbitrarily small $\epsilon,\delta$.
∎

###### Claim 3 (Double iterated limit of supremum) .

It holds that:

$$ $\lim_{N\to\infty}\lim_{k\to\infty}\sup_{\theta\in\Theta}~{}~{}\Big{|}\mathcal{L}(\theta,k,N)-h\left(\max_{\ell_{1},\dots,\ell_{T}}\emph{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)\Big{|}=0.$ $$

###### Proof.

Since $h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))\geq\mathcal{L}(\theta,k,N)$, we will prove

$$ $\lim_{N\to\infty}\lim_{k\to\infty}\sup_{\theta\in\Theta}~{}~{}h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)-\mathcal{L}(\theta,k,N)=0.$ $$

###### Lemma 10 .

$\frac{\sum_{i=1}^{N}f(X_{i},k_{1})h(X_{i})}{\sum_{i=1}^{N}f(X_{i},k_{1})}\leq\frac{\sum_{i=1}^{N}f(X_{i},k_{2})h(X_{i})}{\sum_{i=1}^{N}f(X_{i},k_{2})}$ holds if ${0<}k_{1}\leq k_{2}$ for any real-valued $(X_{i})_{i\in[N]}$.

###### Proof.

By multiplying $({\sum_{i=1}^{N}f(X_{i},k_{1})})({\sum_{i=1}^{N}f(X_{i},k_{2})})$ on both sides of the formula, we know that it is equivalent to $\sum_{1\leq i\neq j\leq N}f(X_{i},k_{1})h(X_{i})f(X_{j},k_{2})\leq\sum_{1\leq i\neq j\leq N}f(X_{i},k_{1})h(X_{j})f(X_{j},k_{2})$. This is equivalent to

$$ $\displaystyle\sum_{1\leq i\neq j\leq N}(f(X_{i},k_{1})f(X_{j},k_{2})-f(X_{j},k_{1})f(X_{i},k_{2}))(h(X_{i})-h(X_{j}))\leq 0,$ $$

which is true since if $X_{i}\geq X_{j}$, $(f(X_{i},k_{1})f(X_{j},k_{2})-f(X_{j},k_{1})f(X_{i},k_{2}))\leq 0$ due to the log-increasing
difference of $f$ ([1](#Thmcondition1)), as $\log f(X_{j},k_{1})-\log f(X_{j},k_{2})\geq\log f(X_{i},k_{1})-\log f(X_{i},k_{2})$ if $X_{i}\geq X_{j}$.
∎

Therefore, $\mathcal{L}(\theta,k,N)$ is a non-decreasing function of $k$ if $N$ is fixed, which indicates that

$$ $\lim_{k\to\infty}~{}~{}\sup_{\theta\in\Theta}~{}~{}h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)-\mathcal{L}(\theta,k,N)$ $$

exists, as $\mathcal{L}(\theta,k,N)$ is also bounded. Therefore, by [Lemma 5](#Thmlemma5) and [2](#Thmclaim2), we know that

$$ $\lim_{N\to\infty}\lim_{k\to\infty}~{}~{}\sup_{\theta\in\Theta}~{}~{}\Big{|}\mathcal{L}(\theta,k,N)-h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)\Big{|}$ $$

exists and this value should be 0.
∎

###### Claim 4 .

It holds that

$$ $\lim_{N,k\to\infty}\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)=\lim_{N\to\infty}\lim_{k\to\infty}\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)=\inf_{\theta\in\Theta}h\left(\max_{\ell_{1},\dots,\ell_{T}}\emph{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right).$ $$

###### Proof.

Firstly, by [Lemma 7](#Thmlemma7), we have $\lim_{N,k\to\infty}\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)=\inf_{\theta\in\Theta}h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$. Plus, we already know that $\mathcal{L}(\theta,k,N)$ is a monotonically non-decreasing function of $k$ for any fixed $N$ ([Lemma 10](#Thmlemma10)), and it is bounded, $\lim_{k\to\infty}\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)$ always exists. Therefore, by [Lemma 5](#Thmlemma5)
, we also have $\lim_{N\to\infty}\lim_{k\to\infty}\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)=\inf_{\theta\in\Theta}h(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]}))$.
∎

###### Condition 1 .

###### Claim 1 (Iterated limit of ℒ ​ ( θ , k , N ) ℒ 𝜃 𝑘 𝑁 \mathcal{L}(\theta,k,N) is the same as double limit of ℒ ​ ( θ , k , N ) ℒ 𝜃 𝑘 𝑁 \mathcal{L}(\theta,k,N) ) .

###### Proof.

###### Lemma 8 .

###### Proof of Lemma 8 .

###### Claim 2 (Uniform convergence of ℒ ​ ( θ , k , N ) ℒ 𝜃 𝑘 𝑁 \mathcal{L}(\theta,k,N) (with respect to k 𝑘 k and N 𝑁 N )) .

###### Proof.

###### Lemma 9 .

###### Proof of Lemma 9 .

###### Claim 3 (Double iterated limit of supremum) .

###### Proof.

###### Lemma 10 .

###### Proof.

###### Claim 4 .

###### Proof.

### D.3 Deferred Proofs of Theorem 2 and Corollary 1

See [2](#Thmtheorem2)

Before proving the theorem, we remark on what LLM structure enjoys the Lipschitz-continuity. We provide two auxiliary results in the following proposition.
The first result is from (Bai et al., , Section J.1), which is about the Lipschitzness of Transformers.
The second result is regarding processing the output of Transformers.
In particular, the output of Transformers is usually not directly used, but passed through some matrix multiplication (by some matrix $A$), followed by some projection Operator (to be specified later).

###### Proposition 2 .

The $L$-layer Transformer $\texttt{TF}_{\theta}$ as defined in [Section A.1](#A1.SS1) is $C_{\texttt{TF}}$-Lipschitz continuous with respect to $\theta$ with $C_{\texttt{TF}}:=L\left((1+B_{\texttt{TF}}^{2})(1+B_{\texttt{TF}}^{2}R^{3})\right)^{L}B_{\texttt{TF}}R(1+B_{\texttt{TF}}R^{2}+B_{\texttt{TF}}^{3}R^{2})$, i.e.,

$$ $\displaystyle\|\texttt{TF}_{\theta_{1}}(Z)-\texttt{TF}_{\theta_{2}}(Z)\|_{2,\infty}\leq C_{\texttt{TF}}\|\theta_{1}-\theta_{2}\|_{\texttt{TF}}$ $$

where $\|\cdot\|_{\texttt{TF}}$ is as defined in [Equation 9](#A1.E9), and $R,Z,B_{\texttt{TF}}$ are as introduced in [Section A.1](#A1.SS1). Moreover, the function $\texttt{Operator}(A\cdot\texttt{TF}_{\theta}(\cdot)_{-1})$ is $\|A\|_{\texttt{op}}C_{\texttt{TF}}$-Lipschitz continuous with respect to $\theta$, i.e.,

$$ $\displaystyle\|\texttt{Operator}(A\cdot\texttt{TF}_{\theta_{1}}(Z)_{-1})-\texttt{Operator}(A\cdot\texttt{TF}_{\theta_{2}}(Z)_{-1})\|_{2}\leq\|A\|_{\texttt{op}}C_{\texttt{TF}}\|\theta_{1}-\theta_{2}\|_{\texttt{TF}}.$ $$

Here, Operator is either the projection operator onto some convex set, or the Softmax function.

###### Proof.

The first result is from (Bai et al., , Section J.1). The second result comes from

- •
If Operator is a projection onto the convex set, then $\|\texttt{Operator}(x)-\texttt{Operator}(y)\|_{2}\leq\|x-y\|_{2}$;
- •
If Operator is Softmax, then $\|\texttt{Softmax}(x)-\texttt{Softmax}(y)\|_{2}\leq\|x-y\|_{2}$ (Gao and Pavel, , Corollary 3).

Note that the only condition that we require for Operator is its non-expansiveness.
∎

###### Proof of Theorem 2 .

Let $C_{\text{LLM}}$ denote the Lipschitz-continuity constant for $\text{LLM}_{\theta}$ with respect to some norm $\|\cdot\|_{\text{LLM}}$, where $\|\cdot\|_{\text{LLM}}$ denotes any norm defined on the parameter space of LLM (e.g., the norm $\|\cdot\|_{\texttt{TF}}$ above in [Proposition 2](#Thmproposition2)). Now, we prove that regret is also a Lipschitz-continuous function with respect to the LLM’s parameter.

###### Lemma 11 (Lipschitzness of regret) .

The function $\text{Regret}_{\text{LLM}_{\theta}}$ is $C_{\texttt{Reg}}:=BC_{\text{LLM}}T$-Lipschitz continuous with respect to $\theta$, i.e.,

$$ $\displaystyle\Big{|}\emph{Regret}_{\text{LLM}_{\theta_{1}}}$ $\displaystyle((\ell_{t})_{t\in[T]})-\emph{Regret}_{\text{LLM}_{\theta_{2}}}((\ell_{t})_{t\in[T]})\Big{|}\leq C_{\texttt{Reg}}\|\theta_{1}-\theta_{2}\|_{\text{LLM}}.$ $$

###### Proof.

By definition, we have

$$ $\displaystyle\Big{|}\text{Regret}_{\text{LLM}_{\theta_{1}}}$ $\displaystyle((\ell_{t})_{t\in[T]})-\text{Regret}_{\text{LLM}_{\theta_{2}}}((\ell_{t})_{t\in[T]})\Big{|}=\left|\sum_{t=1}^{T}\langle\ell_{t},\text{LLM}_{\theta_{1}}(Z_{t-1})-\text{LLM}_{\theta_{2}}(Z_{t-1})\rangle\right|$ $\displaystyle=B\sum_{t=1}^{T}\|\text{LLM}_{\theta_{1}}(Z_{t-1})-\text{LLM}_{\theta_{2}}(Z_{t-1})\|$ $\displaystyle\leq BC_{\text{LLM}}T\|\theta_{1}-\theta_{2}\|_{\text{LLM}}$ $$

where $Z_{t}:=(\ell_{1},\dots,\ell_{t},c)$ for all $t\in[T]$ and $Z_{0}=(c)$ where $c$ is a $d$-dimensional vector.
∎

Now, we will prove the Lipschitzness of

$$ $\displaystyle C\left((\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta\right):=\frac{\sum_{j\in[N]}h(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))f(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}{\sum_{j\in[N]}f(\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[t]}),k)}$ (16) $$

with respect to the model parameter $\theta$.

###### Claim 5 .

For any $R>0$, there exists $\beta_{R}>0$ such that if $\beta>\beta_{R}$, we have

$$ $\displaystyle\left|\frac{\sum_{n\in[N]}x_{n}f(x_{n},\beta)}{\sum_{n\in[N]}f(x_{n},\beta)}-\frac{\sum_{n\in[N]}y_{n}f(y_{n},\beta)}{\sum_{n\in[N]}f(y_{n},\beta)}\right|\leq 2\|x-y\|_{\infty}$ $$

for every $x,y\in\mathbb{R}^{n}$ such that $|x_{i}|\leq R,|y_{i}|\leq R$ for all $i\in[N]$.

###### Proof.

If $\beta=\infty$, we have

$$ $\displaystyle\lim_{\beta\to\infty}$ $\displaystyle\left(\left|\frac{\sum_{n\in[N]}x_{n}f(x_{n},\beta)}{\sum_{n\in[N]}f(x_{n},\beta)}-\frac{\sum_{n\in[N]}y_{n}f(y_{n},\beta)}{\sum_{n\in[N]}f(y_{n},\beta)}\right|\Big{/}\|x-y\|_{\infty}\right)=\frac{|\max_{n\in[N]}x_{n}-\max_{n\in[N]}y_{n}|}{\|x-y\|_{\infty}}\leq 1$ $$

holds. Moreover, consider the following constrained optimization problem:

$$ $\displaystyle\max_{{x,y\in\mathbb{R}^{n}}}\quad\left(\left|\frac{\sum_{n\in[N]}x_{n}f(x_{n},\beta)}{\sum_{n\in[N]}f(x_{n},\beta)}-\frac{\sum_{n\in[N]}y_{n}f(y_{n},\beta)}{\sum_{n\in[N]}f(y_{n},\beta)}\right|\Big{/}\|x-y\|_{\infty}\right)$ $\displaystyle\qquad\text{subject to }\qquad|x_{i}|\leq R,~{}~{}~{}|y_{i}|\leq R~{}~{}~{}\text{ for all }i\in[N],$ $$

whose optimum is denoted as $F(R,\beta)$.
Then, since $\|x\|_{\infty}\leq R$ and $\|y\|_{\infty}\leq R$ is a compact set, by Berge’s Maximum Theorem (Berge, ), we have that $F(R,\beta)$ is a continuous function for $\beta$. Moreover, we know that $F(R,\infty)\leq 1$, which indicates that we can find a large enough $\beta_{R}$ such that if $\beta>\beta_{R}$, $F(R,\beta)\leq 2$.
∎

Note that [5](#Thmclaim5) does not hold if either $x_{i}$ or $y_{i}$ is unbounded. Now, we will apply [5](#Thmclaim5) to [Equation 16](#A4.E16). We can guarantee that $\left|\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right|\leq\text{diam}(\Pi,\|\cdot\|_{2})TB$.

Also, note that the domain of $h:\mathbb{R}\to\mathbb{R}^{+}$ is effectively *constrained* to the range that $\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})$ can achieve, which means that we can regard $h$ as $h:[-\text{diam}(\Pi,\|\cdot\|_{2})TB,\text{diam}(\Pi,\|\cdot\|_{2})TB]\to\mathbb{R}^{+}$. Due to the continuity of $h^{\prime}$, and the fact that $h$ has a compact domain, we know that $h(\cdot)$ is $C_{h}$-Lipschitz continuous for some $C_{h}>0$ on this interval of $[-\text{diam}(\Pi,\|\cdot\|_{2})TB,\text{diam}(\Pi,\|\cdot\|_{2})TB]$.

###### Lemma 12 (Lipschitzness of $C$ in Equation 16 ) .

The function $C$ in [Equation 16](#A4.E16) is $C_{\texttt{cost}}:=2C_{h}C_{\text{Reg}}$-Lipschitz continuous with respect to $\theta$, if $k>k_{\text{diam}(\Pi,\|\cdot\|_{2})TB}$ for some $k_{\text{diam}(\Pi,\|\cdot\|_{2})TB}>0$, i.e.,

$$ $\displaystyle\Big{|}C\left((\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta_{1}\right)-C\left((\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta_{2}\right)\Big{|}\leq C_{\texttt{cost}}\|\theta_{1}-\theta_{2}\|_{\text{LLM}}.$ $$

###### Proof.

$$ $\displaystyle\big{|}C(($ $\displaystyle\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta_{1})-C((\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta_{2})\big{|}$ $\displaystyle\underset{(i)}{\leq}2\|h(\text{Regret}_{\text{LLM}_{\theta_{1}}}((\ell_{t}^{(j)})_{t\in[T]}))-h(\text{Regret}_{\text{LLM}_{\theta_{2}}}((\ell_{t}^{(j)})_{t\in[T]}))\|_{\infty}$ $\displaystyle\underset{(ii)}{\leq}2C_{h}\|\text{Regret}_{\text{LLM}_{\theta_{1}}}((\ell_{t}^{(j)})_{t\in[T]})-\text{Regret}_{\text{LLM}_{\theta_{2}}}((\ell_{t}^{(j)})_{t\in[T]})\|_{\infty}$ $\displaystyle\underset{(iii)}{\leq}2C_{h}C_{\text{Reg}}\|\theta_{1}-\theta_{2}\|_{\text{LLM}}=C_{\texttt{cost}}\|\theta_{1}-\theta_{2}\|_{\text{LLM}}.$ $$

Here, (i) holds due to [5](#Thmclaim5), (ii) holds since $h$ is $C_{h}$-Lipschitz continuous on the range of $\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})$, and (iii) holds due to [Lemma 11](#Thmlemma11).
∎

For completeness of the paper, we provide the definition of covering set and covering number.

###### Definition 4 (Covering set and covering number) .

For $\delta>0$, a metric space $(X,\|\cdot\|)$, and subset $Y\subseteq X$, set $C\subset Y$ is a $\delta$-covering of $Y$ when $Y\subseteq\cup_{c\in C}B(c,\delta,\|\cdot\|)$ holds. $\delta$-covering number $N(\delta;Y,\|\cdot\|)$ is defined as the minimum cardinality of any covering set.

By (Wainwright, , Example 5.8), for any $r>0$, we can verify that the $\delta$-covering number $N(\delta;B(0,r,\|\cdot\|_{\text{LLM}}),\|\cdot\|_{\text{LLM}})$ can be bounded by

$$ $\displaystyle\log N(\delta;B(0,r,\|\cdot\|_{\text{LLM}}),\|\cdot\|_{\text{LLM}})\leq d_{\theta}\log(1+2r/\delta),$ $$

where $d_{\theta}$ is the dimension of the LLM’s whole parameter. For example, if we use the $\|\cdot\|_{\texttt{TF}}$ and consider the Transformer model as defined in [Section A.1](#A1.SS1), for any $r>0$,

$$ $\displaystyle\log N(\delta;B(0,r,\|\cdot\|_{\text{LLM}}),\|\cdot\|_{\text{LLM}})\leq L(3Md^{2}+2d(dd^{\prime}+3md^{2}))\log(1+2r/\delta).$ $$

Since we consider a compact $\Theta$ (as several recent works on analyzing Transformers (Bai et al., ; Lin et al., )), let $R_{\Theta}:=\text{diam}(\Theta,\|\cdot\|_{\text{LLM}})$ (which corresponds to $B_{\texttt{TF}}$ for the Transformer models as defined in [Section A.1](#A1.SS1), with $\|\cdot\|_{\text{LLM}}=\|\cdot\|_{\texttt{TF}}$), then there exists a set $\Theta_{0}$ with $\log|\Theta_{0}|=d_{\theta}\log(1+2R_{\Theta}/\delta)$ such that for any $\theta\in\Theta$, there exists a $\theta_{0}\in\Theta_{0}$ with

$$ $\displaystyle\Big{|}C\left((\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta\right)-C\left((\ell_{t}^{(j)})_{t\in[T],j\in[N]},k,\theta_{0}\right)\Big{|}\leq C_{\texttt{cost}}\delta.$ $$

Then, by the standard result from statistical learning theory (Wainwright, , Chapter 5), when trained with $N_{T}$ samples, for every $0<\epsilon<1/2$, with probability at least $1-\epsilon$, we have

$$ $\displaystyle\mathcal{L}(\widehat{\theta}_{k,N,N_{T}},k,N)$ $\displaystyle-\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)\leq\sqrt{\frac{2(\log|\Theta_{0}|+\log(2/\epsilon))}{N_{T}}}+2C_{\texttt{cost}}\delta.$ $$

Setting $\delta=\Omega(\sqrt{\log(\epsilon)/N_{T}})$, we further obtain

$$ $\displaystyle\mathcal{L}(\widehat{\theta}_{k,N,N_{T}},k,N)$ $\displaystyle-\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)\leq\widetilde{\mathcal{O}}\left(\sqrt{\frac{\log|\Theta_{0}|+\log(1/\epsilon)}{N_{T}}}\right)$ $$

with probability at least $1-\epsilon$, completing the proof.
∎

See [1](#Thmcorollary1)

###### Proof.

The limit on the right-hand side of [Equation 5](#S5.E5) remains as $\widetilde{\mathcal{O}}\left(\sqrt{\frac{d_{\theta}+\log(1/\epsilon)}{N_{T}}}\right)$, since we firstly take $\lim_{k\to\infty}$ and then take $\lim_{N\to\infty}$, thanks to the fact that [Theorem 2](#Thmtheorem2) holds for large enough $k$ and any $N$.
Next, we have

$$ $\displaystyle\lim_{N\to\infty}\lim_{k\to\infty}\bigg{|}\mathcal{L}(\widehat{\theta}_{k,N,N_{T}},k,N)-h\left(\lim_{N\to\infty}\lim_{k\to\infty}\max_{\|\ell_{t}\|_{\infty}\leq B}\text{Regret}_{\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}((\ell_{t})_{t\in[T]})\right)\bigg{|}$ $\displaystyle\quad\leq\lim_{N\to\infty}\lim_{k\to\infty}\bigg{|}\mathcal{L}(\widehat{\theta}_{k,N,N_{T}},k,N)-h\left(\max_{\|\ell_{t}\|_{\infty}\leq B}\text{Regret}_{\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}((\ell_{t})_{t\in[T]})\right)\bigg{|}+$ $\displaystyle\quad\qquad\lim_{N\to\infty}\lim_{k\to\infty}\bigg{|}h\left(\max_{\|\ell_{t}\|_{\infty}\leq B}\text{Regret}_{\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}((\ell_{t})_{t\in[T]})\right)-h\left(\lim_{N\to\infty}\lim_{k\to\infty}\max_{\|\ell_{t}\|_{\infty}\leq B}\text{Regret}_{\text{LLM}_{\widehat{\theta}_{k,N,N_{T}}}}((\ell_{t})_{t\in[T]})\right)\bigg{|}$ $\displaystyle\quad\leq\lim_{N\to\infty}\lim_{k\to\infty}\sup_{\theta\in\Theta}\bigg{|}\mathcal{L}({\theta},k,N)-h\left(\max_{\|\ell_{t}\|_{\infty}\leq B}\text{Regret}_{\text{LLM}_{{\theta}}}((\ell_{t})_{t\in[T]})\right)\bigg{|}+0=0,$ $$

due to the continuity of $h$
and [3](#Thmclaim3).
Finally, we have

$$ $\displaystyle\lim_{N\to\infty}\lim_{k\to\infty}\inf_{\theta\in\Theta}\mathcal{L}(\theta,k,N)=\inf_{\theta\in\Theta}h\left(\max_{\ell_{1},\dots,\ell_{T}}\text{Regret}_{\text{LLM}_{\theta}}((\ell_{t})_{t\in[T]})\right)$ $$

due to [4](#Thmclaim4), which, combined with the fact that $h$ is non-decreasing, completes the proof.
∎

###### Remark 5 (Dynamic-regret loss) .

So far, we have focused on the canonical online learning setting with regret being the metric. One can also generalize the results to the non-stationary setting, with dynamic regret being the metric.
Specifically, one can define the *dynamic-regret-loss* function as follows:

$$ $\displaystyle\mathcal{L}(\theta,k,N):=\mathbb{E}\left[\frac{\sum_{j\in[N]}h(\text{D-Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}))f(\text{D-Regret}_{\text{LLM}_{\theta}}((\ell_{t}^{(j)})_{t\in[T]}),k)}{\sum_{j\in[N]}f(\text{D-Regret}_{\text{LLM}_{\theta}}((\ell_{i}^{(j)})_{t\in[T]}),k)}\right].$ $$

Then, one can also establish similar results as before, since the analysis does not utilize other properties of the regret except its boundedness, and the Lipschitz-continuity of LLM with respect to $\theta$. To be specific, [Lemma 11](#Thmlemma11) holds due to the reason that we can bound the difference of the regret with the term

$$ $\left|\sum_{t=1}^{T}\langle\ell_{t},(\text{LLM}_{\theta_{1}}(Z_{t-1})-\text{LLM}_{\theta_{2}}(Z_{t-1}))\rangle\right|,$ $$

as well as the fact that $\inf_{\pi_{i}\in\Pi}\langle\ell_{i},\pi_{i}\rangle$ will be canceled. One can verify that all the arguments in [Section D.2](#A4.SS2) also hold for similar reasons.

###### Proposition 2 .

###### Proof.

###### Proof of Theorem 2 .

###### Lemma 11 (Lipschitzness of regret) .

###### Proof.

###### Claim 5 .

###### Proof.

###### Lemma 12 (Lipschitzness of C 𝐶 C in Equation 16 ) .

###### Proof.

###### Definition 4 (Covering set and covering number) .

###### Proof.

###### Remark 5 (Dynamic-regret loss) .

### D.4 Deferred Proof of Theorem 3

See [3](#Thmtheorem3)

###### Proof.

We will locally use $\mathcal{A}=[d]$ without losing generality as $\mathcal{A}$ is finite with $|\mathcal{A}|=d$, and will interchangeably use $\ell_{i}(j)$ and $\ell_{ij}$ for notational convenience. Define $a:=K^{\intercal}(Qc+q_{c})\in\mathbb{R}^{d}$ and $b_{t-1}:=\beta\boldsymbol{1}_{t-1}:=k_{c}^{\intercal}(Qc+q_{c})\boldsymbol{1}_{t-1}\in\mathbb{R}^{t-1}$. With $N=1$, $h(x)=x^{2}$, and the choice of $\Pi$, the loss function ([Equation 3](#S5.E3)) can be written as follows:

$$ $\displaystyle f(V,a,(b_{t})_{t\in[T-1]},v_{c}):=\mathbb{E}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)^{2},$ $$

where for $t=1$, we use the output of the single-layer self-attention as $v_{c}$ and we will write it as $(V\ell_{1:0}+v_{c}\boldsymbol{1}_{0}^{\intercal})\texttt{Softmax}(\ell_{1:0}^{\intercal}a+b_{0})$ for notational consistency with $t\geq 2$. Also, we will define empty sum $\sum_{i=1}^{0}a_{i}=0$ for any sequence $(a_{i})_{i\in\mathbb{N}^{+}}$.

Step 1. Calculating $\frac{\partial f}{\partial a}$.

For $x\in[d]$, we calculate the corresponding directional derivative with the following equation for $t\geq 2$:

$$ $\displaystyle\frac{\partial}{\partial a_{x}}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})$ $\displaystyle=\frac{\partial}{\partial a_{x}}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})e_{i}\frac{\exp(e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}{\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}$ $\displaystyle=\frac{\sum_{i=1}^{t-1}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})e_{i}\exp(e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))\frac{\partial e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1})}{\partial a_{x}}(\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1})))}{(\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1})))^{2}}$ $\displaystyle\qquad-\frac{\sum_{i=1}^{t-1}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})e_{i}\exp(e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))\left(\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))\frac{\partial e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1})}{\partial a_{x}}\right)}{(\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1})))^{2}}.$ $$

Plugging $a=\boldsymbol{0}_{d}$ and $v_{c}=\boldsymbol{0}_{d}$, and $(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}$ provides

$$ $\displaystyle\frac{\partial}{\partial a_{x}}$ $\displaystyle\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\ell_{ix}}{(t-1)}-\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\left(\sum_{s=1}^{t-1}\ell_{sx}\right)}{(t-1)^{2}}.$ $$

For $t=1$, as $\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})=\ell_{1}^{\intercal}v_{c}$, $\frac{\partial}{\partial a_{x}}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}=0$, so we can use the same formula as $t\geq 2$ with empty sum $\sum_{i=1}^{t-1}$. Using the above calculation, we can further compute $\frac{\partial f}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ as follows:

$$ $\displaystyle\frac{\partial f(V,a,(b_{t})_{t\in[T-1]},v_{c})}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\mathbb{E}\frac{\partial}{\partial a_{x}}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)^{2}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle\qquad\qquad\frac{\partial}{\partial a_{x}}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}\Bigg{]}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}V\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{i}+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\sum_{t=1}^{T}\left(\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\ell_{ix}}{(t-1)}-\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\left(\sum_{s=1}^{t-1}\ell_{sx}\right)}{(t-1)^{2}}\right)\Bigg{]}$ (17) $\displaystyle=0,$ $$

where we used the fact that $\ell_{i}$ is drawn from a symmetric distribution, and flipping the sign of the variable as $-\ell_{i}$ yields the same distribution, which leads to the following:

$$ $\displaystyle\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}V\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{i}+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\sum_{t=1}^{T}\left(\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\ell_{ix}}{(t-1)}-\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\left(\sum_{s=1}^{t-1}\ell_{sx}\right)}{(t-1)^{2}}\right)\Bigg{]}$ $\displaystyle\quad=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}V\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{i}+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\sum_{t=1}^{T}\left(-\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\ell_{ix}}{(t-1)}+\sum_{i=1}^{t-1}\frac{\ell_{t}^{\intercal}V\ell_{i}\left(\sum_{s=1}^{t-1}\ell_{sx}\right)}{(t-1)^{2}}\right)\Bigg{]}.$ $$

This yields [Equation 17](#A4.E17)=0.

Step 2. Calculating $\frac{\partial f}{\partial v_{c}}$.

We will use the following equation for $t\geq 2$:

$$ $\displaystyle\frac{\partial}{\partial v_{c}}$ $\displaystyle\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})$ $\displaystyle=\frac{\partial}{\partial v_{c}}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})e_{i}\frac{\exp(e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}{\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}=\ell_{t}.$ $$

For $t=1$, we define $\frac{\partial}{\partial v_{c}}\ell_{1}^{\intercal}(V\ell_{1:0}+v_{c}\boldsymbol{1}_{0}^{\intercal})\texttt{Softmax}(\ell_{1:0}^{\intercal}a+b_{0})=\ell_{1}$, so that we can use the same formula as $t\geq 2$.
Therefore, we can calculate $\frac{\partial f}{\partial v_{c}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ as follows:

$$ $\displaystyle\frac{\partial f(V,a,(b_{t})_{t\in[T-1]},v_{c})}{\partial v_{c}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\mathbb{E}\frac{\partial}{\partial v_{c}}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)^{2}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle\qquad\qquad\frac{\partial}{\partial v_{c}}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}\Bigg{]}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=2}^{T}\ell_{t}^{\intercal}V\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{i}+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\sum_{t=1}^{T}\ell_{t}\Bigg{]}=0.$ $$

The last line is due to the same reason as the last part of Step 1.

Step 3. Calculating $\frac{\partial f}{\partial V}$.

We calculate the following equation, which will be used to calculate $\frac{\partial f}{\partial V}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ for $t\geq 2$:

$$ $\displaystyle\frac{\partial}{\partial V}$ $\displaystyle\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\frac{\partial}{\partial V}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})e_{i}\frac{\exp(e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}{\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\sum_{i=1}^{t-1}\ell_{t}\ell_{i}^{\intercal}\frac{\exp(e_{i}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}{\sum_{s=1}^{t-1}\exp(e_{s}^{\intercal}(\ell_{1:t-1}^{\intercal}a+b_{t-1}))}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}=\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{t}\ell_{i}^{\intercal}.$ $$

For $t=1$, note that $\frac{\partial}{\partial V}\ell_{t}^{\intercal}v_{c}=\boldsymbol{O}_{d\times d}$, so we can use the same formula as $t\geq 2$ with empty sum $\sum_{i=1}^{t-1}$.

Therefore, we have

$$ $\displaystyle\frac{\partial f(V,a,(b_{t})_{t\in[T-1]},v_{c})}{\partial V}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\mathbb{E}\frac{\partial}{\partial V}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)^{2}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}$ $\displaystyle\qquad\qquad\frac{\partial}{\partial V}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}(V\ell_{1:t-1}+v_{c}\boldsymbol{1}_{t-1}^{\intercal})\texttt{Softmax}(\ell_{1:t-1}^{\intercal}a+b_{t-1})+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]}}\Bigg{]}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\ell_{t}^{\intercal}V\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{i}+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)\sum_{t=1}^{T}\sum_{i=1}^{t-1}\frac{1}{t-1}\ell_{t}\ell_{i}^{\intercal}\Bigg{]}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\left(\frac{1}{t-1}\ell_{t}^{\intercal}V\ell_{i}\right)\left(\frac{1}{t-1}\ell_{t}\ell_{i}^{\intercal}\right)+R_{\Pi}T\|\sum_{t^{\prime}=1}^{T}\ell_{t^{\prime}}\|_{2}\ell_{t}\ell_{i}^{\intercal}\right)\Bigg{]}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\sum_{x=1}^{d}\sum_{y=1}^{d}v_{xy}\ell_{tx}\ell_{iy}\left(\frac{1}{t-1}\right)^{2}[\ell_{tz}\ell_{iw}]_{(z,w)}+R_{\Pi}T\|\sum_{t^{\prime}=1}^{T}\ell_{t^{\prime}}\|_{2}\ell_{t}\ell_{i}^{\intercal}\right)\Bigg{]}$ $\displaystyle=\sum_{t=1}^{T}\sum_{i=1}^{t-1}\sum_{x=1}^{d}\sum_{y=1}^{d}\frac{1}{(t-1)^{2}}[\sigma_{xz}v_{xy}\sigma_{yw}]_{(z,w)}+\mathbb{E}\Bigg{[}R_{\Pi}T\|\sum_{t^{\prime}=1}^{T}\ell_{t^{\prime}}\|_{2}\ell_{t}\ell_{i}^{\intercal}\Bigg{]}$ $\displaystyle=\left(\sum_{t=1}^{T-1}\frac{1}{t}\right)\Sigma V\Sigma+\mathbb{E}\Bigg{[}R_{\Pi}T\|\sum_{t^{\prime}=1}^{T}\ell_{t^{\prime}}\|_{2}\ell_{t}\ell_{i}^{\intercal}\Bigg{]}.$ $$

Therefore, if $V^{\star}=R_{\Pi}\frac{T}{\sum_{t=1}^{T-1}1/t}\Sigma^{-1}\mathbb{E}\Bigg{[}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\ell_{t}\ell_{i}^{\intercal}\Bigg{]}\Sigma^{-1}$, then $\frac{\partial f}{\partial V}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]},V=V^{\star}}=\bm{O}_{d\times d}$.
Lastly, we have

$$ $\displaystyle\frac{\partial f}{\partial K}\big{|}_{K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d},V=V^{\star}}=\left(\frac{\partial f}{\partial a}\frac{\partial a}{\partial K}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]},V=V^{\star}}=\boldsymbol{O}_{d\times d}$ $\displaystyle\frac{\partial f}{\partial Q}\big{|}_{K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d},V=V^{\star}}=\left(\frac{\partial f}{\partial a}\frac{\partial a}{\partial Q}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]},V=V^{\star}}=\boldsymbol{O}_{d\times d}$ $\displaystyle\frac{\partial f}{\partial q_{c}}\big{|}_{K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d},V=V^{\star}}=\left(\frac{\partial f}{\partial a}\frac{\partial a}{\partial q_{c}}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=\boldsymbol{0}_{d},(b_{t}=\beta\boldsymbol{1}_{t})_{t\in[T-1]},V=V^{\star}}=\boldsymbol{0}_{d}$ $$

which means that such configurations are first-order stationary points of [Equation 3](#S5.E3) with $N=1$, $h(x)=x^{2}$, and $\Pi=B(0,R_{\Pi},\|\cdot\|)$.
∎

###### Proof.

### D.5 Deferred Proof of Theorem 4

See [4](#Thmtheorem4)

###### Proof.

The output of the single-layer linear self-attention structure is as follows:

$$ $\displaystyle g($ $\displaystyle Z_{t};V,K,Q,v_{c},k_{c},q_{c})$ (18) $\displaystyle=\sum_{i=1}^{t}\left(V\ell_{i}\ell_{i}^{\intercal}(K^{\intercal}(Qc+q_{c}))+\left(Vk_{c}^{\intercal}(Qc+q_{c})+v_{c}(Qc+q_{c})^{\intercal}K\right)\ell_{i}+v_{c}k_{c}^{\intercal}(Qc+q_{c})\right),$ $$

which can be expressed with a larger class

$$ $\displaystyle g(Z_{t},\mathbb{A},\beta,\mathbb{C},\delta):=\sum_{i=1}^{t}(\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta+\mathbb{C}\ell_{i}+\delta),$ (19) $$

where $\mathbb{A}\in\mathbb{R}^{d\times d}$, $\beta,\mathbb{C},\delta\in\mathbb{R}^{d}$. Then, if a minimizer of

$$ $\displaystyle f(\mathbb{A},\beta,\mathbb{C},\delta):$ $\displaystyle=\mathbb{E}\left(\sum_{t=1}^{T}\langle\ell_{t},\sum_{i=1}^{t-1}\left(\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta+\mathbb{C}\ell_{i}+\delta\right)\rangle-\inf_{\pi\in\Pi}\left\langle\sum_{t=1}^{T}\ell_{t},\pi\right\rangle\right)^{2}$ $$

can be expressed as $\mathbb{A}=V,\beta=K^{\intercal}(Qc+q_{c}),\mathbb{C}=Vk_{c}^{\intercal}(Qc+q_{c})+v_{c}(Qc+q_{c})^{\intercal}K,\beta=v_{c}k_{c}^{\intercal}(Qc+q_{c})$, then we can conclude that the corresponding $V,Q,K,v_{c},q_{c},k_{c}$ are also a minimizer of

$$ $\displaystyle\mathbb{E}\left(\sum_{t=1}^{T}\langle\ell_{t},g(Z_{t-1})\rangle-\inf_{\pi\in\Pi}\left\langle\sum_{t=1}^{T}\ell_{i},\pi\right\rangle\right)^{2},$ $$

since the corresponding $V,Q,K,v_{c},q_{c},k_{c}$ constitute a minimizer among a larger class. Now, since $\Pi=B(\boldsymbol{0}_{d},R_{\Pi},\|\cdot\|)$, we can rewrite $f$ as

$$ $\displaystyle f(\mathbb{A},\beta,\mathbb{C},\delta)=\mathbb{E}\left(\sum_{t=1}^{T}\langle\ell_{t},\sum_{i=1}^{t-1}\left(\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta+\mathbb{C}\ell_{i}+\delta\right)\rangle+R_{\Pi}\bigg{\|}\sum_{t=1}^{T}\ell_{i}\bigg{\|}_{2}\right)^{2}.$ (20) $$

Step 1. Finding condition for $\frac{\partial f}{\partial\delta}=0$.

Due to the Leibniz rule, if we calculate the partial derivative of [Equation 20](#A4.E20) w.r.t. $\delta$, we have

$$ $\displaystyle\frac{\partial f(\mathbb{A},\beta,\mathbb{C},\delta)}{\partial\delta}=\frac{\partial}{\partial\delta}\mathbb{E}\left(\sum_{t=1}^{T}\langle\ell_{t},\sum_{i=1}^{t-1}\left(\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta+\mathbb{C}\ell_{i}+\delta\right)\rangle+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)^{2}$ $\displaystyle\qquad=\mathbb{E}\frac{\partial}{\partial\delta}\left(\sum_{t=1}^{T}\langle\ell_{t},\sum_{i=1}^{t-1}\left(\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta+\mathbb{C}\ell_{i}+\delta\right)\rangle+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|_{2}\right)^{2}$ $\displaystyle\qquad=\mathbb{E}\sum_{t=1}^{T}\ell_{t}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}(t-1)\ell_{t}^{\intercal}\left(\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta+\mathbb{C}\ell_{i}+\delta\right)+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|\right).$ (21) $$

Since the expectation of either odd-order polynomial or even-order polynomial times $\|\cdot\|_{2}$ is 0, due to that $\ell_{t}$ follows a symmetric distribution, we have

$$ $\displaystyle\mathbb{E}\sum_{t=1}^{T}(t-1)\ell_{t}R_{\Pi}\Big{\|}\sum_{t=1}^{T}\ell_{t}\Big{\|}_{2}=0,\qquad\mathbb{E}\sum_{t=1}^{T}(t-1)\ell_{t}\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}=0.$ $$

Now, we calculate

$$ $\displaystyle\mathbb{E}\sum_{t=1}^{T}$ $\displaystyle(t-1)\ell_{t}\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta=\mathbb{E}\sum_{t_{1}=1}^{T}\sum_{t=1}^{T}\sum_{i=1}^{t-1}(t_{1}-1)\ell_{t_{1}}\ell_{t}^{\intercal}\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta$ $\displaystyle\underset{(i)}{=}\mathbb{E}\sum_{t=1}^{T}\sum_{i=1}^{t-1}(t-1)\ell_{t}\ell_{t}^{\intercal}\mathbb{A}\ell_{i}\ell_{i}^{\intercal}\beta=\mathbb{E}\sum_{t=1}^{T}(t-1)^{2}\ell_{t}\ell_{t}^{\intercal}\mathbb{A}\Sigma\beta=\frac{1}{6}T(2T^{2}-3T+1)\Sigma\mathbb{A}\Sigma\beta,$ $$

where $(i)$ holds since if $t_{1}\neq t$, due to the independence of $\ell_{t}$, $\ell_{t_{1}}$, we can use $\mathbb{E}\ell_{t}=0$. Lastly,

$$ $\displaystyle\mathbb{E}\sum_{t=1}^{T}$ $\displaystyle(t-1)\ell_{t}\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\delta=\mathbb{E}\sum_{t_{1}=1}^{T}\sum_{t=1}^{T}(t_{1}-1)(t-1)\ell_{t_{1}}\ell_{t}^{\intercal}\delta=\frac{1}{6}T(2T^{2}-3T+1)\Sigma\delta.$ $$

Plugging the above equations into [Equation 21](#A4.E21), we have

$$ $\displaystyle\frac{\partial f(\mathbb{A},\beta,\mathbb{C},\delta)}{\partial\delta}=\frac{1}{6}T(2T^{2}-3T+1)(\Sigma\mathbb{A}\Sigma\beta+\Sigma\delta).$ $$

Due to the optimality condition, we have

$$ $\displaystyle\mathbb{A}\Sigma\beta+\delta=0.$ (22) $$

Step 2. Plugging the optimality condition for $\frac{\partial f}{\partial\delta}$ into [Equation 20](#A4.E20).

Plugging [Equation 22](#A4.E22) to [Equation 20](#A4.E20), $f$ can be written as

$$ $\displaystyle f(\mathbb{A},$ $\displaystyle\beta,\mathbb{C},-\mathbb{A}\Sigma\beta)=\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\left(\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta+\mathbb{C}\ell_{i}\right)+R_{\Pi}\bigg{\|}\sum_{t=1}^{T}\ell_{t}\bigg{\|}_{2}\right)^{2}$ $\displaystyle=\underbrace{\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right)^{2}}_{(i)}+\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\right)^{2}+\mathbb{E}\left(R_{\Pi}\bigg{\|}\sum_{t=1}^{T}\ell_{t}\bigg{\|}_{2}\right)^{2}$ $\displaystyle\quad+\underbrace{2\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right)\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\right)}_{(ii)}$ $\displaystyle\qquad+\underbrace{2\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right)\left(R_{\Pi}\bigg{\|}\sum_{t=1}^{T}\ell_{t}\bigg{\|}_{2}\right)}_{(iii)}$ $\displaystyle\qquad\quad+2\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\right)\left(R_{\Pi}\bigg{\|}\sum_{t=1}^{T}\ell_{t}\bigg{\|}_{2}\right).$ $$

For the part $(i)$, we have

$$ $\displaystyle\mathbb{E}$ $\displaystyle\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right)^{2}=\mathbb{E}\left[\sum_{t_{1}=1}^{T}\sum_{i_{1}=1}^{t_{1}-1}\sum_{t=1}^{T}\sum_{i=1}^{t-1}\beta^{\intercal}(\ell_{i_{1}}\ell_{i_{1}}^{\intercal}-\Sigma)\mathbb{A}^{\intercal}\ell_{t_{1}}\ell_{t}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right]$ $\displaystyle\qquad\underset{(1)}{=}\mathbb{E}\left[\sum_{t=1}^{T}\sum_{i_{1}=1}^{t-1}\sum_{i=1}^{t-1}\beta^{\intercal}(\ell_{i_{1}}\ell_{i_{1}}^{\intercal}-\Sigma)\mathbb{A}^{\intercal}\ell_{t}\ell_{t}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right]$ $\displaystyle\qquad\underset{(2)}{=}\mathbb{E}\left[\sum_{t=1}^{T}\sum_{i=1}^{t-1}\beta^{\intercal}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\mathbb{A}^{\intercal}\ell_{i}\ell_{i}^{\intercal}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta\right]$ $\displaystyle\qquad=\frac{(T-1)T}{2}\beta^{\intercal}\mathbb{E}\left[(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\mathbb{A}^{\intercal}\Sigma\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\right]\beta$ (23) $\displaystyle\qquad=\frac{(T-1)T}{2}\beta^{\intercal}\mathbb{E}\left[(\sqrt{\Sigma}A(\ell_{i}\ell_{i}^{\intercal}-\Sigma))^{\intercal}(\sqrt{\Sigma}A(\ell_{i}\ell_{i}^{\intercal}-\Sigma))\right]\beta.$ $$

Here, $(1)$ holds because if $t_{1}\neq t$, we know that $\mathbb{E}\ell_{t_{1}}=\mathbb{E}\ell_{t}=0$, and they are independent, and $(2)$ holds because if $i_{1}\neq i$, we can calculate $\mathbb{E}(\ell_{i_{1}}\ell_{i_{1}}^{\intercal}-\Sigma)=\bm{O}_{d\times d}$. In addition, we can easily check that $(ii)$ and $(iii)$ are 0 0 as they are polynomials of odd degrees and we have $Z\overset{d}{=}-Z$. Note that [Equation 23](#A4.E23) is minimized when $\mathbb{P}(\sqrt{\Sigma}\mathbb{A}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta=\boldsymbol{0}_{d})=1$.

If $\mathbb{A}\neq\bm{O}_{d\times d}$, suppose that the singular value decomposition of $A=U\Lambda V$ yields that $\Lambda$ is a diagonal matrix whose first diagonal element is non-zero, and $U,V$ are orthogonal matrices. Then, we want to find $\beta$ that $\sqrt{\Sigma}U\Lambda V(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta=\boldsymbol{0}_{d}$ for any $\ell_{i}$ such that $p(\ell_{i})\neq 0$, where $p$ indicates the probability density function of loss vectors. Since $\Sigma$ and $U$ are invertible, we only need to consider $\Lambda V(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta=\boldsymbol{0}_{d}$. Since $\Lambda$’s first diagonal component is non-zero, we will consider equation $e_{1}^{\intercal}\Lambda V(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta=0$. This is equivalent to $V_{1}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\beta=0$, where $V_{1}$ is the first row of $V$, and is a non-zero vector.

Now, we will generally consider $a_{x,y}(v):=vv^{\intercal}x-y$ where $x,y,v\in\mathbb{R}^{d}$ and $a_{x,y}:B(\boldsymbol{0}_{d},2\epsilon_{1},\|\cdot\|)\to\mathbb{R}^{d}$ function. Then, we can check that the Jacobian of $a_{x,y}(v)$ is $vx^{\intercal}+(v\cdot x)I$, and we can find that the determinant of the Jacobian is nonzero when $v=\epsilon_{1}x$ if $x\neq\boldsymbol{0}_{d}$. Therefore, the volume of $(V_{1}(\ell_{i}\ell_{i}^{\intercal}-\Sigma))$ for $\ell_{i}\in B(\boldsymbol{0}_{d},c_{z},\|\cdot\|)$ is greater than the volume of $(V_{1}(vv^{\intercal}-\Sigma))$ for $v\in B(\epsilon_{1}V_{1}^{\intercal},\epsilon_{2},\|\cdot\|)$, where $c_{z}$ is a constant such that $B(\boldsymbol{0}_{d},c_{z},\|\cdot\|)\subseteq\text{supp}(Z)$, and $\epsilon_{1},\epsilon_{2}>0$ satisfy that $\epsilon_{1}|V_{1}|+\epsilon_{2}<c_{z}$. Here, we define $\epsilon_{2}>0$ sufficiently small so that the determinant of $\text{Jacobian}(vv^{\intercal}V_{1}^{\intercal}-\Sigma V_{1}^{\intercal})>0$ for $v\in B(\epsilon_{1}V_{1}^{\intercal},\epsilon_{2},\|\cdot\|)$, and $v\to vv^{\intercal}V_{1}^{\intercal}-\Sigma V_{1}^{\intercal}$ is a one-to-one correspondence, by inverse function theorem. Therefore, the volume of $(V_{1}(vv^{\intercal}-\Sigma))$ for $v\in B(\epsilon_{1}V_{1}^{\intercal},\epsilon_{2},\|\cdot\|)$ can be calculated as

$$ $\displaystyle[\text{Volume $(V_{1}(vv^{\intercal}-\Sigma))$ for $v\in B(\epsilon_{1}V_{1}^{\intercal},\epsilon_{2},\|\cdot\|)$}]=\int_{v\in B(\epsilon_{1}V_{1}^{\intercal},\epsilon_{2},\|\cdot\|)}\big{|}\text{det(Jacobian}(V_{1}(vv^{\intercal}-\Sigma)))\big{|}dv>0.$ $$

Therefore, $\text{Volume}(V_{1}(vv^{\intercal}-\Sigma))$ where $v\in B(\epsilon_{1}V_{1}^{\intercal},\epsilon_{2},\|\cdot\|)$ is non-zero, so that we can find $d$ loss vectors $\{\ell_{i}\}_{i\in[d]}$ such that the vectors $\{V_{1}(\ell_{i}\ell_{i}^{\intercal}-\Sigma)\}_{i\in[d]}$ are linearly independent.
Hence, if we want to minimize [Equation 23](#A4.E23), either $A=\bm{O}_{d\times d}$ or $\beta=\boldsymbol{0}_{d}$ should hold. In both cases, [Equation 19](#A4.E19) can be re-written as

$$ $\displaystyle g(Z_{t};\mathbb{A},\beta,\mathbb{C},\delta):=\sum_{i=1}^{t}\mathbb{C}\ell_{i},$ $$

and this is covered by the original parametrization ([Equation 18](#A4.E18)) with $K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d}$.

Step 3. Calculating $\frac{\partial f}{\partial\mathbb{C}}$.

Now, we optimize over $\mathbb{C}$, by minimizing the following objective:

$$ $\displaystyle f(\mathbb{C}):$ $\displaystyle=\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}+R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|\right)^{2}$ $\displaystyle=\underbrace{\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\right)^{2}}_{(i)}+{2\mathbb{E}\left(\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\right)R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|\right)}+\mathbb{E}\left(R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|\right)^{2}$ $\displaystyle=\frac{T(T-1)}{2}\mathop{\mathrm{Tr}}\left(\mathbb{C}^{\intercal}\Sigma\mathbb{C}\Sigma\right)+{2\mathbb{E}\left(B\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\|\sum_{j=1}^{T}\ell_{j}\|\right)}+\mathbb{E}\left(R_{\Pi}\|\sum_{t=1}^{T}\ell_{t}\|\right)^{2}.$ $$

Here, $(i)$ can be calculated as follows:

$$ $\displaystyle\mathbb{E}$ $\displaystyle\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}^{\intercal}\mathbb{C}\ell_{i}\right)^{2}=\mathbb{E}\left(\sum_{t_{1}=1}^{T}\sum_{i_{1}=1}^{t_{1}-1}\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{i_{1}}^{\intercal}\mathbb{C}^{\intercal}\ell_{t_{1}}\ell_{i}^{\intercal}\mathbb{C}\ell_{i}\right)$ $\displaystyle\underset{(1)}{=}\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i_{1}=1}^{i-1}\sum_{i=1}^{t-1}\ell_{i_{1}}^{\intercal}\mathbb{C}^{\intercal}\ell_{i}\ell_{i}^{\intercal}\mathbb{C}\ell_{i}\right)=\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i_{1}=1}^{i-1}\sum_{i=1}^{t-1}\ell_{i_{1}}^{\intercal}\mathbb{C}^{\intercal}\Sigma\mathbb{C}\ell_{i}\right)$ $\displaystyle\underset{(2)}{=}\mathbb{E}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{k}^{\intercal}\mathbb{C}^{\intercal}\Sigma\mathbb{C}\ell_{i}\right)\underset{(3)}{=}\mathbb{E}\mathop{\mathrm{Tr}}\left(\sum_{t=1}^{T}\sum_{i=1}^{t-1}\mathbb{C}^{\intercal}\Sigma\mathbb{C}\ell_{i}\ell_{k}^{\intercal}\right)=\frac{T(T-1)}{2}\mathop{\mathrm{Tr}}\left(\mathbb{C}^{\intercal}\Sigma\mathbb{C}\Sigma\right),$ $$

since (1) holds because if $t_{1}\neq t$, we already know that $\mathbb{E}\ell_{t}=\mathbb{E}\ell_{t_{1}}=0$, (2) holds due to a similar reason, and (3) comes from $\mathop{\mathrm{Tr}}(AB)=\mathop{\mathrm{Tr}}(BA)$.

We calculate $\frac{\partial f(\mathbb{C})}{\partial\mathbb{C}}$:

$$ $\displaystyle\frac{\partial f(\mathbb{C})}{\partial\mathbb{C}}=T(T-1)\Sigma\mathbb{C}\Sigma+2R_{\Pi}\mathbb{E}\left(\|\sum_{j=1}^{T}\ell_{j}\|\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}\ell_{i}^{\intercal}\right).$ $$

Hence, the optimal $\mathbb{C}=-\frac{2R_{\Pi}}{T(T-1)}\Sigma^{-1}\mathbb{E}\left(\|\sum_{j=1}^{T}\ell_{j}\|\sum_{t=1}^{T}\sum_{i=1}^{t-1}\ell_{t}\ell_{i}^{\intercal}\right)\Sigma^{-1}.$

Now, we see that for the special case of $\Sigma=I$, we have $\mathbb{C}=-R_{\Pi}\mathbb{E}\left(\|\sum_{j=1}^{T}\ell_{j}\|\ell_{t}\ell_{i}^{\intercal}\right)$. If we calculate the $(a,b)$-coordinate of $\mathbb{C}$, we need to calculate

$$ $\displaystyle\mathbb{E}_{\ell}\left[\sqrt{\sum_{o=1}^{d}(\sum_{s=1}^{T}\ell_{so})^{2}}\ell_{ia}\ell_{kb}\right].$ $$

If $a\neq b$, then since $Z$ is symmetric, the term above becomes zero. Therefore, we only need to consider the case when $a=b$, which is $\mathbb{E}_{\ell}\left[\sqrt{\sum_{o=1}^{d}(\sum_{s=1}^{T}\ell_{so})^{2}}\ell_{ia}\ell_{ka}\right]$, and it will be the same value for all $a\in[d]$ since $\ell_{i}$’s coordinates are independent.

Now, we calculate the scale of $\mathbb{E}_{\ell}\left[\sqrt{\sum_{o=1}^{d}(\sum_{s=1}^{T}\ell_{so})^{2}}\ell_{i1}\ell_{k1}\right]$.
We have $Z:=\frac{\sum_{o=1}^{d-1}(\sum_{s=1}^{T}\ell_{so})^{2}}{T(d-1)}\overset{a.s.}{\to}1$ as $d\to\infty$ (by the law of large numbers) and we define $W:=\sum_{s\neq i,k}\ell_{s1}/\sqrt{T}$ which is independent of $\ell_{i1}$ and $\ell_{k1}$.

$$ $\displaystyle\mathbb{E}_{\ell}$ $\displaystyle\left[\sqrt{\sum_{o=1}^{d}(\sum_{s=1}^{T}\ell_{so})^{2}}\ell_{i1}\ell_{k1}\right]=\mathbb{E}_{Z,W,\ell_{i1},\ell_{k1}}\left[\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}+\ell_{k1})^{2}}\ell_{i1}\ell_{k1}\right]$ $\displaystyle=\mathbb{E}_{Z,W,\ell_{i1},\ell_{k1}\geq 0}\left[\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}+\ell_{k1})^{2}}\ell_{i1}\ell_{k1}-\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}-\ell_{k1})^{2}}\ell_{i1}\ell_{k1}\right]$ $\displaystyle=\mathbb{E}_{Z,W,\ell_{i1},\ell_{k1}\geq 0}\left[\frac{4(\sqrt{T}W+\ell_{i1})\ell_{k1}}{\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}+\ell_{k1})^{2}}+\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}-\ell_{k1})^{2}}}\ell_{i1}\ell_{k1}\right].$ $$

Taking $d\to\infty$, we have

$$ $\displaystyle\frac{\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}+\ell_{k1})^{2}}+\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}-\ell_{k1})^{2}}}{2\sqrt{Td}}\overset{d}{\to}1,$ $$

which further implies

$$ $\displaystyle\sqrt{Td}\frac{4(\sqrt{T}W+\ell_{i1})\ell_{k1}}{\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}+\ell_{k1})^{2}}+\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}-\ell_{k1})^{2}}}\ell_{i1}\ell_{k1}$ $\displaystyle\qquad\overset{d}{\to}\sqrt{Td}\frac{4(\sqrt{T}W+\ell_{i1})\ell_{k1}}{2\sqrt{Td}}\ell_{i1}\ell_{k1}=2(\sqrt{T}W+\ell_{i1})\ell_{i1}\ell_{k1}$ $$

as $d\to\infty$. Therefore,

$$ $\displaystyle\lim_{d\to\infty}\mathbb{E}_{Z,W,\ell_{i1},\ell_{k1}\geq 0}\left[{\sqrt{Td}}\frac{4(\sqrt{T}W+\ell_{i1})\ell_{k1}}{\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}+\ell_{k1})^{2}}+\sqrt{T(d-1)Z+(\sqrt{T}W+\ell_{i1}-\ell_{k1})^{2}}}\ell_{i1}\ell_{k1}\right]$ $\displaystyle=\mathbb{E}_{Z,W,\ell_{i1},\ell_{k1}\geq 0}\left[2(\sqrt{T}W+\ell_{i1})\ell_{i1}\ell_{k1}\right]=\mathbb{E}_{\ell_{i1},\ell_{k1}\geq 0}\left[\ell_{i1}^{2}\ell_{k1}\right]$ $$

which is a constant. The last equality came from the fact that $W$, $\ell_{i1}$, $\ell_{k1}$ are independent random variables, and expectation of $\ell_{i1}$ is zero. Therefore, the output of the single-layer linear self-attention provides us with online gradient descent with step-size $\Theta(R_{\Pi}/\sqrt{Td})$. In the online learning literature, we usually set the gradient step size as $\Theta(R_{\Pi}/\sqrt{Td})$ (Hazan, , Theorem 3.1), which is consistent with the result above.
∎

###### Proof.

### D.6 Empirical Validation of Theorem 3 and Theorem 4

We now provide empirical validations for [Theorem 3](#Thmtheorem3) and [Theorem 4](#Thmtheorem4). We provide the training details and the results as follows.

#### D.6.1 Empirical Validation of Theorem 3

Our model architecture is defined as follows: the number of layers $T$ is set to 30 and the dimensionality $d$ to 32, with the loss vector $\ell_{i}$’s distribution $Z$ following a standard normal distribution $\mathcal{N}(0,1)$. During training, we conducted 40,000 epochs with a batch size of 512. We employed the Adam optimizer, setting the learning rate to 0.001. We initialized the value, query, and key vectors ($v_{c},q_{c},k_{c}$) as zero vectors.

Our empirical analysis aims to demonstrate that the optimized model inherently emulates online gradient descent. To illustrate this, we will focus on two key convergence properties: $K^{\intercal}Q$ approaching the zero matrix $\boldsymbol{O}_{d\times d}$ and $V$ converging to $a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$, where $a$ and $b$ are constants in $\mathbb{R}$. The conditions $K^{\intercal}Q=\boldsymbol{O}_{d\times d}$ and $V=a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$ imply that the function $g(Z_{t};V,Q,K)=\sum_{i=1}^{t}(b-a)\ell_{i}$, effectively emulating the process of an online gradient descent method. We repeated 10 times of the experiments. For verifying $K^{\intercal}Q=\boldsymbol{O}_{d\times d}$, we will measure Frobenius norm ($\|\cdot\|_{F}$) of $K^{\intercal}Q$. Also for measuring the closeness of $V$ and $a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$, we will measure $\min_{a,b\in\mathbb{R}}\|V-(a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d})\|_{F}/b$. The results are demonstrated in the first plot of [Figure 15](#A4.F15).

#### D.6.2 Empirical Validation of Theorem 4

We now focus on two key convergence properties: $K^{\intercal}(Q\boldsymbol{1}_{d}+q_{c})$ approaching the zero vector $\boldsymbol{0}_{d}$ and $V$ converging to $a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$, where $a$ and $b$ are constants in $\mathbb{R}$. The conditions $K^{\intercal}(Q\boldsymbol{1}_{d}+q_{c})=\boldsymbol{0}_{d}$ and $V=a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$ imply that the function $g(Z_{t};V,Q,K)=\sum_{i=1}^{t}(b-a)\ell_{i}$, effectively emulating the process of an online gradient descent method. We repeated 10 times. For verifying $K^{\intercal}(Q\boldsymbol{1}_{d}+q_{c})=\boldsymbol{0}_{d}$, we will measure 2-norm of $K^{\intercal}(Q\boldsymbol{1}_{d}+q_{c})$. Also for measuring the closeness of $V$ and $a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$, we will measure $\min_{a,b\in\mathbb{R}}\|V-(a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d})\|_{F}/b$. The results are demonstrated in the second plot of [Figure 15](#A4.F15).

Figure: Figure 15: Empirical validation of [Theorem 3](#Thmtheorem3) (top), [Theorem 4](#Thmtheorem4) (middle), and [6](#Thmtheorem6) (bottom). The observed convergence in [Theorem 3](#Thmtheorem3) and [6](#Thmtheorem6)’s result suggests that configuration in [Theorem 3](#Thmtheorem3) and [6](#Thmtheorem6) are not only the local optimal point, but it has the potential as being the global optimizer.
Refer to caption: /html/2403.16843/assets/x25.png

### D.7 Discussions on the Production of FTRL with Entropy Regularization

Now, we will consider projecting a single-layer linear self-attention model into a constrained domain such as a simplex, which is more amenable to the Experts Problem setting.
To this end, we consider the following parameterization by adding an additional *non-linear* structure for the single-layer linear self-attention:

$$ $\displaystyle g(Z_{t};V,K,Q,v_{c},k_{c},q_{c})=\texttt{Operator}\left(\sum_{i=1}^{t}(V\ell_{i}+v_{c})((K\ell_{i}+k_{c}))^{\intercal}\cdot(Qc+q_{c}))\right),$ (24) $$

where the Operator denotes projection to the convex set.

###### Conjecture 6 .

Assume $\Sigma=I$. Then, the configuration that $K^{\intercal}(Qc+q_{c})=v_{c}=\boldsymbol{0}_{d}$ and $V=\widetilde{\Omega}\left(-\frac{1}{\sqrt{nd}}\right)I_{d\times d}$ is a first-order stationary point of [Equation 3](#S5.E3) with $N=1$ and $h(x)=x^{2}$ when $\text{LLM}_{\theta}$ is parameterized with [Equation 24](#A4.E24), $\texttt{Operator}=\texttt{Softmax}$, and $\Pi=\Delta(\mathcal{A})$. This configuration performs FTRL with an entropy regularizer which is a no-regret algorithm.

We provide an idea for proving the conjecture, together with its numerical validation. Also, we have observed in [Figure 15](#A4.F15) that [Theorem 3](#Thmtheorem3) and [6](#Thmtheorem6) might also be a global optimizer, as training results have provided the configuration that [Theorem 3](#Thmtheorem3) and [6](#Thmtheorem6) have suggested.

To be specific, we will consider

$$ $\displaystyle f(V,a,\beta,v_{c})$ $\displaystyle=\mathbb{E}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)^{2}$ $$

and will try to prove that $a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI$ is a first-order stationary point.

Step 1. Calculating $\frac{\partial f}{\partial v_{c}}$.

We use the following formula: for $x\in[d]$ and $t\geq 2$, we have

$$ $\displaystyle\frac{\partial}{\partial v_{cx}}\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\frac{\partial}{\partial v_{cx}}\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\sum_{i=1}^{t}\left(a^{\intercal}\ell_{i}\ell_{i}^{\intercal}e_{x}+\beta\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=t\beta\exp(v\beta)\exp(\beta k\sum_{i=1}^{t}\ell_{iy}),$ $$

and for $t=1$, $\frac{\partial}{\partial v_{cx}}\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=0$, so we can use the same formula with $t\geq 2$. Thus, we have

$$ $\displaystyle\frac{\partial}{\partial v_{cx}}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\beta\exp(v\beta)$ $\displaystyle\qquad\sum_{t=1}^{T}t\sum_{s=1}^{d}\ell_{ts}\frac{\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)-\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)}{\left(\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}\beta V\ell_{j}\right)\right)^{2}}$ $\displaystyle=0.$ $$

Therefore,

$$ $\displaystyle\frac{\partial f(V,a,\beta,v_{c})}{\partial v_{cx}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\frac{\partial}{\partial v_{cx}}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)\Bigg{]}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=0.$ $$

Step 2. Calculating $\frac{\partial f}{\partial V}$.

The following formula will be used for calculating $\frac{\partial f}{\partial V}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$: for $r,c\in[d]$, we have

$$ $\displaystyle\frac{\partial}{\partial V_{rc}}$ $\displaystyle\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\frac{\partial}{\partial V_{rc}}\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(\sum_{i=1}^{t}k\beta\ell_{iy}+v\beta\right)\sum_{i=1}^{t}\beta\boldsymbol{1}(y=r)\ell_{ic}.$ $$

Therefore,

$$ $\displaystyle\frac{\partial f(V,a,\beta,v_{c})}{\partial V_{rc}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\frac{\partial}{\partial V_{rc}}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)\Bigg{]}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}+v\beta\right)}{\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta V\ell_{jy}+v\beta\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\Biggl{(}\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\sum_{j=1}^{t-1}\beta\boldsymbol{1}(s=r)\ell_{jc}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}+v\beta\right)\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}+v\beta\right)}{\left(\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}+v\beta\right)\right)^{2}}$ $\displaystyle\qquad\qquad-\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}+v\beta\right)\sum_{y=1}^{d}\left(\sum_{j=1}^{t-1}\beta\boldsymbol{1}(y=r)\ell_{jc}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}+v\beta\right)\right)}{\left(\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}+v\beta\right)\right)^{2}}\Biggr{)}\Bigg{]}$ $\displaystyle=\beta\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)}{\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta V\ell_{jy}\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\qquad\qquad\Biggl{(}\underbrace{\frac{\sum_{t=1}^{T}\sum_{j=1}^{t-1}\sum_{y=1}^{d}\ell_{tr}\ell_{jc}\exp\left(\beta k\sum_{j=1}^{t-1}\ell_{jr}\right)\exp\left(\beta k\sum_{j=1}^{t-1}\ell_{jy}\right)}{\left(\sum_{y=1}^{d}\exp\left(\beta k\sum_{j=1}^{t-1}\ell_{jy}\right)\right)^{2}}}_{(i)}$ $\displaystyle\qquad\qquad\qquad\qquad-\underbrace{\frac{\sum_{t=1}^{T}\sum_{j=1}^{t-1}\sum_{y=1}^{d}\ell_{ty}\ell_{jc}\exp\left(\beta k\sum_{j=1}^{t-1}\ell_{jr}\right)\exp\left(\beta k\sum_{j=1}^{t-1}\ell_{jy}\right)}{\left(\sum_{y=1}^{d}\exp\left(\beta k\sum_{j=1}^{t-1}\ell_{jy}\right)\right)^{2}}}_{(ii)}\Biggr{)}\Bigg{]}.$ $$

We can observe the followings: 1) if $r_{1}\neq c_{1}$ and $r_{2}\neq c_{2}$, $\frac{\partial f}{\partial V_{r_{1}c_{1}}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=\frac{\partial f}{\partial V_{r_{2}c_{2}}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ holds, and 2) $\frac{\partial f}{\partial V_{r_{1}r_{1}}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=\frac{\partial f}{\partial V_{r_{2}r_{2}}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$.

Step 3. Calculating $\frac{\partial f}{\partial\beta}$.

The following formula will be used for calculating $\frac{\partial f}{\partial\beta}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$:

$$ $\displaystyle\frac{\partial}{\partial\beta}$ $\displaystyle\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\frac{\partial}{\partial\beta}\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=tv\beta\exp\left(\sum_{i=1}^{t}k\beta\ell_{iy}+v\beta\right).$ $$

Further, we have

$$ $\displaystyle\frac{\partial}{\partial\beta}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=v\beta\exp(v\beta)$ $\displaystyle\qquad\sum_{t=1}^{T}t\sum_{s=1}^{d}\ell_{ts}\frac{\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)-\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)}{\left(\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}\beta V\ell_{j}\right)\right)^{2}}$ $\displaystyle=0.$ $$

Step 4. Calculating $\frac{\partial f}{\partial a}$.

Note that

$$ $\displaystyle\frac{\partial}{\partial a_{x}}$ $\displaystyle\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\frac{\partial}{\partial a_{x}}\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(e_{y}^{\intercal}\sum_{i=1}^{t}(V\ell_{i}\ell_{i}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{i}+v_{c}\beta)\right)\sum_{i=1}^{t}\left(e_{y}^{\intercal}V\ell_{i}\ell_{i}^{\intercal}e_{x}+e_{y}^{\intercal}v_{c}\ell_{i}^{\intercal}e_{x}\right)\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\exp\left(\sum_{i=1}^{t}\beta k\ell_{iy}+v\beta\right)\sum_{i=1}^{t}(k\ell_{iy}\ell_{ix}+v\ell_{ix}).$ $$

Therefore,

$$ $\displaystyle\frac{\partial f(V,a,\beta,v_{c})}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\frac{\partial}{\partial a_{x}}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(e_{s}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}{\sum_{y=1}^{d}\exp\left(e_{y}^{\intercal}\sum_{j=1}^{t-1}(V\ell_{j}\ell_{j}^{\intercal}a+(\beta V+v_{c}a^{\intercal})\ell_{j}+v_{c}\beta)\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)\Bigg{]}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ $\displaystyle=\mathbb{E}\Bigg{[}\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)}{\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\Biggl{(}\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\sum_{j=1}^{t-1}(k\ell_{js}\ell_{jx}+v\ell_{jx})\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)}{\left(\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\right)^{2}}$ $\displaystyle\qquad\qquad-\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)\sum_{y=1}^{d}\left(\sum_{j=1}^{t-1}(k\ell_{jy}\ell_{jx}+v\ell_{jx})\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\right)}{\left(\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\right)^{2}}\Biggr{)}\Bigg{]}$ $\displaystyle=\mathbb{E}\Bigg{[}k\left(\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)}{\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)}-\min_{s}\sum_{t=1}^{T}\ell_{ts}\right)$ $\displaystyle\qquad\Biggl{(}\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\sum_{j=1}^{t-1}\ell_{js}\ell_{jx}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)}{\left(\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\right)^{2}}$ $\displaystyle\qquad\qquad-\sum_{t=1}^{T}\sum_{s=1}^{d}\ell_{ts}\frac{\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{js}\right)\sum_{y=1}^{d}\left(\sum_{j=1}^{t-1}\ell_{jy}\ell_{jx}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\right)}{\left(\sum_{y=1}^{d}\exp\left(\sum_{j=1}^{t-1}\beta k\ell_{jy}\right)\right)^{2}}\Biggr{)}\Bigg{]}$ $$

Note that the value does not depend on $x$, which means that $\frac{\partial f}{\partial a}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=\widetilde{c}\boldsymbol{1}_{d}$ for some constant $\widetilde{c}$.

###### Conjecture 6 .

#### D.7.1 Numerical Analysis of Step 2 and Step 4

In Steps 2 and 4 above, we were not able to show that a $k$ whose value becomes zero exists. We hence provide some empirical evidence here. First, we attach the estimated $\frac{\partial f}{\partial V_{rc}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ ($r\neq c$), $\frac{\partial f}{\partial V_{rr}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$, $\frac{\partial f}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ and $\frac{\partial f}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ graph with respect to $k$ value when $\ell_{ts}\sim\text{Unif}([0,1])$ for all $t\in[T],s\in[d]$. While the graph of $\frac{\partial f}{\partial V}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ is not stable, we can see that $k$ for $\frac{\partial f}{\partial V_{rc}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=0$, $\frac{\partial f}{\partial V_{rr}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=0$ and $\frac{\partial f}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}=0$ is very similar in [Figure 16](#A4.F16). We used the Monte Carlo estimation of $1,000,000$ times.

Figure: Figure 16: Calculation of $20\frac{\partial f}{\partial V_{rc}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ ($r\neq c$)(red), $20\frac{\partial f}{\partial V_{rr}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ (blue), and $\frac{\partial f}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ (black). We experimented with $n\in[4,9]$ and $d\in[4,9]$. The figure might indicate that $\beta k$ that makes the derivative zero of $\frac{\partial f}{\partial V_{rc}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ ($r\neq c$), $\frac{\partial f}{\partial V_{rr}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$, and $\frac{\partial f}{\partial a_{x}}\bigg{|}_{a=\boldsymbol{0}_{d},v_{c}=v\boldsymbol{1}_{d},V=kI}$ would coincide.
Refer to caption: /html/2403.16843/assets/figs/monte_carlo_estimate_f.png

#### D.7.2 Empirical Validation

Our model architecture is defined as follows: the number of layers $T$ is set to 30 and the dimensionality $d$ to 32, with the loss vector $l_{i}$’s distribution $Z$ following a standard normal distribution $\mathcal{N}(0,1)$. During training, we conducted 40,000 epochs with a batch size of 512. We employed the Adam optimizer, setting the learning rate to 0.001. We focus on two key convergence properties: $K^{\intercal}(Q\boldsymbol{1}+q_{c})$ approaching the zero vector $\boldsymbol{0}_{d}$ and $V$ converging to $a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$, where $a$ and $b$ are constants in $\mathbb{R}$. The conditions $K^{\intercal}(Q\boldsymbol{1}+q_{c})=\boldsymbol{0}_{d}$ and $V=a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$ imply that the function $g(Z_{t};V,Q,K)=\sum_{i=1}^{t}(b-a)l_{i}$, effectively emulating the process of an online gradient descent method. We repeated 10 times. For verifying $K^{\intercal}(Q\boldsymbol{1}+q_{c})=\boldsymbol{0}_{d}$, we will measure 2-norm of $K^{\intercal}(Q\boldsymbol{1}+q_{c})$. Also for measuring the closeness of $V$ and $a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d}$, we will measure $\min_{a,b\in\mathbb{R}}\|V-(a\boldsymbol{1}_{d}\boldsymbol{1}^{\intercal}_{d}+bI_{d\times d})\|_{2,2}/b$. The results are demonstrated in the third plot of [Figure 15](#A4.F15).

### D.8 Training Details of Section 5.4

We provide the training details of [Section 5.4](#S5.SS4).
For the multi-layer Transformer training, we used 4 layers, 1 head Transformer. For both single-layer and multi-layer, we employed the Adam optimizer, setting the learning rate to 0.001. During training, we conducted 2,000 epochs with a batch size 512. Moreover, when we trained for the loss sequences with the predictable trend, we used 4 layers, 1 head Transformer. For both single-layer and multi-layer, we employed the Adam optimizer, setting the learning rate to 0.001. During training, we conducted 9,000 epochs with a batch size of 512.

### D.9 Ablation Study on Training Equation 3

In this section, we provide an ablation study that changes $N$ and $k$ in [Equation 3](#S5.E3). To be specific, we will set $N=1,2,4$, $f(x,k)=\max(x,0)^{k}$, $h(x)=\max(x,0)^{2}$, and $k=1,2$. For the multi-layer Transformer training, we used 4 layers and 1 head Transformer. For both single-layer and multi-layer, we employed the Adam optimizer, setting the learning rate to 0.001. During training, we conducted 2,000 epochs with a batch size of 512. We experimented on the randomly generated loss sequences. Especially, we used the uniform loss sequence ($\ell_{t}\sim\text{Unif}([0,10]^{2})$), with the results in [Figure 17](#A4.F17) and [Figure 18](#A4.F18); and the Gaussian loss sequence ($\ell_{t}\sim\mathcal{N}(5\cdot\boldsymbol{1}_{2},I)$), with the results in [Figure 19](#A4.F19) and [Figure 20](#A4.F20).

Figure: Figure 17: Ablation study for the uniform loss sequence trained with single-layer self-attention layer and Softmax projection.
Refer to caption: /html/2403.16843/assets/x28.png

Figure: Figure 18: Ablation study for the uniform loss sequence trained with multi-layer self-attention layer and Softmax projection.
Refer to caption: /html/2403.16843/assets/x29.png

Figure: Figure 19: Ablation study for the Gaussian loss sequence trained with single-layer self-attention layer and Softmax projection.
Refer to caption: /html/2403.16843/assets/x30.png

Figure: Figure 20: Ablation study for the Gaussian loss sequence trained with single-layer self-attention layer and Softmax projection.
Refer to caption: /html/2403.16843/assets/x31.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors study instruction-tuned models. These typically undergo multiple training stages: a pre-training stage consisting of next-token prediction, and multiple fine-tuning stages that align models with desired behavior.

Have the authors considered comparing the behavior of LLMs at various stages of the training process? This question strikes me as relevant in the context of the discussion in Section 4.2, which makes assumption on the pre-training stage specifically.
soundness: 2
strengths: - Given the amount of information conveyed, overall the paper is relatively well written, with a sound structure that helps guiding the reader through the large number of findings.
- The contributions appear to be quite substantial. Not being a domain expert, I cannot make a confident judgment as to originality and significance.
- There is a good combination of theory and experiments. The experiments are comprehensive, and the theory appears to be relevant & meaningfully complement the empirical findings.
summary: This paper studies large language models from a game-theoretic perspective. Observing that LLM-agents are becoming prevalent, they investigate the performance of such LLM agents in terms of regret.

Empirical findings include:

- LLMs exhibit sublinear regret in certain online learning problems
- equilibria emerge when LLM agents interact through repeated games

The authors also develop theory that is consistent with the empirical observations. Finally, the authors propose a new loss function tailored towards regret minimization. They study models optimized under this loss function both theoretically and empirically.
weaknesses: - There is a case to be made that the authors are trying to describe too many findings in the main paper. Some figures are hard to read (Figure 3 is a good example of this). The paper ends rather abruptly, with what is essentially a list of pointers to the supplementary material. It appears to me that the layout is deliberately compressed to gain space (e.g., page 10: space above header of Sec. 5.4, space around Eq. 5.3, math fontsize in Thm 5.1).
    - I am genuinely impressed by how comprehensive the study is (there are 77 pages when including the supplementary material). However, I think the main text would benefit from being simplified by omitting certain results. Remember that this is a conference submission with limited space.
    - Again, _conditioned on the set of results presented in the main text_, I don't see many opportunities to improve the organization and presentation, which I find rather good.
- Proposition 1 is problematic. As far as I understand, this proposition rests on an unwritten assumption that the indicators for _difference of regrets is > 0_ are independent. I don't see why this would hold?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the authors clarify the use of the term "no-regret" in this context?
2. In structured, non-adversarial environments (such as linear or sine-trend losses), GPT-4 appears to outperform traditional no-regret algorithms like FTRL/FTPL. Does this suggest that the model may be doing more than no-regret learning, possibly adapting to or fitting the specific trends in the loss sequences through in-context learning?
soundness: 4
strengths: The strengths of the paper include: 

- **Comprehensive Empirical Analysis:** The authors conduct a thorough empirical study across diverse environment setups and various LLM models. They evaluate LLM performance in both standard online learning scenarios and multi-agent games, identifying cases where LLMs exhibit no-regret behaviors and outperform classical algorithms like FTRL and FTPL. Importantly, the authors also examine situations where LLMs, such as GPT, fail to achieve sublinear regret in highly adaptive or adversarial environments. This balanced examination provides valuable insights into the conditions under which LLMs succeed or struggle, helping us better understand the decision-making processes and strategic adaptations LLMs employ in online game contexts. 

- **Conduct Ablation Study:** The authors conduct an ablation study to systematically examine the contributions of various components in their proposed framework.

- **Theoretical Justification for No-Regret Behaviors:** The authors offer a theoretical explanation on how next-token prediction pretraining enables LLMs to develop no-regret behaviors in online learning. Under certain structured noise assumption, the optimal behaviors of the LLM under the next-token prediction loss can be related to FTPL.

- The no-regret loss seems to be novel, and enhances the model's performance.
summary: The authors investigate whether LLMs can exhibit no-regret behavior in online learning scenarios. Their main contributions include:

- **Empirical Validation:** The authors empirically show that both commercial and open-source LLMs can achieve no-regret learning in various environments, including dynamic settings and multi-agent games. They demonstrate that LLMs often outperform traditional no-regret algorithms like FTRL and FTPL, with the models effectively incorporating randomization in decision-making.
- **Limitations in Adversarial Scenarios:** The study identifies cases where advanced LLMs, such as GPT models, struggle to achieve sublinear regret, particularly in highly adaptive or adversarial environments. 
- **Theoretical Insights and Enhancements:** The authors provide a theoretical framework explaining how next-token prediction during pretraining enables LLMs to develop capabilities aligned with no-regret algorithms. They also introduce a novel "regret-loss" training objective designed to further strengthen the no-regret properties of LLMs, even in the absence of optimal action labels.
weaknesses: I found the following weakness points: 

- **Ambiguity in No-Regret Definition and Experimental Setup:** The standard definition of "no-regret" in online learning typically requires that an algorithm achieves sublinear regret for all possible loss sequences, including adversarial ones. However, the authors claim that the LLM behaves as a no-regret algorithm based primarily on experiments involving uniform or Gaussian losses and losses with specific trends, which are not adversarial. This raises some confusion, as these setups do not fully reflect the robustness expected in no-regret algorithms. In fact, the experiments on the "Adaptive loss sequence" seem more directly relevant to evaluating this claim.

- **Lack of Explanation for Model Performance:** Following the previous point, while the authors show that the LLM outperforms traditional no-regret algorithms like FTRL and FTPL, there is limited insight into why this occurs. For instance, the model may be learning the underlying distribution of the loss or fitting specific loss trends, but this is not explicitly demonstrated. Additional analysis on how GPT models achieve superior performance would provide a more comprehensive understanding of these results.

- **Theoretical Limitation**: In Theorem 5.2, you only show the results for $N=1$. If my understanding is correct, this basically means that there is no maximum taken across the input loss sequence $\ell_{1:t}$. Is this also the setup for training the single/multi-layer transformer model under the noisy alternating/adaptive loss as you show in Figure 3.4, or did you use larger $N$ to produce the results? I didn't find the experiment setup for these results in Figure 3.4.

- Building on the previous point, there is one minor thing. I feel that Theorem 4.1 may not fully capture the behavior of LLMs. The empirical results show that GPT models outperform standard no-regret algorithms in more structured tasks but perform significantly worse in adversarial settings. However, what this theorem suggests is that the model after pretraining performs something similar to FTPL. This discrepancy suggests that the theoretical framework might not align with the model's observed strengths and weaknesses across different environments.

However, I still find the contributions to be significant. Hence, I'm open to raising my scores upon the concerns being addressed.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. When using GPT4 Turbo and GPT o1 for online decision-making, could the assumption on optimal action labels be relaxed?

2. Why is the regret performance of GPT4 better than single/multi-layer models in the single agent setting, yet vice versa in the multi-agent setting? What factors contribute to this discrepancy in its effectiveness across different settings?

3. What is the technical challenge in the proof of Theorem 5.2? If the result goes beyond simply applying standard learning theory arguments, it's worth highlighting the proof techniques used or created.
soundness: 4
strengths: 1. The paper explores an interesting aspect of using LLM in decision-making tasks. Experimental results are provided.

2. It is interesting that Theorem 5.2 shows a single-layer (linear) self-attention transformer is enough to model no-regret learning algorithms.

3. The paper provides an extensive appendix. The appendix is rich with additional proofs, experimental details, and supplementary discussions. I didn't read all the experimental results and proofs.
summary: The paper discusses the use of LLMs for decision-making in both single and multi-agent settings. Their study focuses on understanding the limitations of LLM agents in interactive environments by using regret as a performance metric. Initial empirical investigations assess no-regret behaviors in non-stochastic online learning and the emergence of equilibria in repeated game scenarios. The paper provides theoretical insights into these behaviors based on assumptions about the data’s origin and the rationality of human decision-makers involved in pre-training.
weaknesses: 1. It seems not that natural to apply LLMs to solve online decision-making problems, given the existence of classic and well-established algorithms. What are the potential applications? The paper needs to be better motivated to help readers understand the contributions. While the application of LLMs in online decision-making presents intriguing possibilities, the practical implications are not immediately clear. To better engage readers and clarify the significance of this research, the paper could benefit from a more compelling motivation. Specifically, detailing potential real-world applications and explicitly connecting these to the experimental findings would enhance the paper’s relevance and impact.

It's worth considering providing specific examples of potential real-world applications where using LLMs for online decision-making could offer advantages over classical algorithms such as FTPL.

2. The paper does not address the capabilities of state-of-the-art LLM APIs like GPT4 Turbo and o1. Discussing their performance might also offer valuable insights into the scalability and applicability of the proposed methods to more complex decision-making tasks. More detailed analysis of this discrepancy are needed.



Overall, I'd like to raise my score if case studies on real-world application can be provided, with both GPT-4 Turbo and o1 evaluated.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How sensitive are the findings to the choice of synthetic games (win-win, prisoner’s dilemma, unfair, cyclic, biased, and second best) used for experiments?
2. How do the authors validate the importance of the research problem:
    
    "Can we examine and better understand the online and strategic decision-making behaviors of LLMs through the lens of regret"?

    The theoretical results, while rigorous, may feel detached from practical applications. The paper's findings on regret behavior might not translate into concrete benefits for LLM-based systems in the real world, potentially rendering the problem it addresses less relevant. This could make the work feel more like a theoretical exercise than a practical contribution to improving LLM capabilities.
soundness: 3
strengths: 1. It offers a theoretical framework for understanding no-regret behaviors in LLMs and introduces a novel training objective (regret-loss) with provable guarantees.
2. It includes a wide range of experiments with different synthetic & toy-case settings (arbitrary, non-stationary, and bandit feedback environments), comparing LLMs with established no-regret algorithms.
summary: This paper investigates the performance of llm agents in online learning and game-theoretic scenarios. It empirically evaluates no-regret behaviors in pre-trained LLMs like GPT-4 across various online learning settings and repeated games, comparing them with classical no-regret algorithms. The authors introduce a new unsupervised training loss called "regret-loss," designed to enhance no-regret behavior without requiring action labels. It provides theoretical insights connecting pre-training to no-regret algorithms, and presents experimental results validating the proposed approaches.
weaknesses: 1. Misalignment of Framework and Problem: The application of online learning and game theory to evaluate LLM agents seems like over-engineering. LLMs are not explicitly designed for decision-making in the classical sense that these fields address (e.g., making repeated decisions based on feedback from an adversarial environment). Using regret minimization as a benchmark for LLM behavior could be seen as an attempt to impose a theoretical construct that may not provide meaningful insights into the actual strengths and weaknesses of LLMs.
2. Lack of Practical Relevance: If the focus is on understanding LLMs' interactive capabilities, other evaluation metrics might be more meaningful. For instance, metrics derived from natural language understanding or task-specific performance might offer more concrete insights into how LLMs perform as interactive agents. The notion of regret, which is central to online learning and game theory, may not be as intuitive or impactful in the context of LLMs used for dialogue, or other applications. There could be alternative frameworks that align better with evaluating and enhancing LLMs. For example, reinforcement learning, interactive learning environments, or even human-in-the-loop evaluations might offer more practical insights than regret-based analysis.

Meta-review: Summary:
This work explores the use of large language model (LLM) agents in playing games. The authors conduct experiments across several LLM instances and problem settings, delivering key insights into LLM performance in gaming scenarios, such as demonstrating that most LLM agents can achieve no-regret performance. Furthermore, the authors propose a distributional assumption, the quantal response model, to characterize LLM response distributions and validate its effectiveness through experiments.

Strengths:

- The paper provides a theoretical framework to understand no-regret behaviors in LLMs and introduces a novel training objective, the regret-loss, with provable guarantees.
- The study features a diverse range of experiments, including synthetic and toy-case settings (arbitrary, non-stationary, and bandit feedback environments), and compares LLMs with well-established no-regret algorithms.

Weaknesses:

The presentation can be improved. Notably, the manuscript is approximately 80 pages long, which is unusually lengthy for a conference submission and may hinder accessibility.

Recommendation:
I recommend acceptance based on the strengths mentioned above. Despite the need for a more concise presentation, the paper offers valuable theoretical insights and empirical findings that contribute significantly to the understanding of LLMs in gaming scenarios.

Decision: Accept (Poster)