SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zaiwei Chen $^{1,\ast}$ , Kaiqing Zhang $^{2}$ , Eric Mazumdar $^{1,\dagger}$ , Asuman Ozdaglar $^{3}$ , Adam Wierman $^{1,\ddagger}$ , CMS, Caltech, *zchen458@caltech.edu,  $^\dagger$ mazumdar@caltech.edu,  $^\ddagger$ adamw@caltech.edu  
 $^{2}$ ECE & ISR, University of Maryland, College Park, kaiqing@umd.edu  
 $^{3}$ EECS, MIT, asuman@mit.edu
In this work, we study two-player zero-sum stochastic games and develop a variant of the smoothed best-response learning dynamics that combines independent learning dynamics for matrix games with the minimax value iteration for stochastic games. The resulting learning dynamics are payoff-based, convergent, rational, and symmetric between the two players. Our theoretical results present to the best of our knowledge the first last-iterate finite-sample analysis of such independent learning dynamics. To establish the results, we develop a coupled Lyapunov drift approach to capture the evolution of multiple sets of coupled and stochastic iterates, which might be of independent interest.
Recent years have seen remarkable successes in reinforcement learning (RL) in a variety of applications, such as board games [1], autonomous driving [2], robotics [3], and city navigation [4]. A common feature of these applications is that there are multiple decision makers interacting with each other in a common environment. Although empirical successes have shown the potential of multi-agent reinforcement learning (MARL) [5, 6], the training of MARL agents is largely based on heuristics and parameter tuning and, therefore, is not always reliable. In particular, many practical MARL algorithms are directly extended from their single-agent counterparts and lack guarantees because of the adaptive strategies of multiple agents.
A growing literature seeks to provide theoretical insights to substantiate the empirical success of MARL and inform the design of efficient and provably convergent algorithms. Work along these lines can be broadly categorized into work on cooperative MARL where agents seek to reach a common goal [7-10], and work on competitive MARL where agents have individual (and possibly misaligned) objectives [11-22]. While some earlier work focused on providing guarantees on the asymptotic convergence, the more recent ones share an increasing interest in understanding the finite-time/sample behavior. This follows from a line of recent advances in establishing finite-sample guarantees of single-agent RL algorithms, see e.g., [23-26] and many others.
In this paper, we focus on the benchmark setting of two-player<sup>1</sup> zero-sum stochastic games, and develop best-response-type learning dynamics with provable finite-sample guarantees. Crucially, our learning dynamics are independent (requiring no coordination between the agents in learning) and rational (each agent will converge to the best response to the opponent if the opponent plays an (asymptotically) stationary policy [27]), and therefore capture learning in settings with multiple game-theoretic agents. Indeed, learning dynamics with self-interested agents should not enforce information communication or coordination among agents. Furthermore, we focus on the more challenging but practically relevant setting of payoff-based learning, where each agent can only
Hereafter, we may use player and agent interchangeably.
observe their realized payoff at each stage, without observing the policy or even the action taken by the opponent. For learning dynamics with such properties, we establish to the best of our knowledge the first last-iterate finite-sample guarantees. We detail our contributions as follows.
We first consider zero-sum matrix games and provide the last-iterate finite-sample guarantees for the smoothed best-response dynamics proposed in [28]. Then, we extend the algorithmic idea to the setting of stochastic games and develop an algorithm called value iteration with smoothed best-response dynamics (VI-SBR) that also enjoys last-iterate finite-sample convergence.
Two-Player Zero-Sum Matrix Games. We start with the smoothed best-response dynamics in [28] and establish the last-iterate finite-sample bounds when using stepsizes of various decay rates. The result implies a sample complexity of  $\mathcal{O}(\epsilon^{-1})$  in terms of the last iterate to find the Nash distribution [29], which is also known as the quantal response equilibrium in the literature [30]. To our knowledge, this is the first last-iterate finite-sample result for best-response learning dynamics that are payoff-based, rational, and symmetric in zero-sum matrix games.
Two-Player Zero-Sum Stochastic Games. Building on the algorithmic ideas for matrix games, we develop best-response-type learning dynamics for stochastic games called VI-SBR, which uses a single trajectory of Markovian samples. Our learning dynamics consist of two loops and can be viewed as a combination of the smoothed best-response dynamics for an induced auxiliary matrix game (conducted in the inner loop) and an independent way of performing minimax value iteration (conducted in the outer loop). In particular, in the inner loop, the iterate of the outer loop, i.e., the value function, is fixed, and the players learn the approximate Nash equilibrium of an auxiliary matrix game induced by the value function; then the outer loop is updated by approximating the minimax value iteration updates for the stochastic game, with only local information.
We establish the last-iterate finite-sample bounds for VI-SBR when using both constant step sizes and diminishing step sizes of  $\mathcal{O}(1 / k)$  decay rate. To the best of our knowledge, this appears to be the first last-iterate finite-sample analysis of best-response-type independent learning dynamics that are convergent and rational for stochastic games. Most existing MARL algorithms are either symmetric across players, but not payoff-based, e.g., [31-35], or not symmetric and thus not rational, e.g., [14, 36-38], or do not have last-iterate finite-time/sample guarantees, e.g., [39, 15, 40].
1.2 Challenges & Techniques
The main challenge in analyzing our learning dynamics is that it maintains multiple sets of stochastic iterates and updates them in a coupled manner. To overcome this challenge, we develop a novel coupled Lyapunov drift approach. Specifically, we construct a Lyapunov function for each set of the stochastic iterates and establish a Lyapunov drift inequality for each. We then carefully combine the coupled Lyapunov drift inequalities to establish the finite-sample bounds. Although a more detailed analysis is provided in the appendices, we briefly give an overview of the main challenges in analyzing the payoff-based independent learning dynamics in stochastic games, as well as our techniques to overcome them.
Time-Inhomogeneous Markovian Noise. The fact that our learning dynamics are payoff-based presents major challenges in handling the stochastic errors in the update. In particular, due to the best-response nature of the dynamics, the behavior policy for sampling becomes time-varying. In fact, the sample trajectory used for learning forms a time-inhomogeneous Markov chain. This makes it challenging to establish finite-sample guarantees, as time-inhomogeneity prevents us from directly exploiting the uniqueness of stationary distributions and the fast mixing of Markov chains. Building on existing work [23, 24, 41, 42], we overcome this challenge by tuning the algorithm parameters (in particular, the stepsizes) and developing a refined conditioning argument.
Non-Zero-Sum Payoffs Due to Independent Learning. As illustrated in Section 1.1, the inner loop of VI-SBR is designed to approximately learn the Nash equilibrium of an auxiliary matrix game induced by the value functions for the two players, which we denote by  $v_{t}^{1}$  and  $v_{t}^{2}$ , where  $t$  is the iteration index of the outer loop. Importantly, the value functions  $v_{t}^{1}$  and  $v_{t}^{2}$  are maintained individually by players 1 and 2, and therefore do not necessarily satisfy  $v_{t}^{1} + v_{t}^{2} = 0$  due to independent learning. As a result, the auxiliary matrix game from the inner loop does not necessarily
admit a zero-sum structure during learning. The error induced from such a non-zero-sum structure appears in existing work [15, 43], and was handled by designing a novel truncated Lyapunov function. However, the truncated Lyapunov function was sufficient to establish the asymptotic convergence, but did not provide the explicit rate at which the induced error goes to zero. To enable finite-sample analysis, we introduce  $\| v_t^1 + v_t^2 \|_{\infty}$  as a Lyapunov function in our coupled Lyapunov framework, which is customized to capture the behavior of the induced error from the non-zero-sum structure of the inner-loop auxiliary matrix game.
Coupled Lyapunov Drift Inequalities. In the existing literature of stochastic iterative algorithms [44, 24, 45, 46], when using a Lyapunov approach for finite-sample analysis, once the Lyapunov drift inequality is established, the finite-sample bound follows straightforwardly by repeatedly invoking the result. However, since our learning dynamics (for stochastic games) maintain multiple sets of stochastic iterates and update them in a coupled manner, the Lyapunov drift inequalities we establish are also highly coupled. Decoupling these Lyapunov inequalities is a major challenge. To overcome it, we develop a systematic approach for decoupling, which crucially relies on a bootstrapping argument. Specifically, we first use the Lyapunov inequalities in a direct way to establish a crude bound for each Lyapunov function. Then, we substitute the crude bounds back into the Lyapunov drift inequalities to obtain tighter bounds. See Appendix B.4 for a more detailed illustration of our decoupling procedure.
1.3 Related Work
Due to space constraints, here we only discuss related work in independent learning in matrix games and stochastic games, and single-agent RL. See Appendix A for a more detailed literature review.
Independent Learning in Matrix Games. Independent learning has been well-studied in the literature on learning in matrix games. Fictitious play (FP) [47] may be viewed as the earliest one of this kind, and its convergence analysis for the zero-sum setting is provided in [48]. Smoothed versions of FP have been developed [49, 50] to make the learning dynamics consistent [51, 52]. It was shown that the behavior of smoothed FP is captured by an ODE known as the smoothed best-response dynamics, which were also studied extensively in the literature [53]. Note that the Lyapunov function used to study the smoothed best-response dynamics is the regularized Nash gap [54, 53, 29], a variant of which is also used in our Lyapunov framework. To make the learning dynamics payoff-based, [28] developed a two time-scale variant of the smoothed best-response dynamics and established the asymptotic convergence. Moreover, no-regret learning algorithms, extensively studied in online learning, can also be used as independent learning dynamics for matrix games [55]. It is known that they are both convergent and rational by the definition in [27], and are usually implemented in a symmetric way. See [55] for a detailed introduction to no-regret learning in games.
Independent Learning in Stochastic Games. For stochastic games, independent and symmetric policy gradient methods have been developed in recent years, mostly for the case of Markov potential games [18, 19, 56]. The zero-sum case is more challenging since there is no single Lyapunov function to capture the learning dynamics (which is why we need to develop a coupled Lyapunov approach with multiple Lyapunov functions), such as the potential function in Markov potential games. For non-potential game settings, symmetric variants of policy gradient methods have been proposed, but have only been studied under the full-information setting without finite-sample guarantees [31, 32, 57, 33-35], with the exception of [58, 59]. However, the learning algorithm in [58] requires some coordination between the players when sampling, and is thus not completely independent; that in [59] is extragradient-based and needs some stage-based sampling process that also requires coordination across players. Best-response-type independent learning for stochastic games has been studied recently in [39, 15, 43, 60, 40, 61, 62], with [15, 43, 40, 61] tackling the zero-sum setting. However, only asymptotic convergence was established in these works, which motivated this work.
Finite-Sample Analysis in Single-Agent RL. The most related works (in single-agent RL) to our paper are those that perform finite-sample analysis for RL in infinite-horizon discounted Markov decision processes following a single trajectory of Markovian samples. See [63, 23, 41, 24-26, 64-67, 10] and the references therein. Among these works, [23, 24] established finite-sample bounds for temporal difference (TD)-learning (with linear function approximation), and [25, 64, 65] established finite-sample bounds for  $Q$ -learning. In both cases, the behavior policy for sampling is a stationary policy. For non-stationary behavior policies as we consider, [41] established finite-sample bounds for SARSA, an on-policy variant of  $Q$ -learning, and [10] provided finite-sample bounds for off-policy actor-critic, which is established based on a general framework of contractive stochastic approximation with time-inhomogeneous Markovian noise.
2 Zero-Sum Matrix Games
We begin by considering zero-sum matrix games. This section introduces algorithmic and technical ideas that are important for the setting of stochastic games. For  $i \in \{1,2\}$ , let  $\mathcal{A}^i$  be the finite action space of player  $i$ , and let  $R_i \in \mathbb{R}^{|\mathcal{A}^i| \times |\mathcal{A}^{-i}|}$  (where  $-i$  denotes the index of player  $i$ 's opponent) be the payoff matrix of player  $i$ . Note that in a zero-sum game we have  $R_1 + R_2^\top = 0$ . Since there are finitely many actions for each player, we assume without loss of generality that  $\max_{a^1, a^2} |R_1(a^1, a^2)| \leq 1$ . Furthermore, we denote  $A_{\mathrm{max}} = \max(|\mathcal{A}^1|, |\mathcal{A}^2|)$ .
The decision variables here are the policies  $\pi^i\in \Delta (\mathcal{A}^i)$ ,  $i\in \{1,2\}$ , where  $\Delta (\mathcal{A}^i)$  denotes the probability simplex supported on  $\mathcal{A}^i$ . Given a joint policy  $(\pi^1,\pi^2)$ , the expected reward received by player  $i$  is  $\mathbb{E}_{A^i\sim \pi^i,A^{-i}\sim \pi^{-i}}[R_i(A^i,A^{-i})] = (\pi^i)^\top R_i\pi^{-i}$ , where  $i\in \{1,2\}$ . Both players aim to maximize their rewards against their opponents. Unlike in the single-agent setting, since the performance of player  $i$ 's policy depends on its opponent  $-i$ 's policy, there is, in general, no universal optimal policy. Instead, we use the Nash gap and the regularized Nash gap as measurements of the performance of the learning dynamics, as formally defined below.
Definition 2.1 (Nash Gap in Matrix Games). Given a joint policy  $\pi = (\pi^1, \pi^2)$ , the Nash gap  $\mathrm{NG}(\pi^1, \pi^2)$  is defined as  $\mathrm{NG}(\pi^1, \pi^2) = \sum_{i=1,2} \max_{\hat{\pi}^i \in \Delta(\mathcal{A}^i)} (\hat{\pi}^i - \pi^i)^\top R_i \pi^{-i}$ .
Note that  $\mathrm{NG}(\pi^1, \pi^2) = 0$  if and only if  $(\pi^1, \pi^2)$  is in a Nash equilibrium of the matrix game (which may not be unique), in which no player has the incentive to change its policy.
Definition 2.2 (Regularized Nash Gap in Matrix Games). Given a joint policy  $\pi = (\pi^1, \pi^2)$  and a constant  $\tau > 0$ , the entropy-regularized Nash gap  $\mathrm{NG}_{\tau}(\pi^1, \pi^2)$  is defined as  $\mathrm{NG}_{\tau}(\pi^1, \pi^2) = \sum_{i=1,2} \left\{ \max_{\hat{\pi}^i \in \Delta(\mathcal{A}^i)} (\hat{\pi}^i - \pi^i)^{\top} R_i \pi^{-i} + \tau \nu(\hat{\pi}^i) - \tau \nu(\pi^i) \right\}$ , where  $\nu(\cdot)$  is the entropy function defined as  $\nu(\pi^i) = -\sum_{a^i \in \mathcal{A}^i} \pi^i(a^i) \log(\pi^i(a^i))$  for  $i \in \{1,2\}$ .
A joint policy  $(\pi^1, \pi^2)$  satisfying  $\mathrm{NG}_{\tau}(\pi^1, \pi^2) = 0$  is called the Nash distribution [29] or the quantal response equilibrium [30], which, unlike Nash equilibria, is unique in zero-sum matrix games. As  $\tau$  approaches 0, the corresponding Nash distribution approximates a Nash equilibrium [68].
2.1 The Learning Dynamics in Zero-Sum Matrix Games
We start by presenting in Algorithm 1 (from the perspective of player  $i$ , where  $i \in \{1,2\}$ ) the independent learning dynamics for zero-sum matrix games, which was first proposed in [28]. Given  $\tau > 0$ , we use  $\sigma_{\tau} : \mathbb{R}^{|\mathcal{A}^i|} \mapsto \mathbb{R}^{|\mathcal{A}^i|}$  for the softmax function with temperature  $\tau$ , that is,  $[\sigma_{\tau}(q^i)](a^i) = \exp(q^i(a^i) / \tau) / \sum_{\tilde{a}^i \in \mathcal{A}^i} \exp(q^i(\tilde{a}^i) / \tau)$  for all  $a^i \in \mathcal{A}^i$ ,  $q^i \in \mathbb{R}^{|\mathcal{A}^i|}$ , and  $i \in \{1,2\}$ .
Algorithm 1 Independent Learning Dynamics in Zero-Sum Matrix Games
1: Input: Integer  $K$ , initializations  $q_0^i = 0 \in \mathbb{R}^{|\mathcal{A}^i|}$  and  $\pi_0^i = \mathrm{Unif}(\mathcal{A}^i)$ .
2: for  $k = 0,1,\dots ,K - 1$  do
3:  $\pi_{k + 1}^{i} = \pi_{k}^{i} + \beta_{k}(\sigma_{\tau}(q_{k}^{i}) - \pi_{k}^{i})$
4: Play  $A_{k}^{i} \sim \pi_{k+1}^{i}(\cdot)$  (against  $A_{k}^{-i}$ ), and receive reward  $R_{i}(A_{k}^{i}, A_{k}^{-i})$ .
5:  $q_{k + 1}^{i}(a^{i}) = q_{k}^{i}(a^{i}) + \alpha_{k}\mathbb{1}_{\{a^{i} = A_{k}^{i}\}}\left(R_{i}(A_{k}^{i},A_{k}^{-i}) - q_{k}^{i}(A_{k}^{i})\right)$  for all  $a^i\in \mathcal{A}^i$
6: end for
To make this paper self-contained, we next provide a detailed interpretation of Algorithm 1, which also motivates our algorithm for stochastic games in Section 3. At a high level, Algorithm 1 can be viewed as a discrete and smoothed variant of the best-response dynamics, where each player constructs an approximation of the best response to its opponent's policy using the  $q$ -function. The update for the  $q$ -function is in the spirit of the TD-learning algorithm in RL [69].
The Policy Update. To understand the update equation for the policies (cf. Algorithm 1 Line 3), consider the discrete version of the smoothed best-response dynamics:
$$
\pi_ {k + 1} ^ {i} = \pi_ {k} ^ {i} + \beta_ {k} \left(\sigma_ {\tau} \left(R _ {i} \pi_ {k} ^ {- i}\right) - \pi_ {k} ^ {i}\right), \quad i \in \{1, 2 \}. \tag {1}
$$
In Eq. (1), each player updates its policy  $\pi_k^i$  incrementally towards the smoothed best response to its opponent's current policy. While the dynamics in Eq. (1) provably converge for zero-sum matrix
games, see e.g., [70], implementing it requires player  $i$  to compute  $\sigma_{\tau}(R_i\pi_k^{-i})$ . Note that  $\sigma_{\tau}(R_i\pi_k^{-i})$  involves the exact knowledge of the opponent's policy and the reward matrix, both of which cannot be accessed in payoff-based independent learning. This leads to the update equation for the  $q$ -functions, which estimate the quantity  $R_i\pi_k^{-i}$  that is needed for implementing Eq. (1).
The  $q$ -Function Update. Suppose for now that we are given a stationary joint policy  $\pi = (\pi^1, \pi^2)$ . Fix  $i \in \{1, 2\}$ , the problem of player  $i$  estimating  $R_i \pi^{-i}$  can be viewed as a policy evaluation problem, which is usually solved with TD-learning in RL [69]. Specifically, the two players repeatedly play the matrix game with the joint policy  $\pi = (\pi^1, \pi^2)$  and produce a sequence of joint actions  $\{(A_k^1, A_k^2)\}_{k \geq 0}$ . Then, player  $i$  forms an estimate of  $R_i \pi^{-i}$  through the following iterative algorithm:
$$
q _ {k + 1} ^ {i} \left(a ^ {i}\right) = q _ {k} ^ {i} \left(a ^ {i}\right) + \alpha_ {k} \mathbb {1} _ {\left\{a ^ {i} = A _ {k} ^ {i} \right\}} \left(R _ {i} \left(A _ {k} ^ {i}, A _ {k} ^ {- i}\right) - q _ {k} ^ {i} \left(A _ {k} ^ {i}\right)\right), \quad \forall a ^ {i} \in \mathcal {A} ^ {i}, \tag {2}
$$
with an arbitrary initialization  $q_0^i \in \mathbb{R}^{|\mathcal{A}^i|}$ , where  $\alpha_{k} > 0$  is the stepsize. To understand Eq. (2), suppose that  $q_{k}^{i}$  converges to some  $\bar{q}^i$ . Then Eq. (2) should be "stationary" at the limit point  $\bar{q}^i$  in the sense that  $\mathbb{E}_{A^i\sim \pi^i(\cdot),A^{-i}\sim \pi^{-i}(\cdot)}[\mathbb{1}_{\{a^i = A^i\}}(R_i(A^i,A^{-i}) - \bar{q}^i (A^i))] = 0$  for all  $a^i \in \mathcal{A}^i$ , which implies  $\bar{q}^i = R_i\pi^{-i}$ , as desired. Although Eq. (2) is motivated by the case when the joint policy  $(\pi^1,\pi^2)$  is stationary, the joint policy  $\pi_k = (\pi_k^1,\pi_k^2)$  from Eq. (1) is time-varying. A natural approach to address this issue is to make sure that the policies evolve at a slower time-scale compared to that of the  $q$ -functions, so that  $\pi_{k}$  is close to being stationary from the perspectives of  $q_{k}^{i}$ .
Remark. In [28], where Algorithm 1 was first proposed, the authors require  $\beta_{k} = o(\alpha_{k})$  to establish the asymptotic convergence, making Algorithm 1 a two time-scale algorithm. In this work, for finite-sample analysis and easier implementation, we update  $\pi_k^i$  and  $q_{k}^{i}$  on a single time scale with only a multiplicative constant difference in their stepsizes, i.e.,  $\beta_{k} = c_{\alpha ,\beta}\alpha_{k}$  for some  $c_{\alpha ,\beta}\in (0,1)$ .
2.2 Finite-Sample Analysis
In this section, we present the finite-sample analysis of Algorithm 1 for the convergence to the Nash distribution [29]. We consider using either constant step sizes, i.e.,  $\alpha_{k} \equiv \alpha$  and  $\beta_{k} \equiv \beta = c_{\alpha,\beta}\alpha$ , or diminishing step sizes with  $\mathcal{O}(1/k)$  decay rate, i.e.,  $\alpha_{k} = \alpha/(k+h)$  and  $\beta_{k} = \beta/(k+h) = c_{\alpha,\beta}\alpha/(k+h)$ . Let  $\ell_{\tau} = [(A_{\max}-1)\exp(2/\tau)+1]^{-1}$  and  $L_{\tau} = \tau/\ell_{\tau} + A_{\max}^{2}/\tau$ . The requirement for choosing the step sizes is stated in the following condition.
Condition 2.1. When using either constant or diminishing step sizes, we choose  $\tau \leq 1$ ,  $\alpha_0 < \frac{2}{\ell_\tau}$ ,  $\beta_0 < \min(2, \frac{\tau}{128A_{\max}^2})$ , and  $c_{\alpha,\beta} = \beta_k / \alpha_k \leq \min\left(\frac{\tau\ell_\tau^3}{32}, \frac{\ell_\tau\tau^3}{128A_{\max}^2}, \frac{2\sqrt{2}}{L_\tau^{1/2}}\right)$ .
We next state the finite-sample bounds of Algorithm 1. See Appendix C for the proof.
Theorem 2.1. Suppose that both players follow the learning dynamics presented in Algorithm 1, and the stepsizes  $\{\alpha_{k}\}$  and  $\{\beta_k\}$  are chosen such that Condition 2.1 is satisfied. Then we have the following results.
(1) When using constant step sizes, i.e.,  $\alpha_{k}\equiv \alpha$  and  $\beta_{k}\equiv \beta$ , we have
$$
\mathbb {E} \left[ N G _ {\tau} \left(\pi_ {K} ^ {1}, \pi_ {K} ^ {2}\right) \right] \leq B _ {i n} \left(1 - \frac {\beta}{4}\right) ^ {K} + 8 L _ {\tau} \beta + \frac {6 4 \alpha}{c _ {\alpha , \beta}},
$$
where  $B_{in} = 4 + 2\tau \log (A_{\mathrm{max}}) + 2A_{\mathrm{max}}$
(2) When using  $\alpha_{k} = \alpha /(k + h)$  and  $\beta_{k} = \beta /(k + h)$ , by choosing  $\beta > 4$ , we have
$$
\mathbb {E} \left[ N G _ {\tau} \left(\pi_ {K} ^ {1}, \pi_ {K} ^ {2}\right) \right] \leq B _ {i n} \left(\frac {h}{K + h}\right) ^ {\beta / 4} + \left(6 4 e L _ {\tau} \beta + \frac {5 1 2 e \alpha}{c _ {\alpha , \beta}}\right) \frac {1}{K + h}.
$$
The convergence bounds in Theorem 2.1 are qualitatively consistent with the existing results on the finite-sample analysis of general stochastic approximation algorithms [44, 46, 24, 65, 23, 42]. Specifically, when using constant step sizes, the bound consists of a geometrically decaying term (known as the optimization error) and two constant terms (known as the statistical error) that are proportional to the step sizes. When using diminishing step sizes with suitable hyperparameters, both the optimization error and the statistical error achieve an  $\mathcal{O}(1 / K)$  rate of convergence.