SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The goal is to compute a near-optimal policy from the given dataset  $D$ , via finding an  $f \in \mathcal{F}$  that approximates the optimal Q-function  $Q$ . The algorithm runs in episodes. For each episode  $k$ , it first generates an augmented dataset  $D_{k}$  using  $\mathrm{ASG}(D, m, \beta_{k})$ , where  $\beta_{k}$  is some distribution of the pseudo-stochastic states satisfies that, for each typical pseudo-stochastic state, it can be generated with at least probability  $\sigma_{1}$ . We assume that the pseudo-stochastic states will not transition to atypical states, such as extremely large values (possibly infinite) in the queuing example, under a reasonable policy. Details can be found in Assumption A.4 in the supplemental material. We remark that we can also use the same  $\beta$  for all the episodes, and in practice, we can simply adopt the estimation of the pseudo-stochastic states (i.e., with Gaussian distribution) as  $\beta_{k}$ . The algorithm then computes  $f_{k}$  as the minimizer that minimizes the squared loss regression objective over  $\mathcal{F}$ . The complete algorithm is given in Algorithm 2.
3.3 Guarantee and Analysis
The guarantee of Algorithm 2 is summarized below.
Theorem 1. We assume the data coverage and completeness assumptions (details in the section A in the supplemental material). Given a dataset  $D = \{s, x, a, r, s', x'\}$  with  $n$  samples, w.p. at least  $1 - \delta$ , the output policy of Algorithm 2 after  $k$  iterations,  $\pi_{f_k}$ , satisfies
$$
\begin{array}{l} v ^ {\pi_ {f _ {k}}} - v ^ {*} \leq \frac {2}{(1 - \gamma) ^ {2}} \left(\sqrt {\frac {(m + 1) C}{m \sigma_ {1}} \left(5 \left(\frac {1}{n} + \frac {1}{m}\right) V _ {\max } ^ {2} \log \left(\frac {n K | \mathcal {F} | ^ {2}}{\delta}\right) + \frac {3 \delta V _ {\max } ^ {2}}{n}\right)} \right. \\ \left. + \sqrt {c _ {0} \sigma_ {2}} V _ {\max } + \gamma^ {k} (1 - \gamma) V _ {\max }\right), \tag {7} \\ \end{array}
$$
where  $C$  is a constant related to the data coverage assumption A.3, and  $c_0, \sigma_1, \sigma_2$  are constants defined in Assumption A.4 to ensure a typical set of reasonable typical pseudo-stochastic states.
Algorithm 2: Batch FQI with ASG
1 Input: A dataset  $D = \{(s, x, a, r, s', x')\}$ , a function class  $\mathcal{F}$ , a positive integer  $m$ , distributions  $\{\beta_k\}_{k=1}^K$ ;
2 Initialize approximate  $Q$  -function: Choose  $f_{0}$  randomly from  $\mathcal{F}$
3 for episode  $k = 1,2,\dots ,K$  do
4 Generate augmented dataset  $\hat{D}_k = \mathbf{ASG}(D,m,\beta_k)$
5 Define loss function as:
$$
\mathcal {L} _ {\hat {D} _ {k}} (f; f _ {k - 1}) = \frac {1}{| \hat {D} _ {k} |} \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in \hat {D} _ {k}} \left(f (s, x, a) - r - \gamma V _ {f _ {k - 1}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2}
$$
$$
\begin{array}{l} f _ {k} = \arg \min  _ {f \in \mathcal {F}} \mathcal {L} _ {\hat {D} _ {k}} (f; f _ {k - 1}), V _ {f _ {k}} (s, x) = \min  _ {a \in \mathcal {A}} f _ {k} (s, x, a); \\ \pi_ {f _ {k}} (s, x) = \arg \min  _ {a \in \mathcal {A}} f _ {k} (s, x, a); \\ \end{array}
$$
7 Output:  $\pi_{f_K}$
Theorem 1 shows that ASG significantly improves the (real) sample complexity, i.e., the number of real data samples needed. When the tail of pseudo-stochastic states decays fast, e.g., an exponential tails link in a typical queueing network, we can choose a sampling distribution  $\beta$  that guarantees that  $\sigma_{2} = \tilde{\mathcal{O}}(1/n)$  and  $\sigma_{1} = \Omega(1/\log n)$ . Therefore, for sufficiently large  $k$  and sufficiently small  $\sigma_{2}$ , the first term is the dominating term. As  $m$  increases from 1 to  $n$ , the first term of the bound decreases from  $\tilde{\mathcal{O}}(1)$  to  $\tilde{\mathcal{O}}(\sqrt{1/n})$ . Remark that we cannot establish a convergence result without using ASG (when  $m = 0$ ) since using the dataset  $D$  alone doesn't have a sufficient data coverage guarantee, which is critical for batch RL. Due to the page limit, we only present the outline behind our analysis. The detailed proof is deferred to the supplemental material.
Proof Outline:
1. First we will show that using performance difference lemma (Kakade and Langford, 2002) it is sufficient by bounding  $\| f_k - Q^* \|_{2,\xi}$ , where  $\xi$  is some distribution.
2. Then, by considering how rare the distribution of the pseudo-stochastic states is under  $\xi$  and then classify the pseudo-stochastic states into typical and atypical sets based on a threshold  $\sigma_{2}$  on the distribution. Along with the augmented data generator under  $\beta_{k}(\cdot)$ . Then later we show that the term  $\| f_{k} - Q^{*}\|_{2,\xi}$  can be bounded by  $\mathcal{O}\left(\| f_k - \mathcal{T}f_{k - 1}\|_{2,\mu_{\beta_k}} + \gamma \| f_{k + 1} - Q^*\|_{2,\xi '} + \sqrt{\sigma_2V_{\max}}\right)$ , where  $\mu_{\beta_k}$  is the data distribution after augmenting virtual samples. The first two terms are related to the typical set given that a sufficient coverage of the data after data augmentation. The last term is easy to obtain by considering the atypical set.
3. Therefore, it is obvious that if we can have a bound on  $\| f_k - \mathcal{T}f_{k-1}\|_{2,\mu_{\beta_k}}$  which is independent of the episode  $k$ , we can prove the final result by expanding  $\| f_k - Q^*\|_{2,\xi}$  iteratively for  $k$  times. We finally show that the term  $\| f_k - \mathcal{T}f_{k-1}\|_{2,\mu_{\beta_k}}$  can indeed be bounded by using the FQI minimizer at each episode and a concentration bound on the offline dataset after augmenting virtual samples.
3.4 Extension to the case when  $|\mathcal{X}|$  is infinite
Theorem 1 assumes the number of the pseudo-stochastic states is finite. The result can be generalized to infinite pseudo-stochastic state space  $|\mathcal{X}|$  if we make the following additional assumption:
Assumption 1. For the typical set  $\mathcal{B}$  of pseudo-stochastic states (formally defined in Assumption A.4 in the supplemental material), for any  $s, a \in S \times \mathcal{A}, f \in \mathcal{F}$ , if  $x \in \mathcal{B}$ , then  $f(s, x, a) \leq V_{\max}$  otherwise if  $x \notin \mathcal{B}$ , we have  $|f(s, x, a) - Q^{*}(s, x, a)| \leq V_{\max}$ . Furthermore, for any given  $f \in \mathcal{F}, (s, x), x \in \mathcal{B}$ , we have  $|V_{f}(s', x') - V_{f}(s'', x'')| \leq V_{\max}$ , where  $x' = g(s, x, \pi_{f}, s'), x'' = g(s, x, \pi_{f}, s'')$ .
The first part in assumption 1 means that the function  $f \in \mathcal{F}$  is always bounded when  $x \in \mathcal{B}$ . Otherwise, the difference between  $f$  and the optimal  $Q$ -function  $Q^{*}$  is bounded, which indicates that although  $f$  could be extremely large or even unbounded for the pseudo-stochastic states, including the infinite case, in the atypical set, it is also true for the optimal  $Q$ -function  $Q^{*}$ . Therefore the difference between  $f$  and  $Q^{*}$  is assumed to be bounded, which is reasonable in practice. The second
part states that for any given  $x \in \mathcal{B}$ , the difference between the value functions of any possible next transitions  $(s', x'')$  and  $(s'', x'')$  is always bounded, which is also true in general especially in queuing literature since the changing of queue lengths between two timeslots is small.
Assumption 1 is easy to be satisfied in a typical queueing system (e.g. the wireless downlink network in Section 2.1). If the control/scheduling policy is a stabilizing policy, then the queues have exponential tails, i.e., the probability decays exponentially as queue length increases, and the policy can drive very large queues to bounded values exponentially fast. Therefore, if all functions in  $F$  are from stabilizing policies, Assumption 1 holds because any policy in  $F$  including the optimal policy can reduce the queue lengths exponentially fast (in expectation) and the difference in  $Q$ -functions therefore is bounded.
Under the additional Assumption 1, we show that the same order of the result is achievable. Details can be found in Section A.2 in the supplemental material.
To evaluate the performance of our approach, we compared our results with several baselines in extensive queuing environments. In our simulations, when augmenting virtual samples, we only require that the augmented samples are valid samples for which the action from the real sample remains to be feasible.
4.1 The Criss-Cross Network
The criss-cross network is shown in Fig. 2a, which consists of two servers and three classes of jobs. Each job class is queued in a separate buffer if not served immediately. Each server can only serve one job at a time. Server 1 can process jobs of both class 1 and class 3; server 2 can only process jobs of class 2. Class 3 jobs become class 2 jobs after being processed, and class 2 and class 3 jobs leave the system after being processed. The service time of a class  $i$  job is exponentially distributed with mean  $m_{i}$ . The service rate of a class  $i$  job is defined to be  $\mu_{i} := 1 / m_{i}$ , and jobs of class 1 and class 3 arrive to the system following the Poisson processes with rates  $\lambda_{1}$  and  $\lambda_{3}$ . To make sure the load can be supportable (i.e. queues can be stabilized), we assume  $\lambda_{1}m_{1} + \lambda_{3}m_{3} < 1$  and  $\lambda_{1}m_{2} < 1$ . In the simulation, we consider the imbalanced medium traffic regime, where  $\mu_{1} = \mu_{3} = 2$ ,  $\mu_{2} = 1.5$ ,  $\lambda_{1} = \lambda_{3} = 0.6$ .
All queues are first-in-first-out (FIFO). Let  $e_t(i) \in \{-1, 0, +1\}$  denote the state of class  $i$  jobs at time  $t$ , where  $-1$  means a departure,  $+1$  means a arrival and  $0$  denotes no changes for job  $i$ . Then  $e_t = (e_t(1), e_t(2), e_t(3))$  be the event happens at time  $t$  in the system, which is the stochastic state in this mixed system. Let  $q_t = (q_t(1), q_t(2), q_t(3))$  be the vector of queue lengths at time  $t$ , where  $q_t(i)$  is the number of class  $i$  jobs in the system. Obviously,  $q_t$  is the pseudo-stochastic state in the system, which can be derived from  $(q_{t-1}, e_{t-1})$  and action  $a_t$ . We combine ASG with  $Q$ -learning and compare it with several baselines: (i) the vanilla Q-learning, (ii) a proximal policy optimization (PPO) based algorithm proposed in Dai and Gluzman (2022) designed for queuing networks, (iii) a random policy, and (iv) a priority policy such that the server always serves class 1 jobs when its queue length is not empty, otherwise it serves class 3 jobs. Simulation results in Fig. 2d demonstrate that using ASG significantly improves the performance, and achieves the performance of the optimal policy, which was obtained by solving a large-scale MDP problem, after only 4 training episodes ( $4k$  training steps). All other policies are far away from the optimal policy, and barely learn anything, as we mentioned that the state space is quite large and is not even sufficiently explored after  $4k$  training steps. To further demonstrate the fundamental difference between ASG and Dyna-type algorithms, we also compared ASQ with the model-based approach: Q-learning with Dyna. We can observe that Dyna also fails to learn a good policy since, as we mentioned before, ASG is built for improving sample efficiency with augmented samples that represent true transitions for possible all all the pseudo-stochastic states, but dyna-type algorithms can only generate addition samples from an estimated model, which are limited by the samples that have been seen before.
4.2 Two-Phase Criss-Cross Network:
To further evaluate the power of ASG beyond the tabular setting, We combine ASG with Deep Q-network (Mnih et al., 2015b) named DQN-ASG and evaluate it on a more complicated variant of the
Criss-Cross network presented in Fig. 2b, where class 3 jobs have two phases. Class 3 jobs at phase 1 become phase 2 after being processed with probability  $1 - p$ , leave the system with probability  $p$ ; and class 3 jobs at phase 2 leave the system after completed processing. Results in Fig. 2e indicate that combining ASG with DQN also has a significant improvement on the performance, and our method only needs 100 episodes to achieve a near-optimal policy. We also remark here that the training time for DQN-ASG is three times faster than the baseline (4 hours vs. 12 hours). In the simulation, all the parameters are set to be the same as those in the previous case, and we use  $p = 0.8$ . Both the criss-cross and two-phase criss-cross networks are continuous-time Markov chains (CTMCs). We use the standard uniformization (Puterman, 2014; Serfozo, 1979) approach to simulate them using discrete-time Markov chains (DTMCs).
We remark here that the two-phase Criss-Cross network cannot be modeled as MDPs with exogenous inputs. Note that the phase a job is in is a stochastic state, which depends on the action (whether a job is selected for service). Since the scheduling action depends on the queue lengths, the evolution of stochastic states depends on the pseudo-stochastic states (the queue lengths). However, the exogenous input in an MDP with exogenous inputs has to be an observable random process independent of the action. Therefore, the current phase of a job of the phases Criss-Cross network is not an exogenous input and the system is not an MDP with exogenous inputs.
4.3 Wireless Networks
In addition to the criss-cross networks, we also evaluate our approach on the downlink network described in Section 2.1 with three mobiles. We let the arrivals are Poisson with rates  $\Lambda = \{2,4,3\}$ , and the channel rates  $\mathbf{O}$  to be  $\{12,12,12\}$ . We use the Max-Weight (Tassiulas and Ephremides, 1992) algorithm, a throughput-optimal and heavy-traffic delay optimal algorithm, as the baseline. As shown in Fig. 2f, the learned policy outperforms Max-Weight. To the best of our knowledge, no efficient RL algorithms exist that can outperform Max-Weight.
(a) Criss-Cross network
(d) Testing reward
Figure 2: Performance on queuing systems
(b) Phases Criss-Cross network
(e) Testing reward
(c) Wireless network
(f) Testing reward
4.4 Additional Simulations on the criss-cross network
To further verify the performance of our algorithm on more scenarios, we first consider a more general criss-cross network where the size of both class 1 and class 3 jobs can vary over time. Then all class 1 and class 3 jobs need multiple timeslots to be processed instead of 1. In particular, when a new job of either class 1 or 2 joins the system, the job size of such job is chosen uniformly. The environment in Fig. 2b can be seen as a special case where class 1, and 3 have a fixed job size 1 and 2, respectively. The performances of different cases are presented in Fig. 3. Our approach still obtains the best result.
The details of the parameters can be found in section B.2. We do not include the optimal solution since, for the general case, the state space is quite large, and using value iteration to obtain the optimal solution is very time-consuming. The priority policy already provides a reasonable good baseline.
(c)
Figure 3: Performance on Criss-Cross network with general job size
4.5 Additional Simulations on the Wireless Network
We also evaluate the performance of our algorithm on the wireless network (Fig. 1) under different sets of arrival rates. The results of average queue length compared with those of Max-Weight are summarized in Table 2. We can observe that ASG performs better than Max-Weight in all the cases. Simulation results on a more complicated two-phase wireless network can be found in the supplemental material (Section B.3).
4.6 Combining ASG with Policy Gradient-type Algorithm
We also investigate the possibilities of using ASG in policy gradient-type algorithms (i.e., TD3 (Fu et al., 2018), SAC (Haarnoja et al., 2018)) and compare our approach with one of the state-of-art algorithms MBPO (Janner et al., 2019). Our approach still achieves the best performance. Due to page limit, the detailed simulation and algorithm are deferred to Section B in the supplemental material.
Arrival Rates	Service Rates	Max-Weight	DQN-ASG
{2,3,4}	{12,12,12}	10.979	10.403
{1,7,2}	{12,12,12}	13.811	13.273
{2,2,6}	{12,12,12}	13.774	13.137
{3,1,5}	{12,12,12}	10.895	10.091
Table 2: Best testing average queue length after training for  ${120k}$  steps
In this work, we considered RL problems for mixed systems which have two types of states: stochastic states and pseudo-stochastic states. We proposed a sample efficient RL approach that accelerates learning by augmenting virtual data samples. We theoretically quantified the (real) sample complexity gain by reducing the optimality gap from  $\tilde{\mathcal{O}}(1)$  to  $\tilde{\mathcal{O}}(1/\sqrt{n})$ . Experimental studies further confirmed the effectiveness of our approach on several queuing systems. The possible limitation of using ASG is that it introduces more computation during each learning iteration when generating virtual samples, though it is not obvious in our experiments.
Agarwal, A., Kakade, S., Krishnamurthy, A., and Sun, W. (2020). Flambe: Structural complexity and representation learning of low rank mdps. In Advances Neural Information Processing Systems (NeurIPS), page 20095-20107. Curran Associates Inc.
Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In Int. Conf. Machine Learning (ICML), pages 1042-1051. PMLR.
In this section, we present the proof of Theorem 1. We first introduce and recall some necessary notations and assumptions. Then, we present some auxiliary lemmas and their proofs. Finally, we combine the lemmas to prove the main result.
Notations: Define  $P(\nu)$  as distribution over states such that  $(s', x') \sim P(\nu) \Leftrightarrow (s, x, a) \sim \nu, s' \sim P(s'|s, a), x' = g(s, a, s')$ . In other words, it is the distribution of the next state if the state action pair follows  $\nu$ . For  $f: \mathcal{S} \times \mathcal{X} \times \mathcal{A} \to \mathbb{R}, \nu \in \Delta(\mathcal{S} \times \mathcal{X} \times \mathcal{A})$ , where  $\Delta(\cdot)$  is the probability simplex, and  $p > 1$ , define  $\|f\|_{p,\nu} = (\mathbb{E}_{(s,x,a) \sim \nu}[|f(s,x,a)|^p])^{1/p}$ . Define  $\pi_{f,f'}(s,x) := \arg \min_{a \in \mathcal{A}} \min \{f(s,x,a), f'(s,x,a)\}$ .
Recall that in Alg. 2, at iteration  $k$
$$
\mathcal {L} _ {\hat {D} _ {k}} (f; f _ {k - 1}) = \frac {1}{| \hat {D} _ {k} |} \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in \hat {D} _ {k}} \left(f (s, x, a) - r - \gamma V _ {f _ {k - 1}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2},
$$
$$
f _ {k} := \arg \min  _ {f \in \mathcal {F}} \mathcal {L} _ {\hat {D} _ {k}} (f; f _ {k - 1}) \quad \text {a n d} \quad V _ {f _ {k}} (s, x) := \min  _ {a \in \mathcal {A}} f _ {k} (s, x, a).
$$
Assumption A.1. (Realizability) For the optimal policy,  $Q^{*}\in \mathcal{F}$
Assumption A.2. (Completeness) For the policy  $\pi$  to be evaluated,  $\forall f\in \mathcal{F},\mathcal{T}f\in \mathcal{F}$ , where  $\mathcal{T}:\mathbb{R}^{S\times \mathcal{X}\times \mathcal{A}}\to \mathbb{R}^{S\times \mathcal{X}\times \mathcal{A}}$  is the Bellman update operator;  $\forall f$  :
$$
(\mathcal {T} f) (s, x, a) := R (s, x, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ V _ {f} (s ^ {\prime}, x ^ {\prime} = g (s, x, a, s ^ {\prime})) ].
$$
We say a distribution  $\nu \in \Delta (\mathcal{S}\times \mathcal{A})$  is admissible in MDP  $(\mathcal{S},\mathcal{X},\mathcal{A},P,R,\gamma)$ , if there exist  $h\geq 0$  and a policy  $\pi$  such that  $\nu (s,a) = \sum_{x\in \mathcal{X}}\operatorname *{Pr}(s_h = s,x_h = x,a_h = a|s_0,x_0,\pi)$ . The following assumption is imposed to limit the distribution shift. Note that "admissible" is defined on the stochastic state and action. Later, we will also abuse the notation and call  $\nu \in \Delta (\mathcal{S}\times \mathcal{X}\times \mathcal{A})$  is admissible if  $\nu (s,a) = \sum_{x}\nu (s,x,a)$  is admissible.
Assumption A.3. For a data distribution  $\mu$ , we assume that there exists  $C < \infty$  such that for any admissible  $v$  and any  $(s, a) \in S \times \mathcal{A}$ ,
$$
\frac {\nu (s , a)}{\mu (s , a)} \leq C,
$$
where  $\nu (s,a) = \sum_{x\in \mathcal{X}}\nu (s,x,a)$  and  $\mu (s,a) = \sum_{x\in \mathcal{X}}\mu (s,x,a)$ .
Assumptions A.2-A.3 are standard assumptions in batch reinforcement learning (Chen and Jiang, 2019). However, in assumption A.3, we only require the data coverage of stochastic states which is the fundamental difference.
Assumption A.4. We assume that there exists a set  $\mathcal{B}$  of typical pseudo-stochastic states such that the distributions  $\beta_{k}(x), k = 1, \ldots, K$  used for augmenting virtual samples satisfy  $\beta_{k}(x) \geq \sigma_{1}, \forall x \in \mathcal{B}$ . We also assume that the marginal distribution over using a reasonable policy  $\pi$ , that is,  $d_{\eta_0}^{\pi}(s, x) := (1 - \gamma)\sum_{t=0}^{\infty}\gamma^{t-1}\operatorname*{Pr}(s_t = s, x_t = x|(s_0, x_0) \sim \eta_0, \pi)$  satisfies  $\sum_{s \in \mathcal{S}}\sum_{x \in \mathcal{X}, x \notin \mathcal{B}}d_{\eta_0}^{\pi}(s, x) \leq \sigma_2$ , where  $\eta_0$  is the initial distribution. Furthermore, if we have for a distribution  $\eta$  of the states satisfying  $\sum_{s \in \mathcal{S}}\sum_{x \in \mathcal{X}, x \notin \mathcal{B}}\eta(s, x) \leq \sigma_2$ , then under any reasonable policy  $\pi$ , the marginal distribution  $\eta_h^\pi(s, x) := \operatorname*{Pr}(s_h = s, x_h = x|(s_0, x_0) \sim \eta, \pi)$  satisfies  $\sum_{s \in \mathcal{S}}\sum_{x \in \mathcal{X}, x \notin \mathcal{B}}\eta_h^\pi(s, x) \leq c_0\sigma_2, \forall h > 0$ , for some constant  $c_0 \geq 1$ . In particular, all the policies at each iteration, the optimal policy and their joint policies are assumed to be reasonable policies.
We remark that in a queuing network, under any stable policy, the queue distribution has an exponential tail; in other words, large queue lengths occur with a small probability. In such a case, we can use a uniform distribution for pseudo-stochastic states in set  $\mathcal{B}$  to guarantee that  $\sigma_{1} = \Theta \left(\frac{1}{\log(1 / \sigma_{2})}\right)$ . Therefore, if we choose  $\sigma_{2} = \frac{1}{n}$ , then  $\sigma_{1} = \frac{1}{\log n}$ .
Auxiliary Lemmas
In the following lemma, we will show that when all admissible distributions are not far away from the data distribution  $\mu$  over stochastic state  $S$  and action  $\mathcal{A}$ , we can have a good coverage of  $S \times \mathcal{X} \times \mathcal{A}$  by generating virtual samples.
For a given dataset  $D$  of size  $|D| = n$  and data distribution  $\mu$ , let  $\bar{\mu}_{\beta}$  denote the expected distribution of the state action pair  $(s, x, a)$  in the combined dataset after using Algorithm 2 with a virtual sample distribution  $\beta(x)$ .
Lemma A.1. Given a virtual sample generating distribution  $\beta(x)$  of the pseudo-stochastic state, if  $\beta(x) \geq \sigma_1, \forall x \in \mathcal{B}$ . Then given any admissible distribution  $\nu$ , then under Assumptions A.3, we have for any  $(s,x,a) \in S \times \mathcal{X} \times \mathcal{A}, x \in \mathcal{B}$ ,
$$
\frac {\nu (s , x , a)}{\bar {\mu} _ {\beta} (s , x , a)} \leq \frac {(m + 1) C}{m \sigma_ {1}},
$$
$$
\bar {\mu} _ {\beta} (s, x, a) = \mu (s, x, a) \frac {n}{n m + n} + \sum_ {\hat {x} \in \mathcal {X}} \mu (s, \hat {x}, a) \left(\beta (x) \frac {n m}{n m + n}\right) \tag {8}
$$
Proof. Given the data distribution  $\mu$ , we know that the real samples are drawn according to  $\mu(s, x, a)$ . Then