SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[58] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. *Dueling network architectures for deep reinforcement learning*. In *International Conference on Machine Learning*, pages 1995–2003. PMLR, 2016.
[59] C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, UK, 1989.
[60] Richard R Weber and Gideon Weiss. On an index policy for restless bandits. Journal of Applied Probability, 27(3):637-648, 1990.
[61] Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of Applied Probability, 25(A):287-298, 1988.
[62] Guojun Xiong, Shufan Wang, and Jian Li. Learning infinite-horizon average-reward restless multi-action bandits via index awareness. Advances in Neural Information Processing Systems, 35:17911-17925, 2022.
[63] Zhe Yu, Yunjian Xu, and Lang Tong. Deadline scheduling as restless bandits. IEEE Transactions on Automatic Control, 63(8):2343-2358, 2018.
[64] Xiangyu Zhang and Peter I Frazier. Near-optimality for infinite-horizon restless bandits with many arms. arXiv preprint arXiv:2203.15853, 2022.
[65] Jiahong Zhou, Shunhui Mao, Guoliang Yang, Bo Tang, Qianlong Xie, Lebin Lin, Xingxing Wang, and Dong Wang. RL-MPCA: A reinforcement learning based multi-phase computation allocation approach for recommender systems. In Proceedings of the ACM Web Conference 2023, pages 3214-3224, 2023.
Appendix to Weakly Coupled Deep Q-Networks
Proof. We prove part the first part of the proposition (weak duality) by induction. First, define
$$
Q _ {0} ^ {*} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) \quad \text {a n d} \quad Q _ {0} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \lambda^ {\intercal} \left[ \boldsymbol {b} (w) - \sum_ {i = 1} ^ {N} \boldsymbol {d} (s _ {i}, a _ {i}) \right],
$$
and suppose we run value iteration for both systems:
$$
Q _ {t + 1} ^ {*} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A} (\boldsymbol {s} ^ {\prime})} Q _ {t} ^ {*} (\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}) \right],
$$
$$
Q _ {t + 1} ^ {\lambda} (\pmb {s}, \pmb {a}) = r (\pmb {s}, \pmb {a}) + \lambda^ {\intercal} \left[ \pmb {b} (w) - \sum_ {i = 1} ^ {N} \pmb {d} (s _ {i}, a _ {i}) \right] + \gamma \mathbb {E} \left[ \max _ {\pmb {a} ^ {\prime} \in \mathcal {A}} Q _ {t} ^ {\lambda} (\pmb {s} ^ {\prime}, \pmb {a} ^ {\prime}) \right].
$$
It is well-known that, by the value iteration algorithm's convergence,
$$
Q ^ {*} (\boldsymbol {s}, \boldsymbol {a}) = \lim  _ {t \to \infty} Q _ {t} ^ {*} (\boldsymbol {s}, \boldsymbol {a}) \quad \text {a n d} \quad Q ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = \lim  _ {t \to \infty} Q _ {t} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}).
$$
Consider a state  $s \in S$  and a feasible action  $\pmb{a} \in \mathcal{A}(s)$ . We have,
$$
Q _ {0} ^ {\lambda} (\pmb {s}, \pmb {a}) = r (\pmb {s}, \pmb {a}) + \lambda^ {\intercal} \left[ \pmb {b} (w) - \sum_ {i = 1} ^ {N} \pmb {d} (s _ {i}, a _ {i}) \right] \geq r (\pmb {s}, \pmb {a}) = Q _ {0} ^ {*} (\pmb {s}, \pmb {a}).
$$
Suppose  $Q_{t}^{\lambda}(s, \boldsymbol{a}) \geq Q_{t}^{*}(s, \boldsymbol{a})$  holds for all  $s \in S$  and  $\boldsymbol{a} \in \mathcal{A}(s)$  for some  $t > 0$  (induction hypothesis). Then,
$$
\begin{array}{l} Q _ {t + 1} ^ {\lambda} (\pmb {s}, \pmb {a}) = r (\pmb {s}, \pmb {a}) + \lambda^ {\intercal} \left[ \pmb {b} (w) - \sum_ {i = 1} ^ {N} \pmb {d} (s _ {i}, a _ {i}) \right] + \gamma \mathbb {E} \bigl [ \max _ {\pmb {a} ^ {\prime} \in \mathcal {A}} Q _ {t} ^ {\lambda} (\pmb {s} ^ {\prime}, \pmb {a} ^ {\prime}) \bigr ] \\ \geq r (\boldsymbol {s}, \boldsymbol {a}) + \lambda^ {\intercal} \left[ \boldsymbol {b} (w) - \sum_ {i = 1} ^ {N} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) \right] + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A} \left(\boldsymbol {s} ^ {\prime}\right)} Q _ {t} ^ {*} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) \right] \\ \geq r (\pmb {s}, \pmb {a}) + \gamma \mathbb {E} \big [ \max _ {\pmb {a} ^ {\prime} \in \mathcal {A} (\pmb {s} ^ {\prime})} Q _ {t} ^ {*} (\pmb {s} ^ {\prime}, \pmb {a} ^ {\prime}) \big ] = Q _ {t + 1} ^ {*} (\pmb {s}, \pmb {a}). \\ \end{array}
$$
Thus, it follows that  $Q^{\lambda}(\pmb {s},\pmb {a})\geq Q^{*}(\pmb {s},\pmb {a})$
For the proof of the second part of the proposition, define
$$
\boldsymbol {B} _ {0} (w) = \boldsymbol {b} (w) \quad \text {a n d} \quad \boldsymbol {B} _ {t + 1} (w) = \boldsymbol {b} (w) + \gamma   \mathbb {E} \big [ \boldsymbol {B} _ {t} (w ^ {\prime}) \big ].
$$
We use an induction proof. We have for all  $(s, a) \in S \times \mathcal{A}$ ,
$$
\begin{array}{l} Q _ {0} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \lambda^ {\intercal} \left[ \boldsymbol {b} (w) - \sum_ {i = 1} ^ {N} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) \right] \\ = \sum_ {i = 1} ^ {N} \left[ r _ {i} \left(s _ {i}, a _ {i}\right) - \lambda^ {\mathsf {T}} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) \right] + \lambda^ {\mathsf {T}} \boldsymbol {b} (w) = \sum_ {i = 1} ^ {N} Q _ {0, i} ^ {\lambda} \left(s _ {i}, a _ {i}\right) + \lambda^ {\mathsf {T}} \boldsymbol {B} _ {0} (w), \\ \end{array}
$$
where  $Q_{0,i}^{\lambda}(s_i,a_i) = r_i(s_i,a_i) - \lambda^{\intercal}\pmb {d}(s_i,a_i)$ . Similarly, for all  $(s,\pmb {a})\in S\times \mathcal{A}$ ,
$$
\begin{array}{l} Q _ {1} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \lambda^ {\intercal} \left[ \boldsymbol {b} (w) - \sum_ {i = 1} ^ {N} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) \right] + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A}} Q _ {0} ^ {\lambda} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) \right] \\ = \sum_ {i = 1} ^ {N} \Big [ r _ {i} (s _ {i}, a _ {i}) - \lambda^ {\mathsf {T}} \pmb {d} (s _ {i}, a _ {i}) \Big ] + \lambda^ {\mathsf {T}} \pmb {b} (w) + \gamma \mathbb {E} \left[ \max _ {\pmb {a} ^ {\prime} \in \mathcal {A}} \left\{\sum_ {i = 1} ^ {N} Q _ {0, i} ^ {\lambda} (s _ {i} ^ {\prime}, a _ {i} ^ {\prime}) + \lambda^ {\mathsf {T}} \pmb {B} _ {0} (w ^ {\prime}) \right\} \right] \\ = \sum_ {i = 1} ^ {N} \left[ r _ {i} \left(s _ {i}, a _ {i}\right) - \lambda^ {\mathsf {T}} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) + \gamma \mathbb {E} \left[ \max  _ {a _ {i} ^ {\prime} \in \mathcal {A} _ {i}} Q _ {0, i} ^ {\lambda} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right) \right] \right] + \lambda^ {\mathsf {T}} \left(\boldsymbol {b} (w) + \gamma \mathbb {E} \left[ \boldsymbol {B} _ {0} \left(w ^ {\prime}\right) \right]\right) \\ = \sum_ {i = 1} ^ {N} Q _ {1, i} ^ {\lambda} \left(s _ {i}, a _ {i}\right) + \lambda^ {\intercal} \boldsymbol {B} _ {1} (w). \\ \end{array}
$$
Continuing in this manner, we arrive at  $Q_{t}^{\lambda}(\boldsymbol{s},\boldsymbol{a}) = \sum_{i=1}^{N} Q_{t,i}^{\lambda}(s_{i},a_{i}) + \lambda^{\mathsf{T}}\boldsymbol{B}_{t}(w)$ . Finally, we have
$$
\begin{array}{l} Q ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = \lim  _ {t \rightarrow \infty} Q _ {t} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) \\ = \lim  _ {t \rightarrow \infty} \sum_ {i = 1} ^ {N} Q _ {t, i} ^ {\lambda} (s _ {i}, a _ {i}) + \lambda^ {\intercal} \boldsymbol {B} _ {t} (w) = \sum_ {i = 1} ^ {N} Q _ {i} ^ {\lambda} (s _ {i}, a _ {i}) + \lambda^ {\intercal} \boldsymbol {B} (w), \\ \end{array}
$$
which follows by the convergence of value iteration.
Proof. First, we define the Bellman operator  $H$ :
$$
\left. \left(H Q ^ {\prime}\right) (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime}} Q ^ {\prime} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) \right], \right.
$$
which is known to be a  $\gamma$ -contraction mapping. Next we define the random noise term
$$
\xi_ {n} (\boldsymbol {s}, \boldsymbol {a}) = \gamma \max  _ {\boldsymbol {a} ^ {\prime}} Q _ {n} ^ {\prime} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) - \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime}} Q _ {n} ^ {\prime} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) \right]. \tag {19}
$$
Analogously, for subproblem  $i \in \{1, \dots, N\}$ , define the subproblem Bellman operator
$$
\left(H _ {i} Q _ {i} ^ {\lambda}\right) \left(s _ {i}, a _ {i}\right) = r _ {i} \left(s _ {i}, a _ {i}\right) - \lambda^ {\intercal} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) + \gamma \mathbb {E} \left[ \max  _ {a _ {i} ^ {\prime}} Q _ {i} ^ {\lambda} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right) \right],
$$
and random noise term
$$
\xi_ {i, n} \left(s _ {i}, a _ {i}\right) = \gamma \max  _ {a _ {i} ^ {\prime}} Q _ {i, n} ^ {\prime} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right) - \gamma \mathbb {E} \left[ \max  _ {a _ {i} ^ {\prime}} Q _ {i, n} ^ {\prime} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right) \right]. \tag {20}
$$
The update rules of WCQL can then be written as
$$
Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) = (1 - \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a})) Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \left[ \left(H Q _ {n} ^ {\prime}\right) (\boldsymbol {s}, \boldsymbol {a}) + \xi_ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) \right],
$$
$$
Q _ {i, n + 1} ^ {\lambda} \left(s _ {i}, a _ {i}\right) = Q _ {i, n} ^ {\lambda} \left(s _ {i}, a _ {i}\right) + \beta_ {n} \left(s _ {i}, a _ {i}\right) \left[ \left(H _ {i} Q _ {i, n} ^ {\prime}\right) \left(s _ {i}, a _ {i}\right) + \xi_ {i, n + 1} \left(s _ {i}, a _ {i}\right) \right], \tag {21}
$$
$$
Q _ {n + 1} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}) = \min  _ {\lambda \in \Lambda} \lambda^ {\intercal} \boldsymbol {B} _ {n} (w) + \sum_ {i = 1} ^ {N} Q _ {i, n} ^ {\lambda} \left(s _ {i}, a _ {i}\right),
$$
$$
Q _ {n + 1} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) = \min  \left(Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}), Q _ {n + 1} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a})\right). \tag {22}
$$
Parts (i) and (ii). By the iteration described in (21), we know that for a fixed  $\lambda$ , we are running Q-learning on an auxiliary MDP with Bellman operator  $H_{i}$ , which encodes a reward  $r_i(s_i,a_i) - \lambda^{\mathsf{T}}d(s_i,a_i)$  and the transition dynamics for subproblem  $i$ . By the standard result for asymptotic convergence of Q-learning [6], we have
$$
\lim  _ {n \rightarrow \infty} Q _ {i, n} ^ {\lambda} \left(s _ {i}, a _ {i}\right) = Q _ {i} ^ {\lambda} \left(s _ {i}, a _ {i}\right). \tag {23}
$$
We now prove the result in (ii):  $\lim_{n\to \infty}Q_n^\lambda (\pmb {s},\pmb {a})\geq Q^* (\pmb {s},\pmb {a})$  . Recall that
$$
Q _ {n} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = \lambda^ {\intercal} \boldsymbol {B} _ {n} (w) + \sum_ {i = 1} ^ {N} Q _ {i, n} ^ {\lambda} \left(s _ {i}, a _ {i}\right).
$$
By standard stochastic approximation theory,  $\lim_{n\to \infty}B_n(w) = B(w)$  for all  $w$  [38]. Combining this with (23), we have  $\lim_{n\to \infty}Q_n^\lambda (\pmb {s},\pmb {a}) = Q^\lambda (\pmb {s},\pmb {a})$  for all  $(s,a)$ , and to conclude that this limit is an upper bound on  $Q^{*}(s,a)$ , we apply Proposition 1.
Part (iii). Assume without loss of generality that  $Q^{*}(s, \pmb{a}) = 0$  for all state-action pairs  $(\pmb{s}, \pmb{a})$ . This can be established by shifting the origin of the coordinate system. We also assume that  $\alpha_{n}(\pmb{s}, \pmb{a}) \leq 1$  for all  $(\pmb{s}, \pmb{a})$  and  $n$ . We proceed via induction. Note that the iterates  $Q_{n}'(\pmb{s}, \pmb{a})$  are bounded in the sense that there exists a constant  $D_{0} = R_{\max} / (1 - \gamma)$ ,  $R_{\max} = \max_{(\pmb{s}, \pmb{a})} |r(\pmb{s}, \pmb{a})|$ , such that  $|Q_{n}'(\pmb{s}, \pmb{a})| \leq D_{0}$  for all  $(\pmb{s}, \pmb{a})$  and iterations  $n$  [17]. Define the sequence  $D_{k+1} = (\gamma + \epsilon)D_{k}$  such that  $\gamma + \epsilon < 1$  and  $\epsilon > 0$ . Clearly,  $D_{k} \to 0$ . Suppose that there exists a random variable  $n_{k}$  representing an iteration threshold such that for all  $(\pmb{s}, \pmb{a})$ ,
$$
- D _ {k} \leq Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) \leq \min  \left\{D _ {k}, Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}) \right\}, \quad \forall n \geq n _ {k}.
$$
We will show that there exists some iteration  $n_{k + 1}$  such that
$$
- D _ {k + 1} \leq Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) \leq \min  \left\{D _ {k + 1}, Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}) \right\} \quad \forall (s, a), n \geq n _ {k + 1},
$$
which implies that  $Q_{n}^{\prime}(s, \pmb{a})$  converges to  $Q^{*}(s, \pmb{a}) = 0$  for all  $(s, \pmb{a})$ .
By part (ii), we know that for all  $\eta > 0$ , with probability 1, there exists some finite iteration  $n_0$  such that for all  $n \geq n_0$ ,
$$
Q ^ {*} (\boldsymbol {s}, \boldsymbol {a}) - \eta \leq Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}). \tag {24}
$$
Now, we define an accumulated noise process started at  $n_k$  by  $W_{n_k,n_k}(s,\pmb{a}) = 0$ , and
$$
W _ {n + 1, n _ {k}} (\boldsymbol {s}, \boldsymbol {a}) = \left(1 - \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a})\right) W _ {n, n _ {k}} (\boldsymbol {s}, \boldsymbol {a}) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \xi_ {n + 1} (\boldsymbol {s}, \boldsymbol {a}), \quad \forall n \geq n _ {k}, \tag {25}
$$
where  $\xi_{n}(s,a)$  is as defined in (19). Let  $\mathcal{F}_n$  be the entire history of the algorithm up to the point where the step sizes at iteration  $n$  are selected. Using Corollary 4.1 in [6] which states that under Assumption 2 on the step size  $\alpha_{n}(\pmb {s},\pmb {a})$ , and if  $\mathbb{E}[\xi_n(\pmb {s},\pmb {a})|\mathcal{F}_n] = 0$  and  $\mathbb{E}[\xi_n^2 (\pmb {s},\pmb {a})|\mathcal{F}_n]\leq A_n$ , where the random variable  $A_{n}$  is bounded with probability 1, the sequence  $W_{n + 1,n_k}(\pmb {s},\pmb {a})$  defined in (25) converges to zero, with probability 1. From our definition of the stochastic approximation noise  $\xi_{n}(s,a)$  in (19), we have
$$
\mathbb {E} \left[ \xi_ {n} (\boldsymbol {s}, \boldsymbol {a}) \mid \mathcal {F} _ {n} \right] = 0 \quad \text {a n d} \quad \mathbb {E} \left[ \xi_ {n} ^ {2} (\boldsymbol {s}, \boldsymbol {a}) \mid \mathcal {F} _ {n} \right] \leq C \left(1 + \max  _ {\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}} Q _ {n} ^ {\prime 2} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right)\right),
$$
where  $C$  is a constant. Then, it follows that
$$
\lim  _ {n \to \infty} W _ {n, n _ {k}} (\boldsymbol {s}, \boldsymbol {a}) = 0 \quad \forall (\boldsymbol {s}, \boldsymbol {a}),   n _ {k}.
$$
We use the following lemma from [6] to bound the accumulated noise.
Lemma A.1 (Lemma 4.2 in [6]). For every  $\delta >0$ , with probability one, there exists some  $n^{\prime}$  such that  $|W_{n,n^{\prime}}(s,a)|\leq \delta$ , for all  $n\geq n'$ .
Now, by Lemma A.1, let  $n_{k'} \geq \max(n_k, n_0)$  such that, for all  $n \geq n_{k'}$  we have
$$
\left| W _ {n, n _ {k ^ {\prime}}} (\boldsymbol {s}, \boldsymbol {a}) \right| \leq \gamma \epsilon D _ {k} <   \gamma D _ {k}.
$$
Let  $\nu_{k}\geq n_{k^{\prime}}$  such that, for all  $n\geq \nu_k$  , by (24) we have
$$
\gamma \epsilon D _ {k} - \gamma D _ {k} \leq Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}).
$$
Define another sequence  $Y_{n}$  that starts at iteration  $\nu_{k}$ .
$$
Y _ {\nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}) = D _ {k} \quad \text {a n d} \quad Y _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) = \left(1 - \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a})\right) Y _ {n} (\boldsymbol {s}, \boldsymbol {a}) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \gamma D _ {k} \tag {26}
$$
Note that it is easy to show that the sequence  $Y_{n}(s, \boldsymbol{a})$  in (26) is decreasing, bounded below by  $\gamma D_k$ , and converges to  $\gamma D_k$  as  $n \to \infty$ . Now we state the following lemma.
Lemma A.2. For all state-action pairs  $(s, a)$  and iterations  $n \geq \nu_k$ , it holds that:
$$
- Y _ {n} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}) \leq Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) \leq \min  \left\{Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}), Y _ {n} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}) \right\}. \tag {27}
$$
Proof. We focus on the right hand side inequality, the left hand side can be proved similarly. For the base case  $n = \nu_k$ , the statement holds because  $Y_{\nu_k}(s, \boldsymbol{a}) = D_k$  and  $W_{\nu_k, \nu_k}(s, \boldsymbol{a}) = 0$ . We assume it is true for  $n$  and show that it continues to hold for  $n + 1$ :
$$
\begin{array}{l} Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) = (1 - \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a})) Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \left[ \left(H Q _ {n} ^ {\prime}\right) (\boldsymbol {s}, \boldsymbol {a}) + \xi_ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) \right] \\ \leq \left(1 - \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a})\right) \min  \left\{Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}), Y _ {n} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}) \right\} \\ + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \left(H Q _ {n} ^ {\prime}\right) (\boldsymbol {s}, \boldsymbol {a}) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \xi_ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) \\ \leq \left(1 - \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a})\right) \left(Y _ {n} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a})\right) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \gamma D _ {k} + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \xi_ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) \\ \leq Y _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n + 1, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}), \\ \end{array}
$$
where we used  $(HQ_n') \leq \gamma \| Q_n' \| \leq \gamma D_k$ . Now, we have
$$
\begin{array}{l} Q _ {n + 1} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) = \min  \left(Q _ {n + 1} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}), Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a})\right) \\ \leq \min  \left\{Q _ {n + 1} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}), Y _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n + 1, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}) \right\}. \\ \end{array}
$$
The inequality holds because
$$
Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) \leq Y _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) + W _ {n + 1, \nu_ {k}} (\boldsymbol {s}, \boldsymbol {a}),
$$
Since  $Y_{n}(\pmb {s},\pmb {a})\to \gamma D_{k}$  and  $W_{n,\nu_k}(\pmb {s},\pmb {a})\rightarrow 0$  , we have
$$
\lim  \sup  _ {n \rightarrow \infty} \| Q _ {n} ^ {\prime} \| \leq \gamma D _ {k} <   D _ {k + 1}.
$$
Therefore, there exists some time  $n_{k + 1}$  such that
$$
- D _ {k + 1} \leq Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) \leq \min  \left\{D _ {k + 1}, Q _ {n} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}) \right\} \forall (\boldsymbol {s}, \boldsymbol {a}), n \geq n _ {k + 1},
$$
B Weakly Coupled Q-learning Algorithm Description
Algorithm 2 Weekly Coupled Q-learning
1: Input: Lagrange multiplier set  $\Lambda$ , initial state distribution  $S_0$ .  
2: Initialize Q-table estimates  $Q_0, \{Q_{0,i}\}_{i=1}^N$ . Set  $Q_0' = Q_0$ .  
3: for  $n = 0,1,2,\ldots$  do  
4: Take an  $\epsilon$ -greedy behavioral action  $a_n$  with respect to  $Q_n'(s_n,a)$ .  
5: // Estimate upper bound by combining subagents  
6: for  $i = 1,2,\ldots,N$  do  
7: Update each subproblem value functions  $Q_{i,n+1}^{\lambda}$  according to (7).  
8: end for  
9: Update right-hand-side estimate  $B_{n+1}(w_n)$  according to (8).  
10: Using (9) and (10), combine subproblems to obtain  $Q_{n+1}^{\lambda^*}(s_n,a)$  for all  $a \in \mathcal{A}(s_n)$ .  
11: // Main agent standard update, followed by projection  
12: Do standard Q-learning update using (11) to obtain  $Q_{n+1}$ .  
13: Perform upper bound projection step:  $Q_{n+1}'(s,a) = Q_{n+1}^{\lambda^*}(s,a) \land Q_{n+1}(s,a)$   
14: end for
C Weakly Coupled DQN Algorithm Implementation
In our implementation of WCDQN, the subproblem  $Q_{i}^{\lambda}$ -network in Algorithm 1 follows the standard network architecture as in [43], where given an input state  $s_i$  the network predicts the  $Q$ -values for all actions. This mandates that all the subproblems have the same number of actions. To address different subproblem action spaces, we can change the network architecture to receive the state-action pair  $(s_i, a_i)$  as input and output the predicted  $Q$ -value. This simple change does not interfere or affect WCDQN's main idea.
Our code is available at https://github.com/ibrahim-elshar/WCDQN_NeurIPS.
D Numerical Experiment Details
A discount factor of 0.9 is used for the EV charging problem and 0.99 for the multi-product inventory and online stochastic ad matching problems. In the tabular setting, we use a polynomial learning rate that depends on the state-action pairs visitation given by  $\alpha_{n}(s,a) = 1 / \nu_{n}(s,a)^{r}$ , where  $\nu_{n}(s,a)$  represent the number of times  $(s,a)$  has been visited up to iteration  $n$ , and  $r = 0.4$ . We also use an  $\epsilon$ -greedy exploration policy, given by  $\epsilon(s) = 1 / \nu(s)^{\epsilon}$ , where  $\nu(s)$  is the number of times the state  $s$  has been visited. We set  $e = 0.4$ . In the function approximation setting, we use an  $\epsilon$ -greedy policy that decays  $\epsilon$  from 1 to 0.05 after 30,000 steps. All state-action value functions are initialized randomly. Experiments were ran on a shared memory cluster with dual 12-core Skylake CPU (Intel Xeon Gold 6126 2.60 GHz) and 192 GB RAM/node.
D.1 EV charging deadline scheduling [63]
In this problem, there are in total three charging spots  $N = 3$ . Each spot represents a subproblem with state  $(c_{t}, B_{t,i}, D_{t,i})$ , where  $c_{t} \in \{0.2, 0.5, 0.8\}$  is the exogenous electric cost,  $B_{t,i} \leq 2$  is the
amount of charge required and  $D_{t,i} \leq 4$  is the remaining time until the EV leaves the system. The state space size is 36 for each subproblem. At a given period  $t$ , the action of each subproblem is whether to charge an EV occupying the charging spot  $a_{t,i} = 1$  or not  $a_{t,i} = 0$ . A feasible action is given by  $\sum_{i=1}^{N} a_{t,i} \leq b(c_t)$ , where  $b(0.2) = 3$ ,  $b(0.5) = 2$ , and  $b(0.8) = 1$ . The reward of each subproblem is given by
$$
r _ {i} \big ((c _ {t}, B _ {t, i}, D _ {t, i}), a _ {t, i} \big) = \left\{ \begin{array}{l l} & (1 - c _ {t})   a _ {t, i} \quad \text {i f}   B _ {t, i} > 0, D _ {t, i} > 1, \\ & (1 - c _ {t})   a _ {t, i} - F (B _ {t, i} - a _ {t, i})   \text {i f}   B _ {t, i} > 0, D _ {t, i} = 1, \\ & 0, \quad \text {o t h e r w i s e}, \end{array} \right.
$$
where  $F(B_{t,i} - a_{t,i}) = 0.2(B_{t,i} - a_{t,i})^2$  is a penalty function for failing to complete charging the EV before the deadline. The endogenous state of each subproblem evolves such that  $(B_{t+1,i}, D_{t+1,i}) = (B_{t,i} - a_{t,i}, D_{t,i} - 1)$  if  $D_{t,i} > 1$ , and  $(B_{t+1,i}, D_{t+1,i}) = (B, D)$  with probability  $q(D, B)$  if  $D_{t,i} \leq 1$ , where  $q(0,0) = 0.3$  and  $q(B, D) = 0.7/11$  for all  $B > 0$  and  $D > 0$ . The exogenous state  $c_t$  evolves following the transition probabilities given by:
$$
q (c _ {t + 1} \mid c _ {t}) = \left( \begin{array}{c c c} 0. 4 & 0. 3 & 0. 3 \\ 0. 2 & 0. 5 & 0. 3 \\ 0. 6 & 0. 2 & 0. 2 \end{array} \right).
$$
D.2 Multi-product inventory control with an exogenous production rate [27]