SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ibrahim El Shar
Hitachi America Ltd., University of Pittsburgh
ibrahim.elshar@hal.hitachi.com
Daniel Jiang
Meta, University of Pittsburgh
drjiang@meta.com
We propose weakly coupled deep  $Q$ -networks (WCDQN), a novel deep reinforcement learning algorithm that enhances performance in a class of structured problems called weakly coupled Markov decision processes (WCMDP). WCMDPs consist of multiple independent subproblems connected by an action space constraint, which is a structural property that frequently emerges in practice. Despite this appealing structure, WCMDPs quickly become intractable as the number of subproblems grows. WCDQN employs a single network to train multiple DQN "subagents," one for each subproblem, and then combine their solutions to establish an upper bound on the optimal action value. This guides the main DQN agent towards optimality. We show that the tabular version, weakly coupled Q-learning (WCQL), converges almost surely to the optimal action value. Numerical experiments show faster convergence compared to DQN and related techniques in settings with as many as 10 subproblems,  $3^{10}$  total actions, and a continuous state space.
Despite achieving many noteworthy and highly visible successes, it remains widely acknowledged that practical implementation of reinforcement learning (RL) is, in general, challenging [15]. This is particularly true in real-world settings where, unlike in simulated settings, interactions with the environment are costly to obtain. One promising path toward more sample-efficient learning in real-world situations is to incorporate known structural properties of the underlying Markov decision process (MDP) into the learning algorithm. As elegantly articulated by [44], structural properties can be considered a type of "side information" that can be exploited by the RL agent for its benefit. Instantiations of this concept are plentiful and diverse: examples include factored decompositions [33, 10, 47], latent or contextual MDPs [21, 39, 52], block MDPs [14], linear MDPs [32], shape-constrained value and/or policy functions [49, 37, 31], MDPs adhering to closure under policy improvement [8], and multi-timescale or hierarchical MDPs [23, 13], to name just a few.
In this paper, we focus on a class of problems called weakly coupled MDPs (WCMDPs) and show how one can leverage their inherent structure through a tailored RL approach. WCMDPs, often studied in the field of operations research, consist of multiple subproblems that are independent from each other except for a coupling constraint on the action space [24]. This type of weakly coupled structure frequently emerges in practice, spanning domains like supply chain management [24], recommender systems [65], online advertising [9], revenue management [53], and stochastic job scheduling [63]. Such MDPs can quickly become intractable when RL algorithms are applied naively, given that their state and action spaces grow exponentially with the number of subproblems [45].
One can compute an upper bound on the optimal value of a WCMDP by performing a Lagrangian relaxation on the action space coupling constraints. Importantly, the weakly coupled structure allows the relaxed problem to be completely decomposed across the subproblems, which are significantly easier to solve than the full MDP [24, 1]. Our goal in this paper is to devise a method that can integrate the Lagrangian relaxation upper bounds into the widely adopted value-based RL approaches of Q-learning
Figure 1: An illustration of our RL approach for WCMDPs. Our approach takes a single "full" transition  $\tau$  (as collected by a standard RL agent) and decomposes it into subproblem transitions  $\tau_{i}$  that are passed to "subagents," which are powered by a single network and aim to solve the easier subproblems. The results of these subagents are then used collectively to guide the main agent toward the optimal policy, whose actions need to satisfy linking constraints. Here, we illustrate the case of a single linking constraint that requires the sum of the actions to be bounded by a right-hand-side quantity  $b(w)$ , where  $w$  is an exogenous state from the environment.
[59] and deep Q-networks (DQN) [43]. Our proposed method is, to our knowledge, the first to explore the use of Lagrangian relaxations to tackle general WcMDPs in a fully model-free, deep RL setting.
Main contributions. We make the following methodological and empirical contributions.
1. First, we propose a novel deep RL algorithm, called weakly coupled deep  $Q$ -networks (WCDQN), that exploits weakly coupled structure by using a set of subagents, each attached to one of the subproblems, whose solutions are combined to help improve the performance of main DQN agent; see Figure 1 for a high-level overview.
2. Second, we also propose and analyze a tabular version of our algorithm called weakly coupled  $Q$ -learning (WCQL), which serves to conceptually motivate WCDQN. We show that WCQL converges almost surely to the optimal action-value.
3. Finally, we conduct numerical experiments on a suite of realistic problems, including electric vehicle charging station optimization, multi-product inventory control, and online stochastic ad matching. The results show that our proposed algorithm outperforms baselines by a relatively large margin in settings with as many as 10 subproblems,  $3^{10}$  total actions, and a continuous state space.
Weakly Coupled MDPs. This line of work began with [61] under the name of restless multi-armed bandits (RMAB), where there are two actions ("active" or "passive") for each subproblem (also known as "project" or "arm"), under a budget constraint on the number of active arms at any given time. As we will see soon, this is a special case of a WCMDP with two actions per subproblem and a single budget constraint. A popular solution approach to RMABs is the Whittle index policy, which was first proposed by [61] and uses the idea of ranking arms by their "marginal productivity." The policy has been extensively studied in the literature from both applied and theoretical perspectives [20, 41, 29, 28, 42, 64]. Whittle conjectured in [61] that the Whittle index policy is asymptotically optimal under a condition called indexability; later, [60] established that asymptotic optimality requires indexability, but also another technical condition, both of which are difficult to verify.
<sup>1</sup>We note that WCMDPs should not be confused with constrained MDPs, where a budget constraint is imposed on the overall cost of the policy in all periods [2].
As discussed in detail by [64], relying on the Whittle index policy in real-world problems can be problematic due to hard-to-verify technical conditions (and if not met, computational robustness and the heuristic's original intuitive motivation may be lost).
A number of recent papers have considered using RL in the setting of RMABs, but nearly all of them are based on Whittle indices [19, 46, 34, 35, 3, 50, 62], and are thus most useful when the indexability condition can be verified. Exceptions are [34] and [35], which propose to run RL directly on the Lagrangian relaxation of the true problem to obtain a "Lagrange policy." Our paper is therefore closest in spirit to these two works, but our methods target the optimal value and policy (with the help of Lagrangian relaxation) rather than pursuing the Lagrange policy as the end goal (which does not have optimality guarantees in general). Moreover, compared to the other RL approaches mentioned above, we do not require the indexability condition and our method works for any WCMDP.
Relaxations of WCMDPs can be performed in several different ways, including approximate linear programming (ALP) [1, 11], network relaxation [45], and Lagrangian relaxation [61, 53, 24, 7, 1, 54, 11]. Notably, [1] provided key results for the ALP and Lagrangian relaxation approaches, and [11] gave theoretical justification for the closeness of the bounds obtained by the approximate linear programming and Lagrangian relaxation approaches, an empirical observation made in [1]. Our work focuses specifically on the Lagrangian relaxation approach, which relaxes the linking constraints on the action space by introducing a penalty in the objective.
DQN and Q-learning. The Q-learning algorithm [59] is perhaps the most popular value-based tabular RL algorithm [30, 55, 6], and the DQN approach of [43] extends the fundamental ideas behind Q-learning to the case where Q-functions are approximated using deep neural networks, famously demonstrated on a set of Atari games. Unfortunately, practical implementation of Q-learning, DQN, and their extensions on real-world problems can be difficult due to the large number of samples required for learning [44].
Various papers have attempted to extend and enhance the DQN algorithm. For example, to overcome the over-estimation problem and improve stability, [57] proposes double DQN, which adapts the tabular approach of double Q-learning from [22] to the deep RL setting. The main idea is to use a different network for the action selection and evaluation steps. [51] modifies the experience replay buffer sampling to prioritize certain tuples, and [58] adds a "dueling architecture" to double DQN that combines two components, an estimator for the state value function and an estimator for the state-dependent action advantage function. Other examples include bootstrapped DQN [48], amortized Q-learning [56], distributional RL [5], and rainbow DQN [26].
Our approach, WCDQN, is also an enhancement of DQN, but differ from the above works in that our focus is on modifying DQN to exploit the structure of a class of important problems that are otherwise intractable, while the existing papers focus on improvements made to certain components of the DQN algorithm (e.g., network architecture, experience replay buffer, exploration strategy). In particular, it should be possible to integrate the main ideas of WCDQN into variations of DQN without much additional work.
Use of constraints and projections in RL. WCDQN relies on constraining the learned Q-function to satisfy a learned upper bound. The work of [25] uses a similar constrained optimization approach to enforce upper and lower bounds on the optimal action value function in DQN. Their bounds are derived by exploiting multistep returns of a general MDP, while ours are due to dynamically-computed Lagrangian relaxations. [25] also does not provide any convergence guarantees for their approach.
In addition, [16] proposed a convergent variant of Q-learning that leverages upper and lower bounds derived using the information relaxation technique of [12] to improve performance of tabular Q-learning. Although our work shares the high-level idea of bounding Q-learning iterates, [16] focused on problems with partially known transition models (which are necessary for information relaxation) and the approach did not readily extend to the function approximation setting. Besides focusing on a different set of problems (WCMDPs), our proposed approach is model-free and naturally integrates with DQN.
In this section, we give some background on WCMDPs, Q-learning, DQN, and the Lagrangian relaxation approach. All proofs throughout the rest of the paper are given in Appendix A.
3.1 Weakly Coupled MDPs
We study an infinite horizon WCMDP with state space  $S = \mathcal{X} \times \mathcal{W}$  and finite action space  $\mathcal{A}$ , where  $\mathcal{X}$  is the endogenous part (i.e., affected by the agent's actions) and  $\mathcal{W}$  is the exogenous part (i.e., unaffected by the agent's actions) of the full state space. We use the general setup of WCMDPs from [11]. A WCMDP can be decomposed into  $N$  subproblems. The state space of subproblem  $i$  is denoted by  $S_{i} = \mathcal{X}_{i} \times \mathcal{W}$  and the action space is denoted by  $\mathcal{A}_{i}$ , such that
$$
\mathcal {X} = \otimes_ {i = 1} ^ {N} \mathcal {X} _ {i} \quad \text {a n d} \quad \mathcal {A} = \otimes_ {i = 1} ^ {N} \mathcal {A} _ {i}.
$$
In each period, the decision maker observes an exogenously and independently evolving state  $w \in \mathcal{W}$ , along with the endogenous states  $\boldsymbol{x} = (x_{1}, x_{2}, \ldots, x_{N})$ , where  $x_{i} \in \mathcal{X}_{i}$  is associated with subproblem  $i$ . Note that  $w$  is shared by all of the subproblems, and this is reflected in the notation we use throughout the paper, where  $s = (x, w) \in S$  represents the full state and  $s_{i} = (x_{i}, w)$  is the state of subproblem  $i$ . In addition to the exogenous state  $w$  being shared across subproblems, there also exist  $L$  linking or coupling constraints that connect the subproblems: they take the form  $\sum_{i=1}^{N} d(s_{i}, a_{i}) \leq b(w)$ , where  $d(s_{i}, a_{i}), b(w) \in \mathbb{R}^{L}$  and  $a_{i} \in \mathcal{A}_{i}$  is the component of the action associated with subproblem  $i$ . The set of feasible actions for state  $s$  is given by
$$
\mathcal {A} (\boldsymbol {s}) = \left\{\boldsymbol {a} \in \mathcal {A}: \sum_ {i = 1} ^ {N} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) \leq \boldsymbol {b} (w) \right\}. \tag {1}
$$
After observing state  $s = (x, w)$ , the decision maker selects a feasible action  $\mathbf{a} \in \mathcal{A}(s)$ .
The transition probabilities for the endogenous component is denoted  $p(\pmb{x}' \mid \pmb{x}, \pmb{a})$  and we assume that transitions are conditionally independent across subproblems:
$$
p \left(\boldsymbol {x} ^ {\prime} \mid \boldsymbol {x}, \boldsymbol {a}\right) = \Pi_ {i = 1} ^ {N} p _ {i} \left(x _ {i} ^ {\prime} \mid x _ {i}, a _ {i}\right),
$$
where  $p_i(x_i' \mid x_i, a_i)$  are the transition probabilities for subproblem  $i$ . The exogenous state transitions according to  $q(w' \mid w)$ . Next, let  $r_i(s_i, a_i)$  be the reward of subproblem  $i$  and let  $r(\mathbf{s}, \mathbf{a}) = \{r_i(s_i, a_i)\}_{i=1}^N$ . The reward of the overall system is additive:  $r(\mathbf{s}, \mathbf{a}) = \sum_{i=1}^{N} r_i(s_i, a_i)$ .
Given a discount factor  $\gamma \in [0,1)$  and a feasible policy  $\pi : S \to \mathcal{A}$  that maps each state  $s$  to a feasible action  $\pmb{a} \in \mathcal{A}(s)$ , the value (cumulative discounted reward) of following  $\pi$  when starting in state  $s$  and taking a first action  $\pmb{a}$  is given by the action-value function  $Q^{\pi}(s, \pmb{a}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} r(\pmb{s}_{t}, \pmb{a}_{t}) \mid \pi, s_{0} = s, \pmb{a}_{0} = \pmb{a}\right]$ . Our goal is to find an optimal policy  $\pi^{*}$ , i.e., one that maximizes  $V^{\pi}(s) = Q^{\pi}(s, \pi(s))$ . We let  $Q^{*}(s, \pmb{a}) = \max_{\pi} Q^{\pi}(s, \pmb{a})$  and  $V^{*}(s) = \max_{\pi} V^{\pi}(s)$  be the optimal action-value and value functions, respectively. It is well-known that the optimal policy selects actions in accordance to  $\pi^{*}(s) = \arg \max_{\pmb{a}} Q^{*}(s, \pmb{a})$  and that the Bellman recursion holds:
$$
Q ^ {*} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A} \left(\boldsymbol {s} ^ {\prime}\right)} Q ^ {*} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) \right], \tag {2}
$$
where  $s' = (x', w')$  is distributed according to  $p(\cdot \mid x, a)$  and  $q(\cdot \mid w)$ .
3.2 Q-learning and DQN
The Q-learning algorithm of [59] is a tabular approach that attempts to learn the optimal action-value function  $Q^{*}$  using stochastic approximation on (2). Using a learning rate  $\alpha_{n}$ , the update of the approximation  $Q_{n}$  from iteration  $n$  to  $n + 1$  is:
$$
Q _ {n + 1} \left(\boldsymbol {s} _ {n}, \boldsymbol {a} _ {n}\right) = Q _ {n} \left(\boldsymbol {s} _ {n}, \boldsymbol {a} _ {n}\right) + \alpha_ {n} \left(\boldsymbol {s} _ {n}, \boldsymbol {a} _ {n}\right) \left[ y _ {n} - Q _ {n} \left(\boldsymbol {s} _ {n}, \boldsymbol {a} _ {n}\right) \right],
$$
where  $y_{n} = r_{n} + \gamma \max_{\pmb{a}^{\prime}}Q_{n}(\pmb{s}_{n + 1},\pmb{a}^{\prime})$  is the target value, computed using the observed reward  $r_n$  at  $(\pmb {s}_n,\pmb {a}_n)$ , the transition to  $\pmb {s}_{n + 1}$ , and the current value estimate  $Q_{n}$ .
The DQN approach of [43] approximates  $Q^{*}$  via a neural network  $Q(s, a; \theta)$  with network weights  $\theta$ . The loss function used to learn  $\theta$  is directly based on minimizing the discrepancy between the two sides of (2):
$$
l (\theta) = \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho} \left[ \left(y - Q (\boldsymbol {s}, \boldsymbol {a}; \theta)\right) ^ {2} \right],
$$
where  $y = r(\pmb{s},\pmb{a}) + \gamma \mathbb{E}\big[\max_{\pmb{a}'}Q(\pmb{s}',\pmb{a}';\theta^{-})\big]$ ,  $\theta^{-}$  are frozen network weights from a previous iteration, and  $\rho$  is a behavioral distribution [43]. In practice, we sample experience tuples  $(s_n,a_n,r_n,s_{n + 1})$  from a replay buffer and perform a stochastic gradient update:
$$
\theta_ {n + 1} = \theta_ {n} + \alpha_ {n} \left[ y _ {n} - Q \left(\boldsymbol {s} _ {n}, \boldsymbol {a} _ {n}; \theta\right) \right] \nabla_ {\theta} Q \left(\boldsymbol {s} _ {n}, \boldsymbol {a} _ {n}; \theta\right),
$$
with  $y_{n} = r_{n} + \gamma \max_{\pmb{a}^{\prime}}Q(\pmb{s}_{n + 1},\pmb{a}^{\prime};\theta^{-})$ . Note the resemblance of this update to that of Q-learning.
3.3 Lagrangian Relaxation
The Lagrangian relaxation approach decomposes WCMDPs by relaxing the linking constraints to obtain separate, easier-to-solve subproblems [1]. The main idea is to dualize the linking constraints  $\sum_{i=1}^{N} d(s_i, a_i) \leq b(w)$  using a penalty vector  $\lambda \in \mathbb{R}_+^L$ . The result is an augmented objective consisting of the original objective plus additional terms that penalize constraint violations. The Bellman equation of the relaxed MDP in (2) is given by:
$$
Q ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \lambda^ {\intercal} \left[ \boldsymbol {b} (w) - \sum_ {i = 1} ^ {N} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) \right] + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A}} Q ^ {\lambda} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) \right]. \tag {3}
$$
With the linking constraints removed, this relaxed MDP can be decomposed across subproblems, so we are able to define the following recursion for each subproblem  $i$ :
$$
Q _ {i} ^ {\lambda} \left(s _ {i}, a _ {i}\right) = r _ {i} \left(s _ {i}, a _ {i}\right) - \lambda^ {\mathsf {T}} \boldsymbol {d} \left(s _ {i}, a _ {i}\right) + \gamma \mathbb {E} \left[ \max  _ {a _ {i} ^ {\prime} \in \mathcal {A} _ {i}} Q _ {i} ^ {\lambda} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right) \right]. \tag {4}
$$
It is well-known from classical results that any penalty vector  $\lambda \geq 0$  produces an MDP whose optimal value function is an upper bound on the  $V^{*}(s)$  [24, 1]. The upcoming proposition is a small extension of these results to the case of action-value functions, which is necessarily for Q-learning.
Proposition 1. For any  $\lambda \geq 0$  and  $s\in S$ , it holds that  $Q^{*}(s,a)\leq Q^{\lambda}(s,a)$  for any  $a\in \mathcal{A}(s)$ . In addition, the Lagrangian action-value function of (3) satisfies
$$
Q ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = \lambda^ {\intercal} \boldsymbol {B} (w) + \sum_ {i = 1} ^ {N} Q _ {i} ^ {\lambda} \left(s _ {i}, a _ {i}\right) \tag {5}
$$
where  $Q_{i}^{\lambda}(s_{i},a_{i})$  is as defined in (4) and  $B(w)$  satisfies the recursion
$$
\boldsymbol {B} (w) = \boldsymbol {b} (w) + \gamma \mathbb {E} \left[ \boldsymbol {B} \left(w ^ {\prime}\right) \right], \tag {6}
$$
with the exogenous next state  $w'$  is distributed according to  $q(\cdot | w)$ .
The first part of the proposition is often referred to as weak duality and the second part shows how the Lagrangian relaxation can be solved by decomposing it across subproblems, dramatically reducing the computational burden. The tightest upper bound is the solution of the Lagrangian dual problem,  $Q^{\lambda^{*}}(s,a) = \min_{\lambda \geq 0}Q^{\lambda}(s,a)$ , where  $\lambda^{*}$  is minimizer.
4 Weakly Coupled Q-learning
In this section, we introduce the tabular version of our RL algorithm, called weakly coupled  $Q$ -learning (WCQL), which will illustrate the main concepts of the deep RL version, WCDQN.
We first state an assumption on when the linking constraint (1), which determines the feasible actions given a state, is observed.
Assumption 1 (Linking constraint observability; general setting). Suppose that upon landing in a state  $s = (\pmb{x}, \pmb{w})$ , the agent observes the possible constraint left-hand-side values  $\pmb{d}(s_i, \cdot)$  for every  $i$ , along with the constraint right-hand-side  $\pmb{b}(w) \in \mathbb{R}^L$ .
Under Assumption 1, the agent is able to determine the feasible action set  $\mathcal{A}(s)$  upon landing in state  $s$ . Accordingly, it can always take a feasible action. In many cases, it is known in advance that the feasible action set is of the multi-action RMAB form: there is a single linking constraint (i.e.,  $L = 1$ ) and the left-hand-side is the sum of subproblem actions (i.e.,  $d(s_{i},a_{i}) = a_{i}$ ). In that case, Assumption 1 reduces to the following simpler statement, which we state for completeness.
Assumption  $1'$  (Linking constraint observability; multi-action RMAB setting). Suppose that we are in a multi-action RMAB setting. When the agent lands in a state  $\boldsymbol{s} = (\boldsymbol{x},\boldsymbol{w})$ , it observes the constraint right-hand-side  $\boldsymbol{b}(w)\in \mathbb{R}$ .
In the numerical example applications of Section 6, for illustrative simplicity, we choose to focus on single-constraint settings where Assumption  $1^{\prime}$  is applicable. Note that the "difficulty" of WCMDPs is largely determined by the number of subproblems and the size of the feasible set compared to the full action space, not necessarily by the number of linking constraints. In each of our example applications, Assumption 1 naturally holds: for example, in the EV charging problem, there are a limited number of available charging stations (which is always observable).
An important part of WCQL is to track an estimate of  $Q^{\lambda^{*}}(s, a)$ , the result of the Lagrangian dual problem. To approximate this value, we replace the minimization over all  $\lambda \geq 0$  by optimization over a finite set of possible multipliers  $\Lambda$ , which we consider as an input to our algorithm. In practice, we find that it is most straightforward to use  $\lambda = \lambda' \mathbf{1}$ , where  $\mathbf{1} \in \mathbb{R}^{L}$  is the all ones vector and  $\lambda' \in \mathbb{R}$  but from the algorithm's point of view, any set  $\Lambda$  of nonnegative multipliers will do.
We denote an experience tuple for the entire WCMDP by  $\tau = (s, a, r, b, s')$ . Similarly, we let  $\tau_i = (s_i, a_i, r_i, s_i')$  be the experience relevant to subproblem  $i$ , as described in (4). Note that  $b$  is excluded from  $\tau_i$  because it does not enter subproblem Bellman recursion.
4.2 Algorithm Description
The WCQL algorithm can be decomposed into three main steps.
Subproblems and subagents. First, for each subproblem  $i \in \{1, 2, \dots, N\}$  and every  $\lambda \in \Lambda$ , we attempt to learn an approximation of  $Q_{i}^{\lambda}(s_{i}, a_{i})$  from (4), which are the  $Q$ -values of the unconstrained subproblem associated with  $\lambda$ . We do this by running an instance of Q-learning with learning rate  $\beta_{n}$ . Letting  $Q_{i,n}^{\lambda}$  be the estimate at iteration  $n$ , the update is given by:
$$
Q _ {i, n + 1} ^ {\lambda} \left(s _ {i}, a _ {i}\right) = Q _ {i, n} ^ {\lambda} \left(s _ {i}, a _ {i}\right) + \beta_ {n} \left(s _ {i}, a _ {i}\right) \left[ y _ {i, n} ^ {\lambda} - Q _ {i, n} ^ {\lambda} \left(s _ {i}, a _ {i}\right) \right], \tag {7}
$$
where the target value is defined as  $y_{i,n}^{\lambda} = r_{i}(s_{i},a_{i}) - \lambda^{\mathsf{T}}\pmb {d}(s_{i},a_{i}) + \gamma \max_{a_{i}^{\prime}}Q_{i,n}^{\lambda}(s_{i}^{\prime},a_{i}^{\prime})$