Title: Weakly Coupled Deep Q-Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

Deep Reinforcement Learning

Weakly Coupled MDPs
Abstract: We propose weakly coupled deep Q-networks (WCDQN), a novel deep reinforcement learning algorithm that enhances performance in a class of structured problems called weakly coupled Markov decision processes (WCMDP). WCMDPs consist of multiple independent subproblems connected by an action space constraint, which is a structural property that frequently emerges in practice. Despite this appealing structure, WCMDPs quickly become intractable as the number of subproblems grows. WCDQN employs a single network to train multiple DQN ``subagents,'' one for each subproblem, and then combine their solutions to establish an upper bound on the optimal action value. This guides the main DQN agent towards optimality. We show that the tabular version, weakly coupled Q-learning (WCQL), converges almost surely to the optimal action value. Numerical experiments show faster convergence compared to DQN and related techniques in settings with as many as 10 subproblems, $3^{10}$ total actions, and a continuous state space.

Weakly Coupled Deep Q-Networks

Abstract

We propose weakly coupled deep Q-networks (WCDQN), a novel deep reinforcement learning algorithm that enhances performance in a class of structured problems called weakly coupled Markov decision processes (WCMDP). WCMDPs consist of multiple independent subproblems connected by an action space constraint, which is a structural property that frequently emerges in practice. Despite this appealing structure, WCMDPs quickly become intractable as the number of subproblems grows. WCDQN employs a single network to train multiple DQN 'subagents,' one for each subproblem, and then combine their solutions to establish an upper bound on the optimal action value. This guides the main DQN agent towards optimality. We show that the tabular version, weakly coupled Q-learning (WCQL), converges almost surely to the optimal action value. Numerical experiments show faster convergence compared to DQN and related techniques in settings with as many as 10 subproblems, 3 10 total actions, and a continuous state space.

1 Introduction

Despite achieving many noteworthy and highly visible successes, it remains widely acknowledged that practical implementation of reinforcement learning (RL) is, in general, challenging [15]. This is particularly true in real-world settings where, unlike in simulated settings, interactions with the environment are costly to obtain. One promising path toward more sample-efficient learning in real-world situations is to incorporate known structural properties of the underlying Markov decision process (MDP) into the learning algorithm. As elegantly articulated by [44], structural properties can be considered a type of 'side information' that can be exploited by the RL agent for its benefit. Instantiations of this concept are plentiful and diverse: examples include factored decompositions [33, 10, 47], latent or contextual MDPs [21, 39, 52], block MDPs [14], linear MDPs [32], shape-constrained value and/or policy functions [49, 37, 31], MDPs adhering to closure under policy improvement [8], and multi-timescale or hierarchical MDPs [23, 13], to name just a few.

In this paper, we focus on a class of problems called weakly coupled MDPs (WCMDPs) and show how one can leverage their inherent structure through a tailored RL approach. WCMDPs, often studied in the field of operations research, consist of multiple subproblems that are independent from each other except for a coupling constraint on the action space [24]. This type of weakly coupled structure frequently emerges in practice, spanning domains like supply chain management [24], recommender systems [65], online advertising [9], revenue management [53], and stochastic job scheduling [63]. Such MDPs can quickly become intractable when RL algorithms are applied naively, given that their state and action spaces grow exponentially with the number of subproblems [45].

One can compute an upper bound on the optimal value of a WCMDP by performing a Lagrangian relaxation on the action space coupling constraints. Importantly, the weakly coupled structure allows the relaxed problem to be completely decomposed across the subproblems, which are significantly easier to solve than the full MDP [24, 1]. Our goal in this paper is to devise a method that can integrate the Lagrangian relaxation upper bounds into the widely adopted value-based RL approaches of Q-learning

Daniel Jiang

[59] and deep Q-networks (DQN) [43]. Our proposed method is, to our knowledge, the first to explore the use of Lagrangian relaxations to tackle general WCMDPs in a fully model-free, deep RL setting.

Main contributions. We make the following methodological and empirical contributions.

First, we propose a novel deep RL algorithm, called weakly coupled deep Q-networks (WCDQN), that exploits weakly coupled structure by using a set of subagents , each attached to one of the subproblems, whose solutions are combined to help improve the performance of main DQN agent; see Figure 1 for a high-level overview.

Second, we also propose and analyze a tabular version of our algorithm called weakly coupled Q-learning (WCQL), which serves to conceptually motivate WCDQN. We show that WCQL converges almost surely to the optimal action-value.

Finally, we conduct numerical experiments on a suite of realistic problems, including electric vehicle charging station optimization, multi-product inventory control, and online stochastic ad matching. The results show that our proposed algorithm outperform baselines by a relatively large margin in settings with as many as 10 subproblems, 3 10 total actions, and a continuous state space.

2 Related Literature

Weakly Coupled MDPs. This line of work began with [61] under the name of restless multi-armed bandits (RMAB), where there are two actions ('active' or 'passive') for each subproblem (also known as 'project' or 'arm'), under a budget constraint on the number of active arms at any given time. 1 As we will see soon, this is a special case of a WCMDP with two actions per subproblem and a single budget constraint. A popular solution approach to RMABs is the Whittle index policy, which was first proposed by [61] and uses the idea of ranking arms by their 'marginal productivity.' The policy has been extensively studied in the literature from both applied and theoretical perspectives [20, 41, 29, 28, 42, 64]. Whittle conjectured in [61] that the Whittle index policy is asymptotically optimal under a condition called indexability ; later, [60] established that asymptotic optimality requires indexability, but also another technical condition, both of which are difficult to verify.

As discussed in detail by [64], relying on the Whittle index policy in real-world problems can be problematic due to hard-to-verify technical conditions (and if not met, computational robustness and the heuristic's original intuitive motivation may be lost).

A number of recent papers have considered using RL in the setting of RMABs, but nearly all of them are based on Whittle indices [19, 46, 34, 35, 3, 50, 62], and are thus most useful when the indexability condition can be verified. Exceptions are [34] and [35], which propose to run RL directly on the Lagrangian relaxation of the true problem to obtain a 'Lagrange policy.' Our paper is therefore closest in spirit to these two works, but our methods target the optimal value and policy (with the help of Lagrangian relaxation) rather than pursuing the Lagrange policy as the end goal (which does not have optimality guarantees in general). Moreover, compared to the other RL approaches mentioned above, we do not require the indexability condition and our method works for any WCMDP.

Relaxations of WCMDPs can be performed in several different ways, including approximate linear programming (ALP) [1, 11], network relaxation [45], and Lagrangian relaxation [61, 53, 24, 7, 1, 54, 11]. Notably, [1] provided key results for the ALP and Lagrangian relaxation approaches, and [11] gave theoretical justification for the closeness of the bounds obtained by the approximate linear programming and Lagrangian relaxation approaches, an empirical observation made in [1]. Our work focuses specifically on the Lagrangian relaxation approach, which relaxes the linking constraints on the action space by introducing a penalty in the objective.

DQN and Q-learning. The Q-learning algorithm [59] is perhaps the most popular value-based tabular RL algorithm [30, 55, 6], and the DQN approach of [43] extends the fundamental ideas behind Q-learning to the case where Q-functions are approximated using deep neural networks, famously demonstrated on a set of Atari games. Unfortunately, practical implementation of Q-learning, DQN, and their extensions on real-world problems can be difficult due to the large number of samples required for learning [44].

Various papers have attempted to extend and enhance the DQN algorithm. For example, to overcome the over-estimation problem and improve stability, [57] proposes double DQN , which adapts the tabular approach of double Q-learning from [22] to the deep RL setting. The main idea is to use a different network for the action selection and evaluation steps. [51] modifies the experience replay buffer sampling to prioritize certain tuples, and [58] adds a 'dueling architecture' to double DQN that combines two components, an estimator for the state value function and an estimator for the statedependent action advantage function. Other examples include bootstrapped DQN [48], amortized Q-learning [56], distributional RL [5], and rainbow DQN [26].

Our approach, WCDQN, is also an enhancement of DQN, but differ from the above works in that our focus is on modifying DQN to exploit the structure of a class of important problems that are otherwise intractable, while the existing papers focus on improvements made to certain components of the DQN algorithm (e.g., network architecture, experience replay buffer, exploration strategy). In particular, it should be possible to integrate the main ideas of WCDQN into variations of DQN without much additional work.

Use of constraints and projections in RL. WCDQNrelies on constraining the learned Q-function to satisfy a learned upper bound. The work of [25] uses a similar constrained optimization approach to enforce upper and lower bounds on the optimal action value function in DQN. Their bounds are derived by exploiting multistep returns of a general MDP, while ours are due to dynamically-computed Lagrangian relaxations. [25] also does not provide any convergence guarantees for their approach.

In addition, [16] proposed a convergent variant of Q-learning that leverages upper and lower bounds derived using the information relaxation technique of [12] to improve performance of tabular Q-learning. Although our work shares the high-level idea of bounding Q-learning iterates, [16] focused on problems with partially known transition models (which are necessary for information relaxation) and the approach did not readily extend to the function approximation setting. Besides focusing on a different set of problems (WCMDPs), our proposed approach is model-free and naturally integrates with DQN.

3 Preliminaries

In this section, we give some background on WCMDPs, Q-learning, DQN, and the Lagrangian relaxation approach. All proofs throughout the rest of the paper are given in Appendix A.

3.1 Weakly Coupled MDPs

We study an infinite horizon WCMDP with state space S = X × W and finite action space A , where X is the endogenous part (i.e., affected by the agent's actions) and W is the exogenous part (i.e., unaffected by the agent's actions) of the full state space. We use the general setup of WCMDPs from [11]. A WCMDP can be decomposed into N subproblems. The state space of subproblem i is denoted by S i = X i ×W and the action space is denoted by A i , such that

In each period, the decision maker observes an exogenously and independently evolving state w ∈ W , along with the endogenous states x = ( x 1 , x 2 , . . . , x N ) , where x i ∈ X i is associated with subproblem i . Note that w is shared by all of the subproblems, and this is reflected in the notation we use throughout the paper, where s = ( x , w ) ∈ S represents the full state and s i = ( x i , w ) is the state of subproblem i . In addition to the exogenous state w being shared across subproblems, there also exist L linking or coupling constraints that connect the subproblems: they take the form ∑ N i =1 d ( s i , a i ) ≤ b ( w ) , where d ( s i , a i ) , b ( w ) ∈ R L and a i ∈ A i is the component of the action associated with subproblem i . The set of feasible actions for state s is given by

After observing state s = ( x , w ) , the decision maker selects a feasible action a ∈ A ( s ) .

The transition probabilities for the endogenous component is denoted p ( x ′ | x , a ) and we assume that transitions are conditionally independent across subproblems:

where p i ( x ′ i | x i , a i ) are the transition probabilities for subproblem i . The exogenous state transitions according to q ( w ′ | w ) . Next, let r i ( s i , a i ) be the reward of subproblem i and let r ( s , a ) = { r i ( s i , a i ) } N i =1 . The reward of the overall system is additive: r ( s , a ) = ∑ N i =1 r i ( s i , a i ) .

Given a discount factor γ ∈ [0 , 1) and a feasible policy π : S → A that maps each state s to a feasible action a ∈ A ( s ) , the value (cumulative discounted reward) of following π when starting in state s and taking a first action a is given by the action-value function Q π ( s , a ) = E [∑ ∞ t =0 γ t r ( s t , a t ) | π, s 0 = s , a 0 = a ] . Our goal is to find an optimal policy π ∗ , i.e., one that maximizes V π ( s ) = Q π ( s , π ( s )) . We let Q ∗ ( s , a ) = max π Q π ( s , a ) and V ∗ ( s ) = max π V π ( s ) be the optimal action-value and value functions, respectively. It is well-known that the optimal policy selects actions in accordance to π ∗ ( s ) = arg max a Q ∗ ( s , a ) and that the Bellman recursion holds:

where s ′ = ( x ′ , w ′ ) is distributed according to p ( · | x , a ) and q ( · | w ) .

3.2 Q-learning and DQN

The Q-learning algorithm of [59] is a tabular approach that attempts to learn the optimal action-value function Q ∗ using stochastic approximation on (2). Using a learning rate α n , the update of the approximation Q n from iteration n to n +1 is:

where y n = r n + γ max a ′ Q n ( s n +1 , a ′ ) is the target value, computed using the observed reward r n at ( s n , a n ) , the transition to s n +1 , and the current value estimate Q n .

The DQN approach of [43] approximates Q ∗ via a neural network Q ( s , a ; θ ) with network weights θ . The loss function used to learn θ is directly based on minimizing the discrepancy between the two sides of (2):

where y = r ( s , a ) + γ E [ max a ′ Q ( s ′ , a ′ ; θ -) ] , θ -are frozen network weights from a previous iteration, and ρ is a behavioral distribution [43]. In practice, we sample experience tuples ( s n , a n , r n , s n +1 ) from a replay buffer and perform a stochastic gradient update:

with y n = r n + γ max a ′ Q ( s n +1 , a ′ ; θ -) . Note the resemblance of this update to that of Q-learning.

3.3 Lagrangian Relaxation

The Lagrangian relaxation approach decomposes WCMDPs by relaxing the linking constraints to obtain separate, easier-to-solve subproblems [1]. The main idea is to dualize the linking constraints ∑ N i =1 d ( s i , a i ) ≤ b ( w ) using a penalty vector λ ∈ R L + . The result is an augmented objective consisting of the original objective plus additional terms that penalize constraint violations. The Bellman equation of the relaxed MDP in (2) is given by:

With the linking constraints removed, this relaxed MDP can be decomposed across subproblems, so we are able to define the following recursion for each subproblem i :

It is well-known from classical results that any penalty vector λ ≥ 0 produces an MDP whose optimal value function is an upper bound on the V ∗ ( s ) [24, 1]. The upcoming proposition is a small extension of these results to the case of action-value functions, which is necessarily for Q-learning.

Proposition 1. For any λ ≥ 0 and s ∈ S , it holds that Q ∗ ( s , a ) ≤ Q λ ( s , a ) for any a ∈ A ( s ) . In addition, the Lagrangian action-value function of (3) satisfies

where Q λ i ( s i , a i ) is as defined in (4) and B ( w ) satisfies the recursion

with the exogenous next state w ′ is distributed according to q ( · | w ) .

The first part of the proposition is often referred to as weak duality and the second part shows how the Lagrangian relaxation can be solved by decomposing it across subproblems, dramatically reducing the computational burden. The tightest upper bound is the solution of the Lagrangian dual problem , Q λ ∗ ( s , a ) = min λ ≥ 0 Q λ ( s , a ) , where λ ∗ is minimizer.

4 Weakly Coupled Q-learning

In this section, we introduce the tabular version of our RL algorithm, called weakly coupled Q-learning (WCQL), which will illustrate the main concepts of the deep RL version, WCDQN.

4.1 Setup

We first state an assumption on when the linking constraint (1), which determines the feasible actions given a state, is observed.

Assumption 1 (Linking constraint observability; general setting) . Suppose that upon landing in a state s = ( x , w ) , the agent observes the possible constraint left-hand-side values d ( s i , · ) for every i , along with the constraint right-hand-side b ( w ) ∈ R L .

Under Assumption 1, the agent is able to determine the feasible action set A ( s ) upon landing in state s . Accordingly, it can always take a feasible action. In many cases, it is known in advance that the feasible action set is of the multi-action RMAB form: there is a single linking constraint (i.e., L = 1 ) and the left-hand-side is the sum of subproblem actions (i.e., d ( s i , a i ) = a i ). In that case, Assumption 1 reduces to the following simpler statement, which we state for completeness.

Assumption 1 ′ (Linking constraint observability; multi-action RMAB setting) . Suppose that we are in a multi-action RMAB setting. When the agent lands in a state s = ( x , w ) , it observes the constraint right-hand-side b ( w ) ∈ R .

In the numerical example applications of Section 6, for illustrative simplicity, we choose to focus on single-constraint settings where Assumption 1 ′ is applicable. Note that the 'difficulty' of WCMDPs is largely determined by the number of subproblems and the size of the feasible set compared to the full action space, not necessarily by the number of linking constraints. In each of our example applications, Assumption 1 naturally holds: for example, in the EV charging problem, there are a limited number of available charging stations (which is always observable).

An important part of WCQL is to track an estimate of Q λ ∗ ( s , a ) , the result of the Lagrangian dual problem. To approximate this value, we replace the minimization over all λ ≥ 0 by optimization over a finite set of possible multipliers Λ , which we consider as an input to our algorithm. In practice, we find that it is most straightforward to use λ = λ ′ 1 , where 1 ∈ R L is the all ones vector and λ ′ ∈ R , but from the algorithm's point of view, any set Λ of nonnegative multipliers will do.

We denote an experience tuple for the entire WCMDP by τ = ( s , a , r , b , s ′ ) . Similarly, we let τ i = ( s i , a i , r i , s ′ i ) be the experience relevant to subproblem i , as described in (4). Note that b is excluded from τ i because it does not enter subproblem Bellman recursion.

4.2 Algorithm Description

The WCQL algorithm can be decomposed into three main steps.

Subproblems and subagents. First, for each subproblem i ∈ { 1 , 2 , . . . , N } and every λ ∈ Λ , we attempt to learn an approximation of Q λ i ( s i , a i ) from (4), which are the Q -values of the unconstrained subproblem associated with λ . We do this by running an instance of Q-learning with learning rate β n . Letting Q λ i,n be the estimate at iteration n , the update is given by:

where the target value is defined as y λ i,n = r i ( s i , a i ) -λ ⊺ d ( s i , a i ) + γ max a ′ i Q λ i,n ( s ′ i , a ′ i ) .

Note that although we are running several Q-learning instances, they all make use of a common experience tuple τ split across subproblems, where subproblem i receives the portion τ i . We remind the reader that each subproblem is dramatically simpler than the full MDP, since it operates on smaller state and action spaces ( S i and A i ) instead of S and A .

We refer to the subproblem Q-learning instances as subagents . Therefore, each subagent is associated with a subproblem i and a penalty λ ∈ Λ and aims to learn Q λ i .

Learning the Lagrangian bounds. Next, at the level of the 'main' agent, we combine the approximations Q λ i,n +1 learned by the subagents to form an estimate of the Lagrangian action-value function Q λ , as defined in (5). To do so, we first estimate the quantity B ( w ) of Proposition 1. This can be done using a stochastic approximation step with a learning rate η n , as follows:

where we recall that w and w ′ come from the experience tuple τ , embedded within s and s ′ . Now, using Proposition 1, we approximate Q λ ( s , a ) using

Finally, we estimate an upper bound on Q ∗ by taking the minimum over Λ :

Q-learning guided by Lagrangian bounds. We would now like to make use of the learned upper bound Q λ ∗ n +1 ( s , a ) when performing Q-learning on the full problem. Denote the WCQL estimate of Q ∗ at iteration n by Q ′ n . We first make a standard update towards an intermediate value Q n +1 using learning rate α n :

where y n = r ( s , a ) + γ max a ′ ∈A ( s ′ ) Q ′ n ( s ′ , a ′ ) . To incorporate the bounds that we previously estimated, we then project Q n +1 ( s , a ) to satisfy the estimated upper bound:

where a ∧ b = min { a, b } . The agent now takes an action in the environment using a behavioral policy, such as the ϵ -greedy policy on Q ′ n +1 ( s , a ) .

The motivation behind this projection is as follows: since the subproblems are significantly smaller in terms of state and action spaces compared to the main problem, the subagents are expected to quickly converge. As a result, our upper bound estimates will get better, improving the the action-value estimate of the main Q-learning agent through the projection step. In addition, WCQL can enable a sort of 'generalization' to unseen states by leveraging the weakly-coupled structure. The following example illustrates this piece of intuition.

Example 1. Suppose a WCMDP has N = 3 subproblems with S i = { 1 , 2 , 3 } and A i = { 1 , 2 } for each i , leading to 3 3 · 2 3 = 216 total state action pairs. For the sake of illustration, suppose that the agent has visited states s = (1 , 1 , 1) , s = (2 , 2 , 2) , and s = (3 , 3 , 3) and both actions from each of these states. This means that from the perspective of every subproblem i , the agent has visited all state-action pairs in S i ×A i , which is enough information to produce an estimate of Q λ i ( s i , a i ) for all ( s i , a i ) and, interestingly, an estimate of Q λ ∗ ( s , a ) for every ( s , a ) , despite having visited only a small fraction ( 6 / 216 ) of the possible state-action pairs. This allows the main Q-learning agent to make use of upper bound information at every state-action pair via the projection step (12). The main intuition is that these upper bound values are likely to be more sensible than a randomly initialized value, and therefore, can aid learning.

The above example is certainly contrived, but hopefully illustrates the benefits of decomposition and subsequent projection. We note that, especially in settings where the limiting factor is the ability to collect enough experience, one can trade-off extra computation to derive these bounds and improve RL performance without the need to collect additional experience. The full pseudo-code of the WCQL algorithm is available in Appendix B.

4.3 Convergence Analysis

In this section, we show that WCQL converges to Q ∗ with probability one. First, we state a standard assumption on learning rates and state visitation.

Assumption 2. We assume the following: (i) ∑ ∞ n =0 α n ( s , a ) = ∞ , ∑ ∞ n =0 α 2 n ( s , a ) < ∞ for all ( s , a ) ∈ S × A ; (ii) analogous conditions to (i) hold for { β n ( s i , a i ) } n and { η n ( w ) } n , and (iii) the behavioral policy is such that all state-action pairs ( s , a ) are visited infinitely often w.p. 1 .

Theorem 1. Under Assumptions 1 and 2, the following statements hold with probability one.

(i) For each i and λ ∈ Λ , lim n →∞ Q λ i,n ( s i , a i ) = Q λ i ( s i , a i ) for all ( s i , a i ) ∈ S i ×A i .

(ii) For each λ ∈ Λ , lim n →∞ Q λ n ( s , a ) ≥ Q ∗ ( s , a ) for all ( s , a ) ∈ S × A .

Theorem 1 ensures that each subagent's value functions converge to the subproblem optimal value. Furthermore, it shows that asymptotically, the Lagrangian action-value function given by (9) will be an upper bound on the optimal action-value function Q ∗ of the full problem and that our algorithm will converge to Q ∗ .

5 Weakly Coupled DQN

In this section, we propose our main algorithm weakly coupled DQN (WCDQN), which integrates the main idea of WCQL into a function approximation setting. WCDQN guides DQN using Lagrangian relaxation bounds, implemented using a constrained optimization approach.

Networks. Analogous to WCQL, WCDQN has a main network Q ′ ( s , a ; θ ) that learns the action value of the full problem. In addition to the main network, WCDQN uses a subagent network Q λ i ( s i , a i ; θ U ) network to learn the subproblem action-value functions Q λ i . As in standard DQN, we also have θ -and θ -U , which are versions of θ and θ U frozen from a previous iteration and used for computing target values [43]. The inputs to this network are ( i, λ, s i , a i ) , meaning that we can use a single network to learn the action-value function for all subproblems and λ ∈ Λ simultaneously. The

Lagrangian upper bound and the best upper bound estimates are:

Loss functions. Before diving into the training process, we describe the loss functions used to train each network, as they are instructive toward understanding the main idea behind WCDQN (and how it differs from standard DQN). Consider a behavioral distribution ρ for state-action pairs and a distribution µ over the multipliers Λ .

where the (ideal) target value is

For the main agent, we propose a loss function that adds a soft penalty for violating the upper bound:

where ( · ) + = max( · , 0) , κ U is a coefficient for the soft penalty, and

The penalty encourages the network to satisfy the bounds obtained from the Lagrangian relaxation.

Training process. The training process resembles DQN, with a few modifications. At any iteration, we first take an action using an ϵ -greedy policy using the main network over the feasible actions, store the obtained transition experience τ in the buffer, and update the estimate of B ( w ) following (8). 2 Each network is then updated by taking a stochastic gradient descent step on its associated loss function, where the expectations are approximated by sampling minibatches of experience tuples τ and λ . The penalty coefficient κ U can either be held constant to a positive value or annealed using a schedule throughout the training. The full details are shown in Algorithm 1 and some further details are given in Appendix C.

6 Numerical Experiments

In this section, we evaluate our algorithms on three different WCMDPs. First, we evaluate WCQL on an electric vehicle (EV) deadline scheduling problem with multiple charging spots and compare its performance with several other tabular algorithms: Q-learning (QL), Double Q-learning (Double-QL) [22], speedy Q-learning (SQL) [4], bias-corrected Q-learning (BCQL) [40], and Lagrange policy Qlearning (Lagrangian QL) [34]. We then evaluate WCDQN on two problems, multi-product inventory control and online stochastic ad matching, and compare against standard DQN, Double-DQN, and the optimality-tightening DQN (OTDQN) algorithm 3 of He et al. [25] as baselines. Further details on environment and algorithmic parameters are in Appendix D.

EV charging deadline scheduling [63]. In this problem, a decision maker is responsible for charging electric vehicles (EV) at a charging service center that consists of N = 3 charging spots. An EV enters the system when a charging spot is available and announces the amount of electricity it needs to be charged, denoted B t , along with the time that it will leave the system, denoted D t . The decision maker also faces exogenous, random Markovian processing costs c t . At each period, the action is to decide which EVs to charge in accordance with the period's capacity constraint. For each unit of power provided to an EV, the service center receives a reward 1 -c t . However, if the EV leaves the system with an unfulfilled charge, a penalty is assessed. The goal is to maximize the revenue minus penalty costs.

Algorithm 1 Weekly Coupled DQN

Multi-product inventory control with an exogenous production rate [27]. Consider the problem of resource allocation for a facility that manufactures N = 10 products. Each product i has an independent exogenous demand given by D i , i = 1 , . . . , N . To meet these demands, the products are made to stock. Limited storage R i is available for each product, and holding a unit of inventory per period incurs a cost h i . Unmet demand is backordered at a cost b i if the number of backorders is less than the maximum number of allowable backorders M i . Otherwise, it is lost with a penalty cost l i . The DM needs to allocate a resource level a i ∈ { 0 , 1 , . . . , U } for product i from a shared finite resource quantity U in response to changes in the stock level of each product, denoted by x i ∈ X i = {-M i , -M i + 1 , . . . , R i } . A negative stock level corresponds to the number of backorders. Allocating a resource level a i yields a production rate given by a function ρ i ( a i , p i ) where p i is an exogenous Markovian noise that affects the production rate. The goal is to minimize the total cost, which consists of holding, back-ordering, and lost sales costs.

Online stochastic ad matching [18]. We study the problem of matching N = 6 advertisers to arriving impressions. In each period, an impression of type e t arrives according to a Markov chain. An action a t,i ∈ { 0 , 1 } assigns impression e t to advertiser i , with a constraint that exactly one advertiser is selected: ∑ N i =1 a t,i = 1 . Advertiser states represent the number of remaining ads to display and evolves according to s t +1 ,i = s t,i -a t,i . The objective is to maximize the discounted sum of expected rewards for all advertisers.

In Figure 2, we show how WCQL's projection method helps it learn a more accurate Q function more quickly than competing tabular methods. The first panel, Figure 2A, shows an example evolution of WCQL's projected value function Q ′ , along with the evolution of the upper bound. We compare this to the evolution of the action-value function in absence of the projection step. In the second panel, Figure 2B, we plot the relative error between the learned value functions of various algorithms compared to the optimal value function. Both plots are from the EV charging example. Detailed descriptions of the results are given in the figure's caption.

The results of our numerical experiments are shown in Figure 3. We see that in both the tabular and the function approximation cases, our algorithms outperformed the baselines, with WCQL and WCDQN achieving the best mean episode total rewards amongst all problems. From Figure 3A, we see that although the difference between WCQL and Lagrangian QL is small towards the end of the training process, there are stark differences earlier on. In particular, the performance curve of WCQL shows significantly lower variance, suggesting more robustness across random seeds. Given that WCQL and Lagrangian QL differ only in the projection step, we can attribute the improved stability to the guidance provided by the Lagrangian bounds. Figure 3B shows that for the multi-product inventory problem, the OTDQN, DQN, and Double DQN baselines show extremely noisy performance compared to WCDQN, whose significantly better and stable performance is likely

10

2

10

3

10

4

10

5

Steps

10

5

0

5

10

15

20

25

Q-value

Q

*

[

s

,

a

]

Q

[

s

,

a

]

QL

[

s

,

a

]

Q

*

[

s

,

a

]

(A) WCQL learning process illustration

0

5

10

15

20

25

30

Episodes (x200)

1.0

1.2

1.4

1.6

1.8

Relative Error

QL

Double-QL

SQL

BCQL

WCQL

(B) Relative error comparison

due to the use of faster converging subagents and better use of the collected experience. Similarly, in the online stochastic ad matching problem, WCDQN significantly outperforms all the baselines.

0

10

20

30

40

50

60

Episodes (x100)

5.5

6.0

6.5

7.0

7.5

Total Reward

QL

Double-QL

SQL

BCQL

Lagrangian QL

WCQL

(A) EV charging

0

10

20

30

40

50

Episodes (x100)

1400

1200

1000

800

600

400

Total Reward

DQN

Double DQN

OTDQN

WCDQN

(B) Multi-product inventory control

0

20

40

60

80

100

Episodes (x100)

60

70

80

90

100

Total Reward

DQN

Double DQN

OTDQN

WCDQN

(C) Online ad matching

7 Conclusion

In this study, we propose the WCQL algorithm for learning in weakly coupled MDPs and we show that our algorithm converges to the optimal action-value function. We then propose WCDQN, which extends the idea behind the WCQL algorithm to the function approximation case. Our algorithms are model-free and learn upper bounds on the optimal action-value using a combination of a Lagrangian relaxation and Q-learning. These bounds are then used within a constrained optimization approach to improve performance and make learning more efficient. Our approaches significantly outperforms competing approaches on several benchmark environments.

Acknowledgments and Disclosure of Funding

This research was supported in part by the University of Pittsburgh Center for Research Computing, RRID:SCR_022735, through the resources provided. Specifically, this work used the H2P cluster, which is supported by NSF award number OAC-2117681.

References

Daniel Adelman and Adam J Mersereau. Relaxations of weakly coupled stochastic dynamic programs. Operations Research , 56(3):712-727, 2008.

Eitan Altman. Constrained Markov decision processes . Routledge, 2021.

Konstantin E Avrachenkov and Vivek S Borkar. Whittle index based Q-learning for restless bandits with average reward. Automatica , 139:110186, 2022.

M. G. Azar, R. Munos, M. Ghavamzadaeh, and H. J. Kappen. Speedy Q-learning. In Advances in Neural Information Processing Systems 24 , 2011.

Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning . MIT Press, 2023.

D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming , volume 5. Athena Scientific Belmont, MA, 1996.

Dimitris Bertsimas and Adam J Mersereau. A learning approach for interactive marketing to a customer segment. Operations Research , 55(6):1120-1135, 2007.

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786 , 2019.

Craig Boutilier and Tyler Lu. Budget allocation using weakly coupled, constrained Markov decision processes. 2016.

Craig Boutilier, Richard Dearden, and Moisés Goldszmidt. Stochastic dynamic programming with factored representations. Artificial Intelligence , 121(1-2):49-107, 2000.

David B Brown and Jingwei Zhang. On the strength of relaxations of weakly coupled stochastic dynamic programs. Operations Research , 2022.

David B Brown, James E Smith, and Peng Sun. Information relaxations and duality in stochastic dynamic programs. Operations Research , 58(4-part-1):785-801, 2010.

Hyeong Soo Chang, Pedram Jaefari Fard, Steven I Marcus, and Mark Shayman. Multitime scale Markov decision processes. IEEE Transactions on Automatic Control , 48(6):976-987, 2003.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419-2468, 2021.

Ibrahim El Shar and Daniel Jiang. Lookahead-bounded Q-learning. In International Conference on Machine Learning , pages 8665-8675. PMLR, 2020.

Eyal Even-Dar, Yishay Mansour, and Peter Bartlett. Learning rates for Q-learning. Journal of Machine Learning Research , 5(1), 2003.

Jon Feldman, Aranyak Mehta, Vahab Mirrokni, and Shan Muthukrishnan. Online stochastic matching: Beating 1-1/e. In 2009 50th Annual IEEE Symposium on Foundations of Computer Science , pages 117-126. IEEE, 2009.

Jing Fu, Yoni Nazarathy, Sarat Moka, and Peter G Taylor. Towards Q-learning the Whittle index for restless bandits. In 2019 Australian & New Zealand Control Conference (ANZCC) , pages 249-254. IEEE, 2019.

Kevin D Glazebrook, Diego Ruiz-Hernandez, and Christopher Kirkbride. Some indexable families of restless bandit problems. Advances in Applied Probability , 38(3):643-672, 2006.

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov decision processes. arXiv preprint arXiv:1502.02259 , 2015.

Hado Hasselt. Double Q-learning. Advances in Neural Information Processing Systems , 23, 2010.

Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Hierarchical solution of Markov decision processes using macro-actions. In Uncertainty in Artificial Intelligence , pages 220-229, 1998.

Jeffrey Thomas Hawkins. A Langrangian decomposition approach to weakly coupled dynamic optimization problems and its applications . PhD thesis, Massachusetts Institute of Technology, 2003.

Frank S He, Yang Liu, Alexander G Schwing, and Jian Peng. Learning to play in a day: Faster deep reinforcement learning by optimality tightening. arXiv preprint arXiv:1611.01606 , 2016.

Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

David J Hodge and Kevin D Glazebrook. Dynamic resource allocation in a multi-product make-to-stock production system. Queueing Systems , 67(4):333-364, 2011.

Yu-Pin Hsu. Age of information: Whittle index for scheduling stochastic arrivals. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 2634-2638. IEEE, 2018.

Weici Hu and Peter Frazier. An asymptotically optimal index policy for finite-horizon restless bandits. arXiv preprint arXiv:1707.00205 , 2017.

T. Jaakkola, M. I. Jordan, and S. P. Singh. Convergence of stochastic iterative dynamic programming algorithms. In Advances in Neural Information Processing Systems , pages 703-710, 1994.

Daniel R Jiang and Warren B Powell. An approximate dynamic programming algorithm for monotone value functions. Operations Research , 63(6):1489-1511, 2015.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored MDPs. In IJCAI , volume 16, pages 740-747, 1999.

Jackson A Killian, Arpita Biswas, Sanket Shah, and Milind Tambe. Q-learning Lagrange policies for multi-action restless bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 871-881, 2021.

Jackson A Killian, Lily Xu, Arpita Biswas, and Milind Tambe. Robust restless bandits: Tackling interval uncertainty with deep reinforcement learning. arXiv preprint arXiv:2107.01689 , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Sumit Kunnumkal and Huseyin Topaloglu. Using stochastic approximation methods to compute optimal base-stock levels in inventory control problems. Operations Research , 56(3):646-664, 2008.

Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applications , volume 35. Springer Science & Business Media, 2003.

Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor. RL for latent MDPs: Regret guarantees and a lower bound. Advances in Neural Information Processing Systems , 34:24523-24534, 2021.

D. Lee and W. B. Powell. Bias-corrected Q-learning with multistate extension. IEEE Transactions on Automatic Control , 2019.

Keqin Liu and Qing Zhao. Indexability of restless bandit problems and optimality of Whittle index for dynamic multichannel access. IEEE Transactions on Information Theory , 56(11): 5547-5567, 2010.

Rahul Meshram, D Manjunath, and Aditya Gopalan. On the Whittle index for restless multiarmed hidden Markov bandits. IEEE Transactions on Automatic Control , 63(9):3046-3053, 2018.

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Aditya Mohan, Amy Zhang, and Marius Lindauer. Structure in reinforcement learning: A survey and open problems. arXiv preprint arXiv:2306.16021 , 2023.

Selvaprabu Nadarajah and Andre Augusto Cire. Self-adapting network relaxations for weakly coupled Markov decision processes. Available at SSRN , 2021.

Khaled Nakhleh, Santosh Ganji, Ping-Chun Hsieh, I Hou, Srinivas Shakkottai, et al. Neurwin: Neural Whittle index network for restless bandits via deep RL. Advances in Neural Information Processing Systems , 34:828-839, 2021.

Ian Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored MDPs. Advances in Neural Information Processing Systems , 27, 2014.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems , 29, 2016.

Warren Powell, Andrzej Ruszczy´ nski, and Huseyin Topaloglu. Learning algorithms for separable approximations of discrete stochastic optimization problems. Mathematics of Operations Research , 29(4):814-836, 2004.

Francisco Robledo, Vivek Borkar, Urtzi Ayesta, and Konstantin Avrachenkov. QWI: Q-learning with whittle index. ACM SIGMETRICS Performance Evaluation Review , 49(2):47-50, 2022.

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015.

Lauren N Steimle, David L Kaufman, and Brian T Denton. Multi-model Markov decision processes. IISE Transactions , 53(10):1124-1139, 2021.

Kalyan Talluri and Garrett Van Ryzin. An analysis of bid-price controls for network revenue management. Management Science , 44(11-part-1):1577-1593, 1998.

Huseyin Topaloglu. Using Lagrangian relaxation to compute capacity-dependent bid prices in network revenue management. Operations Research , 57(3):637-649, 2009.

J. N. Tsitsiklis. Asynchronous stochastic approximation and Q-learning. Machine Learning , 16 (3):185-202, 1994.

Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enormous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116 , 2020.

Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning , pages 1995-2003. PMLR, 2016.

C. J. C. H. Watkins. Learning from Delayed Rewards . PhD thesis, King's College, Cambridge, UK, 1989.

Richard R Weber and Gideon Weiss. On an index policy for restless bandits. Journal of Applied Probability , 27(3):637-648, 1990.

Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of Applied Probability , 25(A):287-298, 1988.

Guojun Xiong, Shufan Wang, and Jian Li. Learning infinite-horizon average-reward restless multi-action bandits via index awareness. Advances in Neural Information Processing Systems , 35:17911-17925, 2022.

Zhe Yu, Yunjian Xu, and Lang Tong. Deadline scheduling as restless bandits. IEEE Transactions on Automatic Control , 63(8):2343-2358, 2018.

Xiangyu Zhang and Peter I Frazier. Near-optimality for infinite-horizon restless bandits with many arms. arXiv preprint arXiv:2203.15853 , 2022.

Jiahong Zhou, Shunhui Mao, Guoliang Yang, Bo Tang, Qianlong Xie, Lebin Lin, Xingxing Wang, and Dong Wang. RL-MPCA: A reinforcement learning based multi-phase computation allocation approach for recommender systems. In Proceedings of the ACM Web Conference 2023 , pages 3214-3224, 2023.

Appendix to Weakly Coupled Deep Q-Networks

A Proofs

A.1 Proof of Proposition 1

Proof. We prove part the first part of the proposition (weak duality) by induction. First, define

and suppose we run value iteration for both systems:

It is well-known that, by the value iteration algorithm's convergence,

Consider a state s ∈ S and a feasible action a ∈ A ( s ) . We have,

Suppose Q λ t ( s , a ) ≥ Q ∗ t ( s , a ) holds for all s ∈ S and a ∈ A ( s ) for some t > 0 (induction hypothesis). Then,

Thus, it follows that Q λ ( s , a ) ≥ Q ∗ ( s , a ) .

For the proof of the second part of the proposition, define

We use an induction proof. We have for all ( s , a ) ∈ S × A ,

where Q λ 0 ,i ( s i , a i ) = r i ( s i , a i ) -λ ⊺ d ( s i , a i ) . Similarly, for all ( s , a ) ∈ S × A ,

Continuing in this manner, we arrive at Q λ t ( s , a ) = ∑ N i =1 Q λ t,i ( s i , a i ) + λ ⊺ B t ( w ) . Finally, we have

which follows by the convergence of value iteration.

A.2 Proof of Theorem 1

Proof. First, we define the Bellman operator H :

which is known to be a γ -contraction mapping. Next we define the random noise term

Analogously, for subproblem i ∈ { 1 , . . . , N } , define the subproblem Bellman operator

and random noise term

The update rules of WCQL can then be written as

Parts (i) and (ii). By the iteration described in (21), we know that for a fixed λ , we are running Q-learning on an auxiliary MDP with Bellman operator H i , which encodes a reward r i ( s i , a i ) -λ ⊺ d ( s i , a i ) and the transition dynamics for subproblem i . By the standard result for asymptotic convergence of Q-learning [6], we have

We now prove the result in (ii): lim n →∞ Q λ n ( s , a ) ≥ Q ∗ ( s , a ) . Recall that

By standard stochastic approximation theory, lim n →∞ B n ( w ) = B ( w ) for all w [38]. Combining this with (23), we have lim n →∞ Q λ n ( s , a ) = Q λ ( s , a ) for all ( s , a ) , and to conclude that this limit is an upper bound on Q ∗ ( s , a ) , we apply Proposition 1.

Part (iii). Assume without loss of generality that Q ∗ ( s , a ) = 0 for all state-action pairs ( s , a ) . This can be established by shifting the origin of the coordinate system. We also assume that α n ( s , a ) ≤ 1 for all ( s , a ) and n . We proceed via induction. Note that the iterates Q ′ n ( s , a ) are bounded in the sense that there exists a constant D 0 = R max / (1 -γ ) , R max = max ( s , a ) | r ( s , a ) | , such that | Q ′ n ( s , a ) | ≤ D 0 for all ( s , a ) and iterations n [17]. Define the sequence D k +1 = ( γ + ϵ ) D k , such that γ + ϵ < 1 and ϵ > 0 . Clearly, D k → 0 . Suppose that there exists a random variable n k , representing an iteration threshold such that for all ( s , a ) ,

We will show that there exists some iteration n k +1 such that

which implies that Q ′ n ( s , a ) converges to Q ∗ ( s , a ) = 0 for all ( s , a ) .

By part (ii), we know that for all η > 0 , with probability 1, there exists some finite iteration n 0 such that for all n ≥ n 0 ,

Now, we define an accumulated noise process started at n k by W n k ,n k ( s , a ) = 0 , and

where ξ n ( s , a ) is as defined in (19). Let F n be the entire history of the algorithm up to the point where the step sizes at iteration n are selected. Using Corollary 4.1 in [6] which states that under Assumption 2 on the step size α n ( s , a ) , and if E [ ξ n ( s , a ) | F n ] = 0 and E [ ξ 2 n ( s , a ) | F n ] ≤ A n , where the random variable A n is bounded with probability 1, the sequence W n +1 ,n k ( s , a ) defined in (25) converges to zero, with probability 1. From our definition of the stochastic approximation noise ξ n ( s , a ) in (19), we have

where C is a constant. Then, it follows that

We use the following lemma from [6] to bound the accumulated noise.

Lemma A.1 (Lemma 4.2 in [6]) . For every δ > 0 , with probability one, there exists some n ′ such that | W n,n ′ ( s , a ) | ≤ δ , for all n ≥ n ′ .

Now, by Lemma A.1, let n k ′ ≥ max( n k , n 0 ) such that, for all n ≥ n k ′ we have

Let ν k ≥ n k ′ such that, for all n ≥ ν k , by (24) we have

Define another sequence Y n that starts at iteration ν k .

Note that it is easy to show that the sequence Y n ( s , a ) in (26) is decreasing, bounded below by γD k , and converges to γD k as n →∞ . Now we state the following lemma.

Lemma A.2. For all state-action pairs ( s , a ) and iterations n ≥ ν k , it holds that:

Proof. We focus on the right hand side inequality, the left hand side can be proved similarly. For the base case n = ν k , the statement holds because Y ν k ( s , a ) = D k and W ν k ,ν k ( s , a ) = 0 . We assume it is true for n and show that it continues to hold for n +1 :

where we used ( HQ ′ n ) ≤ γ ∥ Q ′ n ∥ ≤ γD k . Now, we have

The inequality holds because

which completes the proof.

Therefore, there exists some time n k +1 such that

which completes the induction.

B Weakly Coupled Q-learning Algorithm Description

C Weakly Coupled DQN Algorithm Implementation

In our implementation of WCDQN, the subproblem Q λ i -network in Algorithm 1 follows the standard network architecture as in [43], where given an input state s i the network predicts the Q -values for all actions. This mandates that all the subproblems have the same number of actions. To address different subproblem action spaces, we can change the network architecture to receive the state-action pair ( s i , a i ) as input and output the predicted Q -value. This simple change does not interfere or affect WCDQN's main idea.

Our code is available at https://github.com/ibrahim-elshar/WCDQN_NeurIPS .

D Numerical Experiment Details

A discount factor of 0 . 9 is used for the EV charging problem and 0 . 99 for the multi-product inventory and online stochastic ad matching problems. In the tabular setting, we use a polynomial learning rate that depends on the state-action pairs visitation given by α n ( s , a ) = 1 /ν n ( s , a ) r , where ν n ( s , a ) represent the number of times ( s , a ) has been visited up to iteration n , and r = 0 . 4 . We also use an ϵ -greedy exploration policy, given by ϵ ( s ) = 1 /ν ( s ) e , where ν ( s ) is the number of times the state s has been visited. We set e = 0 . 4 . In the function approximation setting, we use an ϵ -greedy policy that decays ϵ from 1 to 0 . 05 after 30 , 000 steps. All state-action value functions are initialized randomly. Experiments were ran on a shared memory cluster with dual 12-core Skylake CPU (Intel Xeon Gold 6126 2.60 GHz) and 192 GB RAM/node.

D.1 EV charging deadline scheduling [63]

In this problem, there are in total three charging spots N = 3 . Each spot represents a subproblem with state ( c t , B t,i , D t,i ) , where c t ∈ { 0 . 2 , 0 . 5 , 0 . 8 } is the exogenous electric cost, B t,i ≤ 2 is the

amount of charge required and D t,i ≤ 4 is the remaining time until the EV leaves the system. The state space size is 36 for each subproblem. At a given period t , the action of each subproblem is whether to charge an EV occupying the charging spot a t,i = 1 or not a t,i = 0 . A feasible action is given by ∑ N i =1 a t,i ≤ b ( c t ) , where b (0 . 2) = 3 , b (0 . 5) = 2 , and b (0 . 8) = 1 . The reward of each subproblem is given by

where F ( B t,i -a t,i ) = 0 . 2 ( B t,i -a t,i ) 2 is a penalty function for failing to complete charging the EV before the deadline. The endogenous state of each subproblem evolves such that ( B t +1 ,i , D t +1 ,i ) = ( B t,i -a t,i , D t,i -1) if D t,i > 1 , and ( B t +1 ,i , D t +1 ,i ) = ( B,D ) with probability q ( D,B ) if D t,i ≤ 1 , where q (0 , 0) = 0 . 3 and q ( B,D ) = 0 . 7 / 11 for all B > 0 and D > 0 . The exogenous state c t evolves following the transition probabilities given by:

D.2 Multi-product inventory control with an exogenous production rate [27]

We consider manufacturing N = 10 products. The exogenous demand D t,i for each product i ∈ { 1 , 2 , . . . , 10 } follows a Poisson distribution with mean value µ i . The maximum storage capacity and the maximum number of allowable backorders (after which lost sales costs incur) for product i are given by R i and M i , respectively.

The state for subproblem i is given by ( x t,i , p t ) , where x t,i ∈ X i = {-M i , -M i +1 , . . . , R i } is the inventory level for product i , and p t is an exogenous and Markovian noise with support [0 . 8 , 1] . A negative stock level corresponds to the number of backorders. For subproblem i , the action a t,i is the number of resources allocated to the product i . The maximum number of resources available for all products is U = 3 , so feasible actions must satisfy ∑ i a t,i ≤ 3 .

Allocating a resource level a t,i yields a production rate ρ i ( a t,i , p t ) = (12 p t a t,i ) / (5 . 971 + a t,i ) . The cost function for product i is c i ( p t , x t,i , a t,i ) and represents the sum of the holding, backorders, and lost sales costs. We let h i , b i , and l i denote the per-unit holding, backorder, and lost sale costs, respectively. The cost function c i ( x t,i , p t , a t,i ) is given by,

where ( . ) + = max( ., 0) . We summarize the cost parameters and the mean demand for each product in Table 1. Finally, the transition for the inventory state of subproblem i is given by

where the exogenous noise p t evolves according to a transition matrix sampled from a Dirichlet distribution whose parameters are each sampled (once per replication) from a Uniform (1 , 5) distribution.

D.3 Online stochastic ad matching [18]

In this problem, a platform needs to match N = 6 advertisers to arriving impressions [18]. An impression e t ∈ E = { 1 , 2 , . . . , 5 } arrives according to a discrete time Markov chain with transition probabilities given by q ( e t +1 | e t ) , where each row of the transition matrix q is sampled from a Dirichlet distribution whose parameters are sampled (once per replication) from Uniform (1 , 20) .

The action a t,i ∈ { 0 , 1 } is whether to assign impression e t to advertiser i or not. The platform can assign an impression to at most one advertiser: ∑ N i =1 a i,t = 1 .

The state of advertiser i , x i,t gives the number of remaining ads to display and evolves according to x t +1 ,i = x t,i -a t,i . The initial state is x 0 = (10 , 11 , 12 , 10 , 14 , 9) . The reward obtained from advertiser i in state s t,i = ( x t,i , e t ) is r i ( s t,i , a t,i ) = l i,e t min( x t,i , a t,i ) , where the parameters l i,e t are sampled (once per replication) from Uniform (1 , 4) .

D.4 Training parameters

Each method was trained for 6,000 episodes for the EV charging problem, 5,000 for the multi-product inventory control problem, and 10,000 episodes for the online stochastic ad matching problem. The episode lengths for the EV charging, online ad stochastic ad matching, and multi-product inventory control problems are 50 , 30 , and 25 , respectively. We performed 5 independent replications.

We use a neural network architecture that consists of two hidden layers, with 64 and 32 hidden units respectively, for all algorithms. A rectified linear unit (ReLU) is used as the activation function for each hidden layer. The Adam optimizer [36] with a learning rate of 1 . 0 × 10 -4 was used. For OTDQN, we use the same parameter settings as in He et al. [25].

For WCDQN, we use a Lagrangian multiplier λ ∈ [0 , 10] , with a 0 . 01 discretization. We also used an experience buffer of size 100 , 000 and initialized it with 10 , 000 experience tuples that were obtained using a random policy. For the WCDQN algorithm, we set the penalty coefficient κ U to 10 , after performing a small amount of manual hyperparameter tuning on the set { 1 , 2 , 4 , 10 } .

D.5 Sensitivity analysis of WCQL with respect to the number of subproblems

We study the performance improvement from WCQL over vanilla Q-learning as the number of subproblems increases for the EV charging problem. We only vary the number of subproblems (from 2 to 5) and keep all other settings as defined in Appendix D.1. The results, given in Table 2, show that the benefits of WCQL become larger as the number of subproblems increases. This provides some additional evidence for the practicality of our approach, especially in regimes where standard methods fail.

E Limitations and Future Work

One interesting direction to explore for future work is to address the limitation of learning the Lagrangian upper bound using a fixed and finite set Λ . Instead, one can imagine the ability to learn the optimal value of λ and concentrate the computational effort towards learning the Lagrangian upper bound for this particular λ , which could potentially lead to tighter bounds. A possible approach is to apply subgradient descent on λ , similar to what is done in Hawkins [24].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies value-based control agents for Weakly Coupled MDPs (WCMDPs), which comprise independent subproblems linked by an action constraint. The authors generalize an existing Lagrangian relaxation approach to action-value methods learned by stochastic approximation to obtain Weakly Coupled Q-Learning (WCQL). Essentially, a Lagrange multiplier is used to create an overestimate of $Q^*$ called $Q^{\\lambda*}$; taking the minimum of these two value functions can lead to faster convergence without changing the fixed point. A deep-learning extension of this method relying on a soft constraint added to the DQN loss yields the titular method Weakly Coupled Deep Q-Network (WCDQN). The proposed methods are tested against several well-known baselines (such as DQN and Double DQN) and compare favorably on three WCMDPs: electric vehicle charging (tabular only), multi-product inventory control, and ad matching problems.\n\nSTRENGTHS:\n- Self-contained descriptions of Weakly Coupled MDPs and Lagrangian relaxation, as well as a complete related works discussion, make the paper accessible to people (like me) who do not normally work in this setting. The explanations are clear and well written. \n- Interesting theoretical results in the tabular setting. The convergence results look correct and intuitive to me. I checked the proofs in the appendix.\n- Interesting, realistic experiment domains and strong empirical results.\n\nWEAKNESSES:\n- WCQL relies on discretizing $\\lambda \\geq 0$ into a finite set of values $\\Lambda$ to approximate the upper bound $\\min_{\\lambda} Q^{\\lambda}$. Although this does not affect the ultimate convergence to $Q^*$ in the tabular case, I would imagine that this choice has a significant impact on empirical performance, yet the authors do not discuss this or how to choose the set (that I was able to find at least).\n- The computational requirements of the proposed methods are downplayed. The authors state, “We note that, especially in settings where the limiting factor is the ability to collect enough experience, these bounds come nearly for free in that they only require computing a few additional quantities and importantly, they do not require additional experience.” I think this is a strong claim; compared to Q-Learning, at least $N$ additional Q-functions are needed to form the upper bound $Q^{\\lambda*}$. In the deep RL setting, only one extra network appears to be needed, in order to generalize over these $N$ functions, but that still doubles the overall computation. Thus, I do not think it is fair to say that the method comes for free; however, the strong empirical results do justify the extra expense.\n- Although the tabular theory is sound, the implications of porting it to deep function approximation are unclear and not fully appreciated by the paper in my opinion. The authors say, “the [loss] penalty softly ensures that learned network satisfies the bounds obtained from the Lagrangian relaxation.” However, because the constraint is soft, it does not ensure anything. If the approximated upper bound ever dips below Q*, then the wrong value will be bootstrapped by the network and the learned returns will be suboptimal. Of course, this is inevitable with function approximation (a hard constraint cannot be enforced), but it is worthwhile to clarify in the paper.\n\nQUESTIONS:\n- Why is $\\lambda$ called the penalty vector? Is it not a scalar, i.e., the Lagrange multiplier?\n- How long (in timesteps) were the agents trained in each of the experiments? Since the figure x-axes use episodes, did agents receive unequal training time?\n- How was the penalty coefficient $\\kappa_U=10$ chosen?\n\nLIMITATIONS:\n- Need to choose the set $\\Lambda$\n- Added computational expense\n- Upper bound potentially violated with function approximation and soft constraint", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The authors present a new algorithm, "WCDQN" for solving structured problems know as weakly coupled MDPs. The method dynamically estimates upper bounds via Lagrangian relaxation essentially to improve the value of collected experience across the structured subproblems. The authors present a (re)formulation of the problem via the dual bound of the linking constraints and numerically demonstrate the effectiveness of the method on a set of WCMDP problems.

STRENGTHS:
1. Nicely written, enjoyable reading
2. Relatively simple and clear problem formulation

WEAKNESSES:
1. moderate ablation. the proposed method allows for a decomposition based on the number of "subproblems" but the relative effect to this key variable is not explicitly tested

QUESTIONS:
1. It would seem to be the case that this method should scale significantly better for problems with specific substructure. Why not demonstrate this explicitly with a custom problem / domain to validate the extreme case?
2. [note] I think Fig3 is missing a y-axis?
3. [note] your compilation resulted in non-clickable links,

LIMITATIONS:
NA - nothing in addition to what is present in the actual mathematics.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

comments: SUMMARY:
This paper considers the problem of learning optimal policies for a weakly coupled Markov decision process (WCMDP). The paper extends the classical DQN algorithm to this setting by introducing separate lagrangian relaxations. Experiments are conducted on 3 operational research testbeds.

STRENGTHS:
This paper has a very complete structure, with a detailed problem introduction, an insightful algorithm derivation, a theoretical analysis as well as solid numerical studies. The presentation is particularly clear and easy to follow. The algorithm is intuitive and convincing. You cannot really find any flow from this paper.

WEAKNESSES:
My biggest concern about this work is the audience. As a practitioner in deep RL, this is literally the first time for me to hear the problem of WCMDP. Although I do appreciate that this work teaches me a lot, I don't know whether this paper can attract the attention of the deep RL community, particularly considering the test domains are pretty toy. This paper looks to me more like a work that brings deep RL techniques to the OR community rather than the reverse direction.

QUESTIONS:
N/A

LIMITATIONS:
The significance of the problem for the deep learning community.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces novel Q-learning based approaches (WCQL and WCDQN) to solving weakly-coupled Markov decision processes. The approaches work by running Q-learning for the sub-problems of the WCMDP to construct an upper bound on the optimal Q-values of the overall problem. The WCMDP constraint is incorporated into the reward structure of the task via approximate Lagrangian relaxation. In the function approximation case, the constraint is enforced as a penalty to the loss. A theoretical analysis is performed for the tabular case and experiments are run to demonstrate the performance of both the tabular and DQN-based approaches.\n\nSTRENGTHS:\n- The paper is generally easy to follow and clearly written, and the problem setting is well-motivated. \n\n- The proposed approaches appear to be simple and relatively original. \n\n- If this is indeed the first model-free approach to WCMDPs, it represents a significant advance for this problem setting.\n\nWEAKNESSES:\n- The core weakness of both the theoretical and empirical analyses, in my opinion, is the lack of treatment of constraint violations by WCQL and WCDQN. The theoretical analyses, as far as I can tell, assume a fixed, arbitrary Lagrange multiplier and show convergence of the Q-values to the optimal Q-values for *that choice of Lagrange multiplier*. This, however, says nothing about the resulting policy’s constraint violations. Similarly, in the experiments, there are no results showing the constraint violations of each method. Indeed, the “soft constraint” formulation seems strange for problems like the EV task, for in which in practice assigning too many cars to too few charging ports would seem to be a serious problem. I think in particular it would be helpful to add plots showing constraint violations for each method and whether they find feasible policies. \n\n- Related to the above, there’s no discussion of whether even in an idealized case where $\\lambda$ was a continuous variable the proposed approach converges to a saddle point of the Lagrangian (i.e., actually solves the “true” optimization problem), even if in practice discretization is not a significant detriment to performance. \n\n- As far as I can see, there is no discussion of the limitations of the proposed approaches. \n\n- The experimental evaluation is fairly limited.\n\n- More details on the baselines in the main text would be helpful to improve clarity. I would suggest changing the citations to a numerical format to save space. \n\nMinor: \n- The stochastic gradient update at the end of Section 3.2 would ideally reflect the update of minibatch SGD rather than pure SGD. \n\n- The $\\wedge$ symbol in Eq. 11 seems unnecessary, it would be more clear to just write $\\min(\\cdot, \\cdot)$. \n\n- From a clarity perspective, it would be helpful if the initial description of a WCMDP contained an example to anchor readers’ understanding. \n\n- Figure 3A has no y-axis label. Also, I think it would be more informative to plot the upper bound on average across $(s,a)$ pairs rather than at only one point. \n\n- It would make sense to add distributional RL [1] and Rainbow [2] to the list of extensions of DQN in the introduction or related work sections. The authors should also contrast their setting to constrained MDPs [3].\n\n\nI think this is interesting work, and I am open to increasing my score based on the degree to which the above concerns are addressed. \n\n\n[1] https://arxiv.org/abs/1707.06887\n\n[2] https://arxiv.org/abs/1710.02298\n\n[3] https://www-sop.inria.fr/members/Eitan.Altman/TEMP/h.pdf\n\nQUESTIONS:\n- Does the sub-problem action-value network parameterization mean that all of the subproblem action spaces have to have the same number of actions? \n\n- Is there some intuition as to why double Q-learning performs worse than regular Q-learning? I would think the fact that double Q-learning attempts to reduce value overestimation would have a similar effect to ensuring the overall task Q-value estimate doesn’t exceed the estimated upper bound.\n\nLIMITATIONS:\nThe authors do not appear to discuss any limitations of their method (see above). I do not directly see any negative societal implications of this work, though in practice it would depend on the problem to which it is applied and any guarantees (or lack thereof) regarding constraint violations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}