SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that although we are running several Q-learning instances, they all make use of a common experience tuple  $\tau$  split across subproblems, where subproblem  $i$  receives the portion  $\tau_{i}$ . We remind the reader that each subproblem is dramatically simpler than the full MDP, since it operates on smaller state and action spaces  $(S_{i}$  and  $\mathcal{A}_i)$  instead of  $\mathcal{S}$  and  $\mathcal{A}$ .
We refer to the subproblem Q-learning instances as subagents. Therefore, each subagent is associated with a subproblem  $i$  and a penalty  $\lambda \in \Lambda$  and aims to learn  $Q_{i}^{\lambda}$ .
Learning the Lagrangian bounds. Next, at the level of the "main" agent, we combine the approximations  $Q_{i,n+1}^{\lambda}$  learned by the subagents to form an estimate of the Lagrangian action-value function  $Q^{\lambda}$ , as defined in (5). To do so, we first estimate the quantity  $B(w)$  of Proposition 1. This can be done using a stochastic approximation step with a learning rate  $\eta_n$ , as follows:
$$
\boldsymbol {B} _ {n + 1} (w) = \boldsymbol {B} _ {n} (w) + \eta_ {n} (w) [ \boldsymbol {b} (w) + \gamma \boldsymbol {B} _ {n} \left(w ^ {\prime}\right) - \boldsymbol {B} _ {n} (w) ], \tag {8}
$$
where we recall that  $w$  and  $w'$  come from the experience tuple  $\tau$ , embedded within  $s$  and  $s'$ . Now, using Proposition 1, we approximate  $Q^{\lambda}(s, a)$  using
$$
Q _ {n + 1} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}) = \lambda^ {\intercal} \boldsymbol {B} _ {n + 1} (w) + \sum_ {i = 1} ^ {N} Q _ {i, n + 1} ^ {\lambda} \left(s _ {i}, a _ {i}\right). \tag {9}
$$
Finally, we estimate an upper bound on  $Q^{*}$  by taking the minimum over  $\Lambda$ :
$$
Q _ {n + 1} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}) = \min  _ {\lambda \in \Lambda} Q _ {n + 1} ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}). \tag {10}
$$
Q-learning guided by Lagrangian bounds. We would now like to make use of the learned upper bound  $Q_{n+1}^{\lambda^{*}}(s, a)$  when performing Q-learning on the full problem. Denote the WCQL estimate of  $Q^{*}$  at iteration  $n$  by  $Q_{n}^{\prime}$ . We first make a standard update towards an intermediate value  $Q_{n+1}$  using learning rate  $\alpha_{n}$ :
$$
Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}) = Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) + \alpha_ {n} (\boldsymbol {s}, \boldsymbol {a}) \left[ y _ {n} - Q _ {n} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) \right]. \tag {11}
$$
where  $y_{n} = r(\pmb {s},\pmb {a}) + \gamma \max_{\pmb{a}^{\prime}\in \mathcal{A}(\pmb{s}^{\prime})}Q_{n}^{\prime}(\pmb{s}^{\prime},\pmb{a}^{\prime})$  . To incorporate the bounds that we previously estimated, we then project  $Q_{n + 1}(s,a)$  to satisfy the estimated upper bound:
$$
Q _ {n + 1} ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}) = Q _ {n + 1} ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}) \wedge Q _ {n + 1} (\boldsymbol {s}, \boldsymbol {a}), \tag {12}
$$
where  $a \wedge b = \min \{a, b\}$ . The agent now takes an action in the environment using a behavioral policy, such as the  $\epsilon$ -greedy policy on  $Q_{n+1}'(s, a)$ .
The motivation behind this projection is as follows: since the subproblems are significantly smaller in terms of state and action spaces compared to the main problem, the subagents are expected to quickly converge. As a result, our upper bound estimates will get better, improving the the action-value estimate of the main Q-learning agent through the projection step. In addition, WCQL can enable a sort of "generalization" to unseen states by leveraging the weakly-coupled structure. The following example illustrates this piece of intuition.
Example 1. Suppose a WCMDP has  $N = 3$  subproblems with  $S_{i} = \{1,2,3\}$  and  $\mathcal{A}_i = \{1,2\}$  for each  $i$ , leading to  $3^{3} \cdot 2^{3} = 216$  total state action pairs. For the sake of illustration, suppose that the agent has visited states  $s = (1,1,1)$ ,  $s = (2,2,2)$ , and  $s = (3,3,3)$  and both actions from each of these states. This means that from the perspective of every subproblem  $i$ , the agent has visited all state-action pairs in  $S_{i} \times \mathcal{A}_{i}$ , which is enough information to produce an estimate of  $Q_{i}^{\lambda}(s_{i},a_{i})$  for all  $(s_i,a_i)$  and, interestingly, an estimate of  $Q^{\lambda^{*}}(s,\pmb{a})$  for every  $(s,a)$ , despite having visited only a small fraction (6/216) of the possible state-action pairs. This allows the main Q-learning agent to make use of upper bound information at every state-action pair via the projection step (12). The main intuition is that these upper bound values are likely to be more sensible than a randomly initialized value, and therefore, can aid learning.
The above example is certainly contrived, but hopefully illustrates the benefits of decomposition and subsequent projection. We note that, especially in settings where the limiting factor is the ability to collect enough experience, one can trade-off extra computation to derive these bounds and improve RL performance without the need to collect additional experience. The full pseudo-code of the WCQL algorithm is available in Appendix B.
In this section, we show that WCQL converges to  $Q^{*}$  with probability one. First, we state a standard assumption on learning rates and state visitation.
Assumption 2. We assume the following: (i)  $\sum_{n=0}^{\infty} \alpha_n(\pmb{s},\pmb{a}) = \infty$ ,  $\sum_{n=0}^{\infty} \alpha_n^2(\pmb{s},\pmb{a}) < \infty$  for all  $(s,\pmb{a}) \in S \times \mathcal{A}$ ; (ii) analogous conditions to (i) hold for  $\{\beta_n(s_i,a_i)\}_{n}$  and  $\{\eta_n(w)\}_{n}$ , and (iii) the behavioral policy is such that all state-action pairs  $(s,\pmb{a})$  are visited infinitely often w.p.1.
Theorem 1. Under Assumptions 1 and 2, the following statements hold with probability one.
(i) For each  $i$  and  $\lambda \in \Lambda$ ,  $\lim_{n\to \infty}Q_{i,n}^{\lambda}(s_i,a_i) = Q_i^{\lambda}(s_i,a_i)$  for all  $(s_i,a_i)\in S_i\times \mathcal{A}_i$
(ii) For each  $\lambda \in \Lambda$ ,  $\lim_{n\to \infty}Q_n^\lambda (\pmb {s},\pmb {a})\geq Q^* (\pmb {s},\pmb {a})$  for all  $(s,a)\in S\times \mathcal{A}$ .
(iii)  $\lim_{n\to \infty}Q_n'(s,\pmb {a}) = Q^* (s,\pmb {a})$  for all  $(s,a)\in S\times \mathcal{A}$
Theorem 1 ensures that each subagent's value functions converge to the subproblem optimal value. Furthermore, it shows that asymptotically, the Lagrangian action-value function given by (9) will be an upper bound on the optimal action-value function  $Q^{*}$  of the full problem and that our algorithm will converge to  $Q^{*}$ .
5 Weakly Coupled DQN
In this section, we propose our main algorithm weakly coupled DQN (WCDQN), which integrates the main idea of WCQL into a function approximation setting. WCDQN guides DQN using Lagrangian relaxation bounds, implemented using a constrained optimization approach.
Networks. Analogous to WCQL, WCDQN has a main network  $Q^{\prime}(s,a;\theta)$  that learns the action value of the full problem. In addition to the main network, WCDQN uses a subagent network  $Q_{i}^{\lambda}(s_{i},a_{i};\theta_{U})$  network to learn the subproblem action-value functions  $Q_{i}^{\lambda}$ . As in standard DQN, we also have  $\theta^{-}$  and  $\theta_{U}^{-}$ , which are versions of  $\theta$  and  $\theta_{U}$  frozen from a previous iteration and used for computing target values [43]. The inputs to this network are  $(i,\lambda ,s_i,a_i)$ , meaning that we can use a single network to learn the action-value function for all subproblems and  $\lambda \in \Lambda$  simultaneously. The
Lagrangian upper bound and the best upper bound estimates are:
$$
Q ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}; \theta_ {U} ^ {-}) = \lambda^ {\intercal} \boldsymbol {B} (w) + \sum_ {i = 1} ^ {N} Q _ {i} ^ {\lambda} \left(s _ {i}, a _ {i}; \theta_ {U} ^ {-}\right) \text {a n d} Q ^ {\lambda^ {*}} (\boldsymbol {s}, \boldsymbol {a}; \theta_ {U} ^ {-}) = \min  _ {\lambda \in \Lambda} Q ^ {\lambda} (\boldsymbol {s}, \boldsymbol {a}; \theta_ {U} ^ {-}). \tag {13}
$$
Loss functions. Before diving into the training process, we describe the loss functions used to train each network, as they are instructive toward understanding the main idea behind WCDQN (and how it differs from standard DQN). Consider a behavioral distribution  $\rho$  for state-action pairs and a distribution  $\mu$  over the multipliers  $\Lambda$ .
$$
l _ {U} \left(\theta_ {U}\right) = \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho , \lambda \sim \mu} \left[ \sum_ {i = 1} ^ {N} \left(y _ {i} ^ {\lambda} - Q _ {i} ^ {\lambda} \left(s _ {i}, a _ {i}; \theta_ {U}\right)\right) ^ {2} \right], \tag {14}
$$
where the (ideal) target value is
$$
y _ {i} ^ {\lambda} = r _ {i} \left(s _ {i}, a _ {i}\right) - \lambda a _ {i} + \gamma \mathbb {E} \left[ \max  _ {a _ {i} ^ {\prime} \in \mathcal {A} _ {i}} Q _ {i} ^ {\lambda} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}; \theta_ {U} ^ {-}\right) \right]. \tag {15}
$$
For the main agent, we propose a loss function that adds a soft penalty for violating the upper bound:
$$
l (\theta) = \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho , \lambda \sim \mu} \left[ \left(y - Q ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}; \theta)\right) ^ {2} + \kappa_ {U} \left(Q ^ {\prime} (\boldsymbol {s}, \boldsymbol {a}; \theta) - y _ {U}\right) _ {+} ^ {2} \right], \tag {16}
$$
where  $(\cdot)_{+} = \max (\cdot ,0)$ ,  $\kappa_U$  is a coefficient for the soft penalty, and
$$
y = r (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A} \left(\boldsymbol {s} ^ {\prime}\right)} Q ^ {\prime} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}; \theta^ {-}\right) \right], \tag {17}
$$
$$
y _ {U} = r (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {a} ^ {\prime} \in \mathcal {A}} Q ^ {\lambda^ {*}} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}; \theta_ {U} ^ {-}\right) \right]. \tag {18}
$$
The penalty encourages the network to satisfy the bounds obtained from the Lagrangian relaxation.
Training process. The training process resembles DQN, with a few modifications. At any iteration, we first take an action using an  $\epsilon$ -greedy policy using the main network over the feasible actions, store the obtained transition experience  $\pmb{\tau}$  in the buffer, and update the estimate of  $B(w)$  following (8). Each network is then updated by taking a stochastic gradient descent step on its associated loss function, where the expectations are approximated by sampling minibatches of experience tuples  $\pmb{\tau}$  and  $\lambda$ . The penalty coefficient  $\kappa_{U}$  can either be held constant to a positive value or annealed using a schedule throughout the training. The full details are shown in Algorithm 1 and some further details are given in Appendix C.
In this section, we evaluate our algorithms on three different WCMDPs. First, we evaluate WCQL on an electric vehicle (EV) deadline scheduling problem with multiple charging spots and compare its performance with several other tabular algorithms: Q-learning (QL), Double Q-learning (Double-QL) [22], speedy Q-learning (SQL) [4], bias-corrected Q-learning (BCQL) [40], and Lagrange policy Q-learning (Lagrangian QL) [34]. We then evaluate WCDQN on two problems, multi-product inventory control and online stochastic ad matching, and compare against standard DQN, Double-DQN, and the optimality-tightening DQN (OTDQN) algorithm<sup>3</sup> of He et al. [25] as baselines. Further details on environment and algorithmic parameters are in Appendix D.
EV charging deadline scheduling [63]. In this problem, a decision maker is responsible for charging electric vehicles (EV) at a charging service center that consists of  $N = 3$  charging spots. An EV enters the system when a charging spot is available and announces the amount of electricity it needs to be charged, denoted  $B_{t}$ , along with the time that it will leave the system, denoted  $D_{t}$ . The decision maker also faces exogenous, random Markovian processing costs  $c_{t}$ . At each period, the action is to decide which EVs to charge in accordance with the period's capacity constraint. For each unit of power provided to an EV, the service center receives a reward  $1 - c_{t}$ . However, if the EV leaves the system with an unfulfilled charge, a penalty is assessed. The goal is to maximize the revenue minus penalty costs.
Here we use a tabular representation for  $B(w)$  since our example applications do not necessarily have a large exogenous space  $\mathcal{W}$ . When required, WCDQN can be extended to use function approximation (i.e., neural networks) to learn  $B(w)$ .
3We include OTDQN as a baseline because it also makes use of constrained optimization during training.
Algorithm 1 Weekly Coupled DQN
1: Input: Lagrange multiplier set  $\Lambda$  and a distribution  $\mu$  over  $\Lambda$ , penalty coefficient  $\kappa_U$ , initialized replay buffer  $\mathcal{D}$ , target network update frequency  $C_{\mathrm{target}}$ , initial state distribution  $S_0$ .  
2: Initialize main Q-network  $Q'(\cdot, \cdot; \theta)$  and subagent network  $Q_i^\lambda(\cdot, \cdot; \theta_U)$   
3: Initialize target network weights  $\theta^- = \theta$  and  $\theta_U^- = \theta_U$ .  
4: for  $n = 0, 1, 2, \ldots$  do  
5: Take an  $\epsilon$ -greedy behavioral action  $a_n$  with respect to the main network  $Q'(s_n, a; \theta)$ .  
6: Store the observed transition  $\tau_n$  into the replay buffer  $\mathcal{D}$ .  
7: // Update subagents and combine results to estimate upper bound  
8: Sample a minibatch of transitions  $\tau$  from  $\mathcal{D}$  along with a sample of  $\lambda$  from  $\mu$ .  
9: for  $i = 1, 2, \ldots, N$  do  
10: Compute targets  $y_i^{\lambda}$  using (15) and take an optimization step on subagent loss (14).  
11: end for  
12: Update right-hand-side estimate  $B_{n+1}(w_n)$  according to (8).  
13: Using (13), combine subproblems to obtain Lagrangian upper bound  $Q^{\lambda^*}(s, a; \theta_U^-)$ .  
14: // Main agent update with upper bound penalty loss  
15: Compute  $y$  and  $y_U$  using (17) and (18), then take an optimization step on the main loss (16).  
16: Update  $\theta^- = \theta$  and  $\theta_U^- = \theta_U$  every  $C_{\mathrm{target}}$  steps.  
17: end for
Multi-product inventory control with an exogenous production rate [27]. Consider the problem of resource allocation for a facility that manufactures  $N = 10$  products. Each product  $i$  has an independent exogenous demand given by  $D_{i}$ ,  $i = 1, \dots, N$ . To meet these demands, the products are made to stock. Limited storage  $R_{i}$  is available for each product, and holding a unit of inventory per period incurs a cost  $h_{i}$ . Unmet demand is backordered at a cost  $b_{i}$  if the number of backorders is less than the maximum number of allowable backorders  $M_{i}$ . Otherwise, it is lost with a penalty cost  $l_{i}$ . The DM needs to allocate a resource level  $a_{i} \in \{0, 1, \dots, U\}$  for product  $i$  from a shared finite resource quantity  $U$  in response to changes in the stock level of each product, denoted by  $x_{i} \in X_{i} = \{-M_{i}, -M_{i} + 1, \dots, R_{i}\}$ . A negative stock level corresponds to the number of backorders. Allocating a resource level  $a_{i}$  yields a production rate given by a function  $\rho_{i}(a_{i}, p_{i})$  where  $p_{i}$  is an exogenous Markovian noise that affects the production rate. The goal is to minimize the total cost, which consists of holding, back-ordering, and lost sales costs.
Online stochastic ad matching [18]. We study the problem of matching  $N = 6$  advertisers to arriving impressions. In each period, an impression of type  $e_t$  arrives according to a Markov chain. An action  $a_{t,i} \in \{0,1\}$  assigns impression  $e_t$  to advertiser  $i$ , with a constraint that exactly one advertiser is selected:  $\sum_{i=1}^{N} a_{t,i} = 1$ .Advertiser states represent the number of remaining ads to display and evolves according to  $s_{t+1,i} = s_{t,i} - a_{t,i}$ . The objective is to maximize the discounted sum of expected rewards for all advertisers.
In Figure 2, we show how WCQL's projection method helps it learn a more accurate  $Q$  function more quickly than competing tabular methods. The first panel, Figure 2A, shows an example evolution of WCQL's projected value function  $Q'$ , along with the evolution of the upper bound. We compare this to the evolution of the action-value function in absence of the projection step. In the second panel, Figure 2B, we plot the relative error between the learned value functions of various algorithms compared to the optimal value function. Both plots are from the EV charging example. Detailed descriptions of the results are given in the figure's caption.
The results of our numerical experiments are shown in Figure 3. We see that in both the tabular and the function approximation cases, our algorithms outperformed the baselines, with WCQL and WCDQN achieving the best mean episode total rewards amongst all problems. From Figure 3A, we see that although the difference between WCQL and Lagrangian QL is small towards the end of the training process, there are stark differences earlier on. In particular, the performance curve of WCQL shows significantly lower variance, suggesting more robustness across random seeds. Given that WCQL and Lagrangian QL differ only in the projection step, we can attribute the improved stability to the guidance provided by the Lagrangian bounds. Figure 3B shows that for the multi-product inventory problem, the OTDQN, DQN, and Double DQN baselines show extremely noisy performance compared to WCDQN, whose significantly better and stable performance is likely
(A) WCQL learning process illustration
(B) Relative error comparison
Figure 2: Behavior of WCQL's learning process compared to other tabular methods. In Figure 2A, we show an example of the evolution of quantities associated with WCQL for a randomly selected state-action pair in the EV charging problem: upper bound (blue), WCQL Q-value (orange line with 'x' marks indicating the points projected by the bound), standard Q-learning (green) and the optimal action-value (red). Note that sometimes the orange line appears above the blue line since WCQL's projection step is asynchronous, i.e., the projections are made only if the state is visited by the behavioral policy. Notice at the last marker, the bound moved the Q-value in orange to a "good" value, relatively nearby the optimal value. WCQL's Q-value then evolves on its own and eventually converges. On the other hand, standard QL (green), which follows the same behavior policy as WCQL, is still far from optimal. In Figure 2B, we show the relative error, defined as  $\| V_{n} - V^{*}\|_{2} / \| V^{*}\|_{2}$ , where  $V_{n}$  and  $V^{*}$  are the state value functions derived from  $Q$ -iterate on iteration  $n$  and  $Q^{*}$ , respectively. WCQL exhibits the steepest decline in relative error compared to the other algorithms. Note that since Lagrangian QL acts based on the Q-values of the subproblems, there is no Q-value for this algorithm on which to compute a relative error.
due to the use of faster converging subagents and better use of the collected experience. Similarly, in the online stochastic ad matching problem, WCDQN significantly outperforms all the baselines.
(A) EV charging
Figure 3: Benchmarking results for the WCQL (EV charging) and WCDQN (multi-product inventory control, online ad matching) against baseline methods. The plots show mean total rewards and their  $95\%$  confidence intervals across 5 independent replications.
(B) Multi-product inventory control
(C) Online ad matching
In this study, we propose the WCQL algorithm for learning in weakly coupled MDPs and we show that our algorithm converges to the optimal action-value function. We then propose WCDQN, which extends the idea behind the WCQL algorithm to the function approximation case. Our algorithms are model-free and learn upper bounds on the optimal action-value using a combination of a Lagrangian relaxation and Q-learning. These bounds are then used within a constrained optimization approach to improve performance and make learning more efficient. Our approaches significantly outperforms competing approaches on several benchmark environments.
This research was supported in part by the University of Pittsburgh Center for Research Computing, RRID:SCR_022735, through the resources provided. Specifically, this work used the H2P cluster, which is supported by NSF award number OAC-2117681.
[1] Daniel Adelman and Adam J Mersereau. Relaxations of weakly coupled stochastic dynamic programs. Operations Research, 56(3):712-727, 2008.
[2] Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
[3] Konstantin E Avrachenkov and Vivek S Borkar. Whittle index based Q-learning for restless bandits with average reward. Automatica, 139:110186, 2022.
[4] M. G. Azar, R. Munos, M. Ghavamzadaeh, and H. J. Kappen. Speedy Q-learning. In Advances in Neural Information Processing Systems 24, 2011.
[5] Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023.
[6] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming, volume 5. Athena Scientific Belmont, MA, 1996.
[7] Dimitris Bertsimas and Adam J Mersereau. A learning approach for interactive marketing to a customer segment. Operations Research, 55(6):1120-1135, 2007.
[8] Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786, 2019.
[9] Craig Boutilier and Tyler Lu. Budget allocation using weakly coupled, constrained Markov decision processes. 2016.
[10] Craig Boutilier, Richard Dearden, and Moises Goldszmidt. Stochastic dynamic programming with factored representations. Artificial Intelligence, 121(1-2):49-107, 2000.
[11] David B Brown and Jingwei Zhang. On the strength of relaxations of weakly coupled stochastic dynamic programs. Operations Research, 2022.
[12] David B Brown, James E Smith, and Peng Sun. Information relaxations and duality in stochastic dynamic programs. Operations Research, 58(4-part-1):785-801, 2010.
[13] Hyeong Soo Chang, Pedram Jaefari Fard, Steven I Marcus, and Mark Shayman. Multitime scale Markov decision processes. IEEE Transactions on Automatic Control, 48(6):976-987, 2003.
[14] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674. PMLR, 2019.
[15] Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9):2419-2468, 2021.
[16] Ibrahim El Shar and Daniel Jiang. Lookahead-bounded Q-learning. In International Conference on Machine Learning, pages 8665-8675. PMLR, 2020.
[17] Eyal Even-Dar, Yishay Mansour, and Peter Bartlett. Learning rates for Q-learning. Journal of Machine Learning Research, 5(1), 2003.
[18] Jon Feldman, Aranyak Mehta, Vahab Mirrokni, and Shan Muthukrishnan. Online stochastic matching: Beating 1-1/e. In 2009 50th Annual IEEE Symposium on Foundations of Computer Science, pages 117-126. IEEE, 2009.