SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider manufacturing  $N = 10$  products. The exogenous demand  $D_{t,i}$  for each product  $i \in \{1,2,\dots,10\}$  follows a Poisson distribution with mean value  $\mu_{i}$ . The maximum storage capacity and the maximum number of allowable backorders (after which lost sales costs incur) for product  $i$  are given by  $R_{i}$  and  $M_{i}$ , respectively.
The state for subproblem  $i$  is given by  $(x_{t,i}, p_t)$ , where  $x_{t,i} \in X_i = \{-M_i, -M_i + 1, \dots, R_i\}$  is the inventory level for product  $i$ , and  $p_t$  is an exogenous and Markovian noise with support [0.8, 1]. A negative stock level corresponds to the number of backorders. For subproblem  $i$ , the action  $a_{t,i}$  is the number of resources allocated to the product  $i$ . The maximum number of resources available for all products is  $U = 3$ , so feasible actions must satisfy  $\sum_{i} a_{t,i} \leq 3$ .
Allocating a resource level  $a_{t,i}$  yields a production rate  $\rho_{i}(a_{t,i},p_{t}) = (12p_{t}a_{t,i}) / (5.971 + a_{t,i})$ . The cost function for product  $i$  is  $c_{i}(p_{t},x_{t,i},a_{t,i})$  and represents the sum of the holding, backorders, and lost sales costs. We let  $h_i$ ,  $b_i$ , and  $l_{i}$  denote the per-unit holding, backorder, and lost sale costs, respectively. The cost function  $c_{i}(x_{t,i},p_{t},a_{t,i})$  is given by,
$$
\begin{array}{l} c _ {i} \left(x _ {t, i}, p _ {t}, a _ {t, i}\right) = h _ {i} \left(x _ {t, i} + \rho_ {i} \left(a _ {t, i}, p _ {t}\right)\right) _ {+} + b _ {i} \left(- x _ {t, i} - \rho_ {i} \left(a _ {t, i}, p _ {t}\right)\right) _ {+} \\ + l _ {i} \left(\left(D _ {t, i} - x _ {t, i} - \rho_ {i} \left(a _ {t, i}, p _ {t}\right)\right) _ {+} - M _ {i}\right) _ {+}, \\ \end{array}
$$
where  $(.)_{+} = \max (.,0)$ . We summarize the cost parameters and the mean demand for each product in Table 1. Finally, the transition for the inventory state of subproblem  $i$  is given by
$$
x _ {t + 1, i} = \max  \left(\min  \left(x _ {t, i} + \rho_ {i} \left(a _ {t, i}, p _ {t}\right) - D _ {t, i}, R _ {i}\right), - M _ {i}\right),
$$
where the exogenous noise  $p_t$  evolves according to a transition matrix sampled from a Dirichlet distribution whose parameters are each sampled (once per replication) from a Uniform(1, 5) distribution.
Table 1: Multi-product inventory environment parameters
Product i	1	2	3	4	5	6	7	8	9	10
Storage capacity Ri	20	30	10	15	10	10	25	30	15	10
Maximum backordersMi	5	5	5	5	5	5	5	5	5	5
Mean demand μi	0.3	0.7	0.5	1.0	1.4	0.9	1.1	1.2	0.3	0.6
Holding cost hi	0.1	0.2	0.05	0.3	0.2	0.5	0.3	0.4	0.15	0.12
Backorder cost bi	3.0	1.2	5.15	1.3	1.1	1.1	10.3	1.05	1.	3.1
Lost sales cost li	30.1	3.3	10.05	3.9	3.7	3.6	40.3	4.5	12.55	44.1
D.3 Online stochastic ad matching [18]
In this problem, a platform needs to match  $N = 6$  advertisers to arriving impressions [18]. An impression  $e_t \in E = \{1,2,\dots,5\}$  arrives according to a discrete time Markov chain with transition probabilities given by  $q(e_{t+1} \mid e_t)$ , where each row of the transition matrix  $q$  is sampled from a Dirichlet distribution whose parameters are sampled (once per replication) from Uniform(1,20).
The action  $a_{t,i} \in \{0,1\}$  is whether to assign impression  $e_t$  to advertiser  $i$  or not. The platform can assign an impression to at most one advertiser:  $\sum_{i=1}^{N} a_{i,t} = 1$ .
The state of advertiser  $i$ ,  $x_{i,t}$  gives the number of remaining ads to display and evolves according to  $x_{t+1,i} = x_{t,i} - a_{t,i}$ . The initial state is  $x_0 = (10, 11, 12, 10, 14, 9)$ . The reward obtained from advertiser  $i$  in state  $s_{t,i} = (x_{t,i}, e_t)$  is  $r_i(s_{t,i}, a_{t,i}) = l_{i,e_t} \min(x_{t,i}, a_{t,i})$ , where the parameters  $l_{i,e_t}$  are sampled (once per replication) from Uniform(1, 4).
D.4 Training parameters
Each method was trained for 6,000 episodes for the EV charging problem, 5,000 for the multi-product inventory control problem, and 10,000 episodes for the online stochastic ad matching problem. The episode lengths for the EV charging, online ad stochastic ad matching, and multi-product inventory control problems are 50, 30, and 25, respectively. We performed 5 independent replications.
We use a neural network architecture that consists of two hidden layers, with 64 and 32 hidden units respectively, for all algorithms. A rectified linear unit (ReLU) is used as the activation function for each hidden layer. The Adam optimizer [36] with a learning rate of  $1.0 \times 10^{-4}$  was used. For OTDQN, we use the same parameter settings as in He et al. [25].
For WCDQN, we use a Lagrangian multiplier  $\lambda \in [0,10]$ , with a 0.01 discretization. We also used an experience buffer of size 100,000 and initialized it with 10,000 experience tuples that were obtained using a random policy. For the WCDQN algorithm, we set the penalty coefficient  $\kappa_U$  to 10, after performing a small amount of manual hyperparameter tuning on the set  $\{1,2,4,10\}$ .
D.5 Sensitivity analysis of WCQL with respect to the number of subproblems
We study the performance improvement from WCQL over vanilla Q-learning as the number of subproblems increases for the EV charging problem. We only vary the number of subproblems (from 2 to 5) and keep all other settings as defined in Appendix D.1. The results, given in Table 2, show that the benefits of WCQL become larger as the number of subproblems increases. This provides some additional evidence for the practicality of our approach, especially in regimes where standard methods fail.
Table 2: Cumulative reward and percent improvement of WCQL over QL on the EV-charging problem with a different number of subproblems.
Algorithm	Number of Subproblems
2	3	4	5
QL	5.39	6.7	5.2	3.26
WCQL	5.35	7.14	6.28	4.66
Percent improvement	-0.7%	6.6%	20.8%	42.9%
E Limitations and Future Work
One interesting direction to explore for future work is to address the limitation of learning the Lagrangian upper bound using a fixed and finite set  $\Lambda$ . Instead, one can imagine the ability to learn the optimal value of  $\lambda$  and concentrate the computational effort towards learning the Lagrangian upper bound for this particular  $\lambda$ , which could potentially lead to tighter bounds. A possible approach is to apply subgradient descent on  $\lambda$ , similar to what is done in Hawkins [24].