Reliable Off-Policy Learning for Dosage Combinations
Authors: Jonas Schweisthal
Dennis Frauen
Valentyn Melnychuk
Stefan Feuerriegel
Venue: NeurIPS 2023 poster (2023)
Keywords: off-policy learning
causal inference
reliable machine learning
medicine
dosaging
normalizing flows
TL;DR: We develop a novel method for reliable off-policy learning for dosage combinations

Abstract: Decision-making in personalized medicine such as cancer therapy or critical care must often make choices for dosage combinations, i.e., multiple continuous treatments. Existing work for this task has modeled the effect of multiple treatments independently, while estimating the joint effect has received little attention but comes with non-trivial challenges. In this paper, we propose a novel method for reliable off-policy learning for dosage combinations. Our method proceeds along three steps: (1) We develop a tailored neural network that estimates the individualized dose-response function while accounting for the joint effect of multiple dependent dosages. (2) We estimate the generalized propensity score using conditional normalizing flows in order to detect regions with limited overlap in the shared covariate-treatment space. (3) We present a gradient-based learning algorithm to find the optimal, individualized dosage combinations. Here, we ensure reliable estimation of the policy value by avoiding regions with limited overlap. We finally perform an extensive evaluation of our method to show its effectiveness. To the best of our knowledge, ours is the first work to provide a method for reliable off-policy learning for optimal dosage combinations.

Reliable Off-Policy Learning for Dosage Combinations
Jonas Schweisthal, Dennis Frauen, Valentyn Melnychuk & Stefan Feuerriegel
LMU Munich
Munich Center for Machine Learning
scwweisthal, frauen, melnychuk www.eeaccck,eeeeeeeeeeeeeeeeeeee feuerriegel Halmueeeeeeleue..mmmm.mmm
Abstract
Decision-making in personalized medicine such as cancer therapy or critical care must often make choices for dosage combinations, i.e., multiple continuous treat- ments. Existing work for this task has modeled the effect of multiple treatments independently, while estimating the joint effect has received little attention but comes with non-trivial challenges. for reliable off- off-policy learning for dosage combinations. Our method proceeds along three steps: (1) We develop a tailored neural network that estimates the individualized dose-response function while accounting for the joint effect of mul- tiple dependent dosages. (2) We estimate the generalized propensity score using conditional normalizing flows in order to detect regions with limited overlap in the shared covariate-treatment space. (3) We present a gradient-based learning algorithm to find the optimal, individualized dosage combinations. Here, we reliable estimation of the policy value by avoiding regions with limited overlap. We finally perform an extensive evaluation of method to show its effectiveness. To the best of our knowledge, ours is the first work to provide a method for reliable off- off-policy learning for optimal dosage combinations.
this paper, we propose a novel method
Introduction
In personalized medicine, decision-making frequently involves complex choices for dosage com- binations. For example, in cancer therapy, patients with chemotherapy or immunotherapy receive combination of two three drugs. and, for each, the dosage must be carefully personalized [16. 82). In critical care, medical professionals must simultancously control multiple parameters for mechanical ventilation, including respiratory rate and tidal volume [20].
One way to find optimal dosages (i.e., multiple continuous treatments) is through randomized controlled trials. Yet, such randomized controlled trials are often impractical due to the complexity of dosing problems or even unethical [62]. Instead, there direct need for learning optimal, individualized dosage combinations from observational data (so-called off-policy learning).
Several methods for off-policy learning aim at discrete treatments [1, 3, 14, 23, 37, 40, 74, 76) and are thus not applicable to dosage combinations. In contrast to that, there are only a few methods that aim at off off-policy learning for continuous treatments, that is, dosages [6. 10. 41, 44, 87]. To achieve this, some of these methods utilize the individualized dose-response function, which is typically not available and must be estimated from observational data. However, existing works model the effect of multiple continuous treatments independently. Because of that, existing works ignore the dependence structure among dosage combinations, that is, drug-drup interactions. For example, in chemotherapy. the combination of antineoplastic agents with drugs such as warfarin can have adverse effects on patient health [61]. contrast to that, methods for off-policy learning that account for the joint effect of dosage combinations have received little attention and present the focus of our paper.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Crucially, off policy learning for dosage combinations comes with non non-trivial challenges. (1) Empir- ically, datasets in medical practice are often high-dimensional, yet of limited sample size. Hence, in practice, there are often sparse regions in the Dovariate-treatmen space, which, in turn, may lead to estimated individualized dose-response functions that have regions with high uncertainty. (2) Mathematically, off-policy learning commonly relies upon the overlap assumption [65]. Yet, certain drug combinations are rarely used in medical practice and should be avoided for safety reasons. Both challenges lead to areas with limited observed overlap in the data, which results in unreliable estimates of the policy value. Motivated by this, we thus develop a method for reliable off-policy learning.
Proposed method: In this paper, we propose a novel method for reliable off-policy learning for dosage combinations. Our method proceeds along three steps. (1) We develop tailored neural network that estimates the individualized dose-response function while accounting for the joint effect of multiple dependent dosages. Later, we use the network as a plug-in estimator for the policy value. (2) We estimate the generalized propensity score through conditional normalizing flows in order to detect regions with limited overlap in the shared covariate-treatment space. (3) We find the optimal individualized dosage combinations. To achieve that, we present a gradient-descent-ae aal---aeeeetssseee learning algorithm that solves the underlying constrained optimization problem, where we avoid regions with limited overlap and thus ensure a reliable estimation of the policy value. For the latter step, we use an additional policy neural network to predict the optimal policy for reasons of scalability.
Contributions: (1) We propose a novel method for reliable off-policy learning for dosage com- binations. To the best of our knowledge, ours is the first method that accounts for the challenges inherent to dosage combinations. (2) As part of our method, we develop a novel neural network for modeling the individualized dose-response function while accounting for the joint effect of dosage combinations. Thereby, we extend previous works which are limited to mutually exclusive or independent treatments. (3) We perform extensive experiments based on real-world medical data to show the effectiveness of our method.
2 Related Work
Estimating the individualized dose-response function: There are different methods estimate the outcomes of continuous treatments, that, is the individualized dose-response function. One stream uses the generalized propensity score [26, 28, 29], while others adapt generative adversarial networks [5]. An even other stream builds upon the idea of shared representation learning from binary treatments (e.g.. [33, 53, 68, 69, 84)) and extends it to continuous treatments [57, 67]. However, all of the previous methods model the marginal effects of multiple dosages independently and thus without considering the dependence among when multiple treatments are applied simultaneously. Yet, this is unrealistic in medical practice as there are generally drug-drug interactions. To address this, we later model the joint effect of multiple dependent dosages.
There are some approaches for estimating the individualized effects of multi- or high- high-dimensional treatments, yet not for dosages. For example, some methods target conditional multi-causs treatment effect estimation [55, 58, 59, 73, 78, 86] but are restricted to discrete treatments and can not be directly applied to dosages. Other works use techniques tailored for high-dimensional structured treatments [21, 36, 54]. However, they do not focus on smooth and continuous estimates, or reliable off- off-policy learning, which is unlike our method.
Off-policy learning for dosages: There is extensive work on off-policy learning for discrete treat- ments [1, 3, 4, 14, 17, 23, 37, 40, 49, 50, 72, 74, 85). While, in principle, these methods could be applied to our setting by discretizing the dosages, crucial information about the exact individually prescribed dosage values would be lost. This results in less accurate and less personalized dosage recommendations, However, there are only a few papers that address off-policy learning directly for continuous treatments ("dosages").
For dosages, previous works have derived efficient estimators for policy evaluation and subsequent optimization (e.g., [10, 41]). Other works focus directly on the policy optimization objective to find an optimal dosage [6, 44, 87]. In principle, some of these methods are applicable settings with multiple dosages where they act as naive baselines. However, all of them have crucial shortcomings
'Code is available at ttttps//gitthie
http:///ttuuub.ommsseeeeeeeeeeee com/ http:://ww..suuueeaaaaa/aaaaaaas

---

in that they do not account for multiple dependent dosages and/or that they do not aim at reliable decision-make making under limited overlap. Both of the latter are contributions of our method.
Dealing with limited overlap: Various methods have been proposed to address limited overlap for estimating average treatment effects of binary treatments by making use of the propensity score (e.g.. [7, 9, 24, 30, 38, 66]). In contrast, there exist only a few works for off-policy learning with limited overlap. typically for binary treatments [39, 42, 70].
Some works deal with the violation of the overlap assumption by leveraging uncertainty estimates for binary [31] and continuous treatment effect estimation [32]. However, these works do not tackle the problem of direct policy learning. In particular for dosage combinations, this is clear shortcoming, as determining and evaluating policies becomes computationally expensive and infeasible with growing number of samples and treatment dimensions.
Research gap: Personalized maaiing for optimal dosage combinations must (1) estimate the joint effect of dosage combinations, (2) then perform off- aff-policy learning for such dosage combinations, and (3) be reliable even for limited overlap. However, to the best of our knowledge, there is no existing method for this composite task, and we thus present the first method aimed at reliable off-policy learning for dosage combinations.
3 Problem Formulation
Setting: We consider an observational dataset for patients with i.i.d. observations {( 44. V/ (1))) ==1
sampled from population (X, T, Y). where X E X C IRd are the patients' covariates, T E T C R is the assigned p-dimensional dosage combination (i.e., multiple continuous treatments), and Y E R is the observed outcome. For example, in mechanical ventilation setting, X are patient features and measured biosignals, such as age, sex, respiratory measurements, and cardiac measurements; T are the dosages of the ventilation parameters, such respiratory rate and tidal volume; and Y is patient survival. For simplicity, we also refer to the dosages as treatments.
We adopt the potential outcomes framework [64], where (t) is the potential outcome under treatment T = In medicine, one is interested in learning a policy x : X' + T that maps covariates to suggested dosage combinations. Then, the policy value VT)) E|Y ( (T(X)) is the expected outcome of policy т. Our objective to find the optimal policy ग in the policy class II that maximizes V(T)...e..
x² E arg max V(T). REII
(I)
In other words, we want to find x² that minimizes the regret given by R(**) max (TT) V(="), where denotes the set of all possible policies with C Ω.
We further introduce the following notation, which is relevant to our method later. We denote the conditional potential outcome function by p(l, = E[Y(() X = x]. For continuous treatments, this is also called individualized dose-response function. Further, the generalized propensity score (GPS) is given by ft,,z) x) = ffr fr]x==(t) (t), where fr x= denotes the conditional density of T given X = x with respect to the Lebesgue measure. The GPS is the stochastic policy used for assigning the observed treatments (i.e., logging behavioral policy). In our method, both nuisance functions, xf, and ft, x), are not known and must be later estimated from observational data.
Assumptions: To ensure identifiability of the outcome estimation, the following three standard assumptions are commonly made in causal inference [65]: (1) Consistency: Y = Y Y(T). Consistency requires that the patient's observed outcome is equal to the potential outcome under the observed treatment. (2) Ignorability: (()
T X, VL E T. Ignorability ensures that there are no unobserved confounders. (3) Overlap: ztt,z) E, Va E X, T and for some constant e E [0, oc). Overlap (also known as common support or positivity) is necessary to ensure that the outcomes for all potential treatments can be estimated accurately for all individuals.
We distinguish between weak overlap for e = and strong overlap for > O. Since the estimation of the individualized dose-respons function becomes more reliable with increasing in the finite sample setting, we refer to strong overlap in the following when we mention overlap, unless explicitly stated otherwise.
Challenges due to limited overlap: Limited overlap introduces both empirical and theoretical challenges. (1) Datasets with high dimensions and/or small sample size are especially prone to have
3

---

sparse regions in the covariate treatment space X' X T and thus limited overlap [9]. As a result, the estimated individualized dose-response function will have regions with high uncertainty, which preclude reliable decision-making (2) It is likely that e is small for some regions as some drug combinations are rarely prescribed and should be avoided. To address this, we aim at an off-policy method that is reliable, as introduced in Sec. 4.
4 Reliable Off-Policy Learning for Dosage Combinations
In the following, we propose novel method for reliable off-policy learning for dosage combinations. Our method proceeds along three steps (see Fig. 1): (1) We develop tailored neural network that estimates the individualized dose-response function while accounting for the joint effect of multiple dependent dosages (Sec. 4.1). Later, we use the network plug-in estimator for the policy value. (2) We estimate the generalized propensity score through conditional normalizing flows in order to detect regions with limited overlap in the shared dovariate-treatment space (Sec. 4.2). (3) We find optimal individualized dosage combinations (Sec. 4.3). To achieve that, we present gradient- descent-ascees ceaett-sceett-tooe based learning algorithm that solves the underlying constrained optimization problem (Sec. 4.4), where we avoid regions with limited overlap and thus ensure a reliable estimation of the policy value. For the latter step, we use additional policy neural network to predict the optimal policy for fast inference during deployment.
Target
Method
pMt, x.)
1. Alt,
f(t
3. #(t, 2.)
Extireted
Estimated generalized propensity soore
Rotibll policy lamming
Legend
Connectooss
reuti ia apule
Prediction head
310.1)
CNF
Jib.2)
8 A
A
paramzarra nahwork
Netwoi,
Peproserttrr niteo
MLP
Uniilably prodided stoage haaren
pollcy
estination
Figure 1: Overview of our method along steps (1)-(3).
4.1 Neural network for estimating the joint effect of multiple dependent dosages
In the first step, we aim to estimate the individualized dose-response function p(4, ar), which is later used plug-in estimator for the policy value ((T). However, using naïve neural network with t and r concatenated as inputs can lead to diminishing effect estimates t when æ is [68]. In addition, in setting with multiple continuous treatments, want yield smooth estimates of the dose-response surface to ensure better generalization when interpolating between sparse observations. To address this, we thus develop tailored dosage combination network (DCNet) in the following.
Architecture of DCNet: Our DCNet builds upon the varying coefficient network (VCNet) [56] but extends to learn the joint effect of multiple dependent dosages. DCNet consists of (i) a representation network dxxx) and (ii) prediction head hnyee) (Ø), where h is network with d,, different parameters
(!) (m (4) nd. (1)) T The parameters 17(1) are not fixed but depend on the dosage 4. This is
crucial to ensure that the influence does not diminish a setting with high-dimensional covariates

---

To handle multiple dosages (t())....,t t(*), naive way [2] would be to incorporate separate prediction head for each treatment with corresponding parameters (t())) However, this would be ineffective for our task, captures only the marginal effect of dosage but not the joint effect of the dosage combination. Instead, in order to learn the joint effect, we must not only model the marginal effect of dosage (as in (2]) but also drug-drug interactions. We achieve that through a custom prediction head.
Prediction head in DCNet: In DCNet, we a tailored prediction head, which designed model the join! effect of dosage combinations. For this, we leverage a tensor product basis (see, e.g., [48, 81]) to estimate smooth interaction effects in the dose-response surface. As result, we can directly incorporate all dosages simultancously into the parametrization of the network h by using only one head containing the joint dosage information. This is unlike other networks that use p different independent heads [2] and thus cannot capture the joint effect but only marginal effects.
We define each scalar parameter ny nt(t) of our prediction head
K2
K₂
(t({) =
(t 4h (t
2 (((2)
e ( (t6r))
where Ki...., Kp are the number of basis functions w/ of the p dosages (()) ..... {(P) and BJ... are the trainable coefficients for each element of the tensor product. In order to preserve the continuity of the non- non-linear dose- responese surface, we choose ui to be polynomial spline basis functions. Hence, we yield the parameter vector (t)) = B (t)((), where
B1.1..1
BI.KI..KP
B
E ERd, K.)
(3)
Bd,,1...1
Kin
with matrix rows arranged (3j,11..1, ..... ByKKi... .... Bg.Ki.. .... Kp with
(t) = (2) @ 2)(() @ ..... @ (((P)) R K.)
(4)
and with (5(0) = (wi) (i) 1.. where & the Kronecker product. For training DCNet, we minimize the mean squared error loss L = E.= (f(t;, -2)²
Our prediction head has three key benefits: (1) When estimating (u(,, 2)), the tensor product basis enables us to model non-linear interaction effects among different dosage dimensions while still having a dose-response surface that offers continuity, smoothness, and expressiveness.
(2) Having dose-response surface that is smooth between different dosages is beneficial for our task as it allows for better interpolation between sparse data points. (3) Smoothness is also exploited by our muddient-based learning algorithm later. The reason is that a smooth plug-in estimator ju(t, 2) for the estimated policy value V ((T) allows us have more stable gradients and thus more stable parameter updates during learning.
4.2 Conditional normalizing flows for detecting limited overlap
In the second step, we estimate the GPS ft, x:) in order to detect limited overlap, which later allows us to avoid regions with high uncertainty and thus to yield a reliable policy.
Why is the GPS relevant for reliable preision-making, Due to sparsity in the data otherwise limited overlap, the estimates of the individualized dose-responss function fi(t, x) from our DCNet may be naturally characterized by regions with high uncertainty, that is, regions in the covariate-treatmenn space with low GPS. If we would use the estimated fi(t, x)) in a naïve manner as a plug-in estimator for policy learning. we may yield policy where there can be a large discrepancy between the predicted outcome (t,,, .r) and the true outcome p(t,, x)) and, therefore, large discrepancy between (xx) and (iT). By avoiding regions with a low GPS, we can thus ensure reliable decision-making in safety-critical applications such medicine.
Estimation GPS: The GPS (i.e., the logging policy) is typically not known in real-world settings. Instead, must estimate the GPS from observational data. To this end, we leverage conditional
5

---

normalizing flows (CNFs) [75, 80). CNFs are aully-aarametii generative model built on top of normalizing flows [60, 71] that can model conditional densities p x)) by transforming a simple base density p(z) through an invertible transformation with parameters (1r) that depend on the input a.?
In our method, we use CNFs with parameters 7(x) to estimate the probability density function f(t,z) fx)===()) of I conditioned on the patient covariates X x. For our method, CNFs have several advantages. First, CNFs are universal density approximators [11, 12, 15, 27], making them flexible and able to capture even complex density functions for dosage combinations. Second, CNFs are properly normalized. Third, unlike kernel methods, CNFs are fully parametric, and, once trained, the inference time is constant [52]. This is particularly important later in the third step of our method (Sec. 4.3): For each training step and training sample in the batch, the predicted conditional density has to be evaluated again under the updated policy, and, hence, a fast inference time of density is crucial to achieving a significant speed up when training the policy network.
In our method, we use neural spline flows [15], as they allow for flexible estimation of multimodal densities, and, in combination with masked rugo-regressive networks [13], we can extend them to multiple dimensions. This combination allows for fast and exact evaluation of the multidimensional
densities. We also set the base density, p(2), to standard normal distribution. The CNFs are trained by minimizing the negative log- ligelikelihodd loss f = Em= log f 21),
In the following section, we use the estimated GPS f(t, x²) during policy learning in order to avoid regions with limited overlap and thus ensure reliable decision making..
4.3 Constrained optimization to avoid regions with limited overlap
Reliable decision-making: Our aim is to find the policy ne which maximizes the estimated policy value (xx) while ensuring that (xx) can be estimated reliably. For this, we suggest to constrain our search space in the shared covariate-treatmen
space to regions that are sufficiently supported by data, i.e., where overlap is not violated. Formally, we rewrite our objective from Eq. (1) as
ne arg max (IT) with II' { II I (=(x),, E), Vr x),
(5)
*EII'
where E is reliability threshold controlling the minimum overlap required for our policy. In our setting with finite observational data, this yields the constrained optimization problem
max ( s.t. f (T(2)))); 2 e, Vi E (1....,m).
(6)
Here, we use our DCNet as plug-in dose-response estimator fa(t, 2), and we further use our conditional normalizing flows to restrict the search space to policies with a sufficiently large estimated GPS 2t)
Neural network for reliable policy learning: To learn the optimal reliable policy, we further suggest an efficient procedure based on gradient updates in the following. Specifically, we train an additional policy neural network to predict the optimal policy. Our choice has direct benefits in practice. Why not just optimize over the dosage combinations for each patient individually? In principle, one could also use methods for constrained optimization and leverage the nuisance functions and f to directly optimize over the dosage assignments t for each patient, i.e., teel arg max ( 2i)] s.t. f (fi, sra) 2 e. However, this does not warrant scalability at inference time, f,
as it requires that one solves the optimization problem for each incoming patient. The complexity of this optimization problem scales with both the number of dosages and the number of samples in the data to be evaluated. This makes treatment recommendations costly, which prohibits real-time
Normalizing flows were introduced for expressive variational approximations in variational autoencoders [60. 71J. We provide a background in Appendix A.
Most prior works neglect proper estimation of the GPS by mainly using non- non-flexible parametric models [26] or kernel density estimators [41]. Similarly, the varying coefficient network from [56] uses an additional density head but enforces a discretization. It is thus overly simplified and. therefore, not reliable and not continuous. Both are crucial shortcomings for our task. As a remedy, we propose the of CNFs.

---

cecision-making in critical care. As such, direct methods for optimization are rendered impractical for real-world medical applications. Instead, we suggest to directly learn a policy that predicts the optimal dosage combination for incoming patients.
As the constrained optimization problem in Eq. (6) cannot be natively learned by gradient updates, we must first transform Eq. (6) into an unconstrained Lagrangian problem
Σ a (To())). s.t. (To((t)), x;)) E.V. min
Y2
{ (T(O()))) A (TT(OE)))))) -c]} 2:
min max
Ti
where (T²))) is the policy learner with parameters A, and X are the Lagrange multipliers for sample The Lagrangian min-mat cein-max-objecte max-objective can be solved by adversarial learning using gradient descent- ascent optimization (see, e.g., [47] for a background). Details are presented in the next section where we introduce our learning algorithm.
4.4 Learning algorithm
The learning algorithm for our method proceeds along the following steps (see Algorithm
(1) We use our DCNet to estimate the individualized dose-response function p(tt,, zz) for dosage
combinations.
(2) We estimate the GPS 21;) using the CNFs to detect areas with limited overlap.
(3) We plug the estimated j(t, z) and f(t, 2)) into the Lagrangian objective from Eq. (7). As the policy learner, we use multilayer-perceptron neural network #t(r)) to predict the optimal dosage combination given the patient covariates æ. We use stochastic gradient descent-ascent we perform adversarial learning with alternating parameter updates. First, in every training step, we perform a gradient descent step to update the parameters 0 of the policy network Åि with respect to the policy loss
detail,
1 Σ (#o((s)) - As (x)
L- (0,
(8)
where the reliability threshold and where X we perform gradient ascent step to update the penalty parameters À with respect to L. Here, each A is optimized for each sample in the training set.
} are trainable penalty parameters. Then,
As there is no guarantee for global convergence in the non-convex optimization problem, we perform k ..... K random restarts and select the best evaluated on the validation set by
rel ()) ag 0
with = arg &
Σ
(k) (T (x1),
1 {f
(k) 1
(xa),
2 20) (9)
(8) is the learned policy run & and 1{-} denotes the indicator function. As a result, we (m/²) xz:) 2 E, Vi.
where To
select the policy Ti which maximizes (TT) under the overlap constraint
Note that the performance on the validation set, as X; can only be optimized for observations the training set during the policy learning.
the validation loss from Eq. (9) because L. cannot be directly applied for evaluating
Experiments
We perform extensive experiments using semi-synthetic data from real-world medical settings to evaluate the effectiveness of our method. Semi-synthetic data are commonly used to evaluate causal inference methods as it ensures that the causal ground truth is available [8, 83) and thus allows us to benchmark the performance.
7

---

MIMIC-IV: MIMIC-IV [35] is a fate-of-the-ar dataset with de-identified health records from patients admitted intensive care units. Analogous to [67], we aim to learn optimal configurations of mechanical ventilation to maximize patient survival (Y). Here, treatments (7) involve the config- uration of mechanical ventilation in terms of respiratory rate and tidal volume. When filtering for T, we yield a dataset with n 5476 patients. We select 33 variables patient covariates X (e.g.. age, sex, respiratory measurements, cardiac measurements). Unlike [67] where treatments T are modeled as p mutually exclusive dosages, we model the more realistic setting, where T are dosage combinations with a joint effect. Our choice is well aligned with medical literature according to which the parameters of mechanical ventilation are dependent and must be simultaneously controlled [20]. Further details for MIMIC-IV are in Appendix B.
TCGA: TCGA [79] is a diverse collec- tion of gene expression data from patients with different cancer types. Our aim is to learn the optimal dosage combination for chemotherapy (T'). In chemotherapy, com- bining different drugs is the leading clinical option for treating cancer [82]. Let patient survival denote the outcome (Y). We se- lect the same version of the dataset in [5, 67], containing the measurement of 4,000 genes features X from n 9659 pa- tients. Unlike previous works [5, 67], we again model T as different, simultaneously assigned dosages with a joint effect. This follows medical practice where drug com- binations in cancer therapy must be jointly optimized for each patient profile [16, 82]. Given that TCGA is high- -igh-dimensional, we expect it to be an especially challenging setting for evaluation. Further details for TCGA are in Appendix B.
Algorithm 1: Reliable off off-policy learning for dosage combinations
Input data (X, T. Y). reliability threshold
Output: optimal reliable policy 42
// Step 1: Estimate individualized dose-roeponce
function using DCNet
Estimate (at, via loss Cy
// Step Estinate GPS using conditional nornalizing
flovs
Estimate xt) via loss Ey
// Stop 3: Train policy netvork using our reliable learning algorithn
for l (1,2...., K] do
// K runs with random initialization
#8 +- inintalize randomly
initalize randomly for each epoch do for each batch do
Ly c- 1 En
{A (o:))
-h If (s). -E]}
+ 0 1/ VyL. A A V.C.
end
end
end
// select beat learned policy constrained objective on validation set *g x(3), with 1= mag Eisn A (T (m)):s)) If (x).a))) 2z}
In both datasets, the response Y N(u(T, X), can be interpreted as the log-odds ratio of patient survival, which is to be maximized. Due the fact that we work with actual data from medical prac- tice, certain dosage combinations conditional on the patients' features are more likely to be observed. Even other dosage combinations may not be assigned at all as they can cause harm. For our experi- ments, we introduce a parameter a controlling the dosage bias, which changes the observed policy of dosage assignments from random uniform (a 0) to completely deterministic (a oo). Hence, with increasing a, the areas in the oovariate-treatment space will be increasingly prone to limited overlap. We refer to Appendix B for details. When not stated otherwise, we choose a = 2 and p = 2 as the default setting for our experiments.
To the best of our knowledge, there are no existing methods tailored to off- learning for dosage combinations under limited overlap. Hence, we compare our method with the following baselines, which vary different steps in our method and thus allow us to attribute the source of performance gain. Specifically, we vary dose-response estimation (ā(t, x)) vs. optimization (f). (I) For estimating A(t,x), we use two baselines: (i) simple multi-layer perceptron (MLP) as standard neural alternative our DCNet, and (ii) VCNet [56] as the SOTA method for dose- response estimation for independent treatments with a separate head per treatment dimension. (2) For optimization, we minimize the policy loss- E ju (fo(x;) 2x)) without constraints for dealing with limited overlap (naïve).
Baselines:
For training, we split the data into train val / test sets (64 / 16 / 20%). We first train the nuisance models ju(t, z) and f(t, æ²). We then perform k random restarts to train f x). We select the best run with respect to the policy loss on the factual validation dataset. Thereby, we
Evaluation:
carefully adhere to the characteristies in real-world settings, where one does not have access to the true dose-nesponse function p(t, x:) on the validation set for model selection. In our method, we set
8

---

the reliability threshold E to the 5%-quantil of the estimated GPS f(),z) of the train set. Further details including hyperparameter tuning are in Appendix
For evaluation, we compare the methods on the test set using the regret R(#) = max V We report: (1) the regret for the selected policy, which is the policy with the best policy loss on the validation set out of all k policies; (2) the average regret across all k restarts; and (3) the standard deviation better assess the reliability of the method.
Results: Table com- pares the performance of our method against the base- lines. We make the follow-
Methods
MIMIC-IV
TCGA
Selected
Mean
Std
Selected
Mean
Std
Oracle (f)
observed
1.21
1.06
ing observations: (1) DC- Net performs similarly to MLP for low-dimensional data such as MIMIC-IV, as expected. However, we find substantial improve- ments from DCNet over MLP for high- digeesiinnion data such as TCGA, show- ing the advantage of enfore-
MLP
nalve reliable
0.02
1.44
1.56
1.94
1.88
0.12
MLP
0.03
0.03
0.00
1.78
1.78
0.03
VCNe
naive reliable
2.13
2.49
0.92
3.64
2.47
1.19
VCNet
0.07
0.07
0.00
0.06
0.06
0.00
DCNet DCNNtt
naive
2.81
2.07
1.17
1.11
reliable (ours)
0.04
0.04
0.00
0.03
0.03
0.00
Reported: mean staandard deviation (lower beller)
Table I: Performance against baselines. Regret on test set over A = 5
restarts.
ing smoothness and expressiveness in our prediction head. (2) DCNet outperforms VCNet on both datasets, which demonstrates the importance of modeling the joint effect of dosage combinations. (3) Choosing policy that is reliable greatly benefits overall performance. For example, the baselines n+nvv) even perform worse than the observed policy for TCGA, whereas our method (+reliable) is clearly superior. (4) We also observe that our selection of reliable policies greatly reduces the standard deviation of the observed regrets. This further demonstrates the reliability of our proposed method. Overall, our method works best across both datasets. For example, the regret on the TCGA dataset for the selected policy drops from 2.52 (DCNet-naive) to only 0.03 (eeiale))l which is reduction by 98.8%. In sum, this demonstrates the effectiveness of our method.
Comparison to discretization: There exist several methods for off-policy learning for dis- crete treatments (see Sec. 2), including methods targeted for overlap violations and treat- ment combinations. In principle, we could also discretize the dosages in our setting and ap- ply such methods. However, we would then lose information about the exact dosage values and could not exploit the continuity of the dose-response surface. To demonstrate the bene- fits of our method, we benchmark against an "oracle" discretized baseline where we assume perfect knowledge of the dose-response function and perfect individual treatment assignment. This serves as upper bound for all policy learning methods with discretization. We display the results for different granularities of equally-spacce, grid dis- cretization in Table 2. We observe that, even in the oracle setting, discretization itself leads to higher regret than our proposed method, which confirms the advantages of leveraging continuous dosage informa- tion.
Methods
MIMIC-IV
TCGA
A
Regret
Oraele (1)
discrete (3x3)
3.32
2.94
Oraele (u)
discrete (4x4)
0.96
0.95
Oracle (0)
discrete (5x5)
0.05
0.06
DCNNe
reliable (ours)
0.04
0.03
Reported: best/ mean / standard deviation (lower = better)
We evaluate the robustness of our method across settings with different levels of limited overlap. For this, vary the dosage bias (in Fig. 2) and the number of dosages p (in Fig. 3). Here, we use DCNet for modeling ptt,, x) in both methods, but then vary the policy selection (naïve vs. reliable, as defined above). Our method (DCNet+reliable) shows robust behavior. As desired, it achieves a low regret and a low variation in all settings. In contrast, the naïve method leads to high variation. This must be solely attributed to that our constrained optimization for avoiding regions with limited overlap leads to large performance gains. We also observe another disadvantage of the naive method: it may select a suboptimal policy out of the different runs due to the fact that the estimation of ji(t, x) is erroneous in regions with limited overlap. Again, this underlines the benefits of our method for reliable off-policy learning of dosage combinations in settings with limited overlap. In Appendix C, we demonstrate the robustness of our method in further setups.
Robustness checks:
Table 2: Performance against methods based discretized treatments.

---

Shown: selected policy (line) and the range over runs (area). The naive baseline has a large variability across runs while ours highly robust as expected (i.e., we see no variability).
(a) MIMIC-IV
(b) TCGA
23
Natve
+
Rulibtia (ours)
23
Regett
Regrd
4.5
Natro
Obsetied
Relibblo (ours)
Dosage
Dosage bis
Figure 2: Robustness for dosage bias a.
(a) MIMIC-IV
(b) TCGA
22
Obeenved
15
Obsenved
Natre
Natro Retibble (ouns)
25
Reliable (ound)
3/3
23
20
19
15
Dosage dimensions
Dosage dimensions
Figure 3: Robustness for number of dosages p.
6 Discussion
Limitations: Our method makes an important contribution over existing literature in personalized treatment design for dosage combinations by adjusting for the naturally occurring but non-trivial problems of drug-drug interactions and limited overlap. However, the complexity of real-world data in medical practice can limit the applicability of our method in certain ways. (i) Our method does not account for the ignorability assumption, which can result in biased estimates in the case of unobserved confounders or other missing data. (ii) We apply our method in static treatment setting, whereas, in several clinical applications. time-series data are available, e.g., sequences of varying dosages, multiple treatment cycles, and right-censored data. Thus, future work could extend our method to account for causal effect estimates a time-varying setting (e.g.. [18, 25, 5)).. (iii) Our method relies on the for medical applications reasonable- implicit assumption of smooth dose-response surfaces. In settings with extremely unsmooth or even stepwise dose-response surfaces, other methods may be more suitable for dose-response estimation. (iv) Our method aims reliable policy learning by avoiding areas with limited overlap to minimize potential harm. In other applications such as marketing, one may want to explicitly target unreliable regions acquire new customers.
Our method can have a significant impact in the field of medicine, where reliable dosing recommendations crucial for effective treatment. Our approach addresses the limitations of previous methods in terms of reliability and estimating joint effects of dosage combinations, and, hence, can improve decision support for clinicians. The reliability of dosage recommendations is particularly important, as incorrect dosing can have serious consequences for patients, such as adverse effects treatment failure. By addressing this gap, approach provides valuable tool for decision support in medicine, empowering healthcare professionals to make more personalized and safe treatment recommendations.
Broader impact:
10

---

Acknowledgments
SF acknowledges funding from the Swiss National Science Foundation (SNSF) via Grant 186932.
References
[1] Susan Athey and Stefan Wager. "Policy learning with observational data". Econometrica 89.1 (2021), pp. 133-161.
[2] Alexis Bellot, Anish Dhir, and Giulia Prando. "Generalization bounds and algorithms for esti- mating conditional average treatment effect of dosage". In: arXiv preprint urXiv:2205.1469 (2022).
[3] Andrew Bennett and Nathan Kallus. "Efficient policy learning from surrogate-loss classification reductions". In: JCMI.. 2020.
[4] Andrew Bennett, Nathan Kallus, and Tobias Schnabel. "Deep generalized method of moments for instrumental variable analysis". In: NeurIPS. 2019.
[5] Ioana Bica, James Jordon, and Mihacla van der Schaar. "Estimating the effects of continuous- valued interventions using generative adversarial networks". In: NeurIPS. 2020.
[6] Guanhua Chen. Donglin Zeng, and Michael R. Kosorok. "Personalized Dose Finding Using
Outcome Weighted Learning". In: Journal of the American Statistical Association 111.516
(2016), pp. 1509-1521.
[7] W. G. Cochran and Donald B. Rubin. "Controlling bias in observational studies: A review", In:
Sankhya: The Indian Journal of Statistics, Series (1973), pp. 417-446.
[8] Alicia Curth, Changhee Lee, and Mihaela van der Schaar. "SurvITE: Learning heterogeneous treatment effects from time-to-event data". In: NeurIPS. 2021.
[9] Alexander D' Amour et al. "Overlap in observational studies with nigh-dimensional covariates".
In: Journal of Econometrics 221.2 . (2021), pp. 644-654.
[10] Mert Demirer et al. "Semi-Parameen Efficient Policy Learning with Continuous Actions". In:
NeurIPS. 2019.
[11] Laurent Dinh, David Krueger, and Yoshua Bengio. "NICE: Non-linear independent compo-
nents estimation". In: arXiv preprint
(2014).
[12] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. "Density estimation using Real NVP".
In: International Conference on Learning Reprexentations. 2017.
[13] Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. "Invertible generative
modeling using linear rational splines". In: AISTATS. 2020.
[14] Miroslav Dudík et al. "Doubly Robust Policy Evaluation and Optimization". In: Staristical
Science 29.4 (2014).
[15] Conor Durkan al. "Neural spline flows". In: NewrIPS. 2019.
[16] Funmilola A. Fisusi and Emmanuel O. Akala. "Drug combinations in breast cancer therapy".
In: Pharmacentical Nanotechnology 7.1 (2019), pp. 3-23.
[17] Dylan J. Foster and Vasilis Syrgkanis. "Orthogonal statistical learning". In: arXiv preprint
arviv: TTt.:1901.0933. (2019).
[18] Dennis Frauen et al. "Estimating average causal effects from patient trajectories". In: AAAI
2023.
[19] Scott Fujimoto, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration". In: /CML. 2019.
[20] Giacomo Grasselli et al. "Mechanical ventilation parameters in critically ill COVID-19 patients: scoping review"" In: Critical Care 25.1 (2021), p. 115.
[21] Shonosuke Harada and Hisashi Kashima. "GraphITE: Estimating individual effects of graph-
structured treatments".. In: CIKM. 2021.
[22] Negar Hassanpour and Russell Greiner. "Counterfactual regression with importance sampling weights". In: IJCAI. 2019.
[23] Tobias Hatt, Daniel Tschernutter, and Stefan Feuerriegel. "Generalizing off off-pooicy learning under sample selection bias". In: UAL 2022.
[24] James Heckman, Hidehiko. Ichimura, and Petra. Todd. "Matching As An Econometric Eval- uation Estimator: Evidence from Evaluating Job Training Programme". In: The Review of Economic Studies 64.4 (1997), pp. 605-654.

---

[25] Konstantin Heß et al. "Bayesian neural controlled differential equations for treatment effect estimation". In: arXiv preprint arXiv:2310.1744: (2023).
[26] Keisuke Hirano and Guido W. Imbens. "The propensity score with continuous treatments". In: Applied Bayesian Modeling and Causal Inference from Incomplete- -aata Perspectives. 2004,
[27] Chin-Wei Huang et al. "Neural autoregressive flows". In: Infernational Conference on Machine Learning. 2018.
[28] Kosuke Imai and David A. van Dyk. "Causal inference with general treatment regimes". In: Journal of the American Statistical Association 99.467 (2004), pp. 854-866
[29] G. Imbens. "The role the propensity score in estimating dose-response functions". In: Biometrika 87.3 (2000), pp. 706-710.
[30] Edward L. lonides. "Truncated Importance Sampling". In: Journal of Computational and Graphical Statiticc 17.2 (2008), pp. 295-311.
[31] Andrew Jesson et al. "Identifying causal-effect inference failure with uncertainty-awan mod- els". In: NeurIPS. 2020.
[32] Andrew Jesson al. "Scalable sensitivity and uncertainty analysis for causal-effect estimates of continuous- valued interventions". In: NewrIPS. 2022.
[33] Fredrik D. Johansson, Uri Shalit, and David Sonntag. "Learning representations for counter- factual inference". In: JCML. 2016.
[34] Alistair E. W. Johnson et al. "MIMIC-III, a freely accessible critical care database". In:
Scientific Data 3.1 (2016). p. 160035,
[35]
Alistair E. W. Johnson et al. "MIMIC-IV, a freely accessible electronic health record dataset". In: Scientific Data 10.1 (2023), p. 1.
[36]
Jean Kaddour et al. "Causal effect inference for structured treatments". In: NeurIPS, 2021.
[37]
Nathan Kallus. "Balanced policy evaluation and learning". In: NeurlPS, 2018.
[38]
Nathan Kallus. "Generalized optimal matching methods for causal inference". In: Journal of Machine Learning Research (2020).
[39] Nathan Kallus. "More efficient policy learning via optimal retargeting". In: Journal of the American Statistical Association 116.534 12021), pp. 646-658.
[40] Nathan Kallus and Miruna Oprescu. "Robust and agnostic learning of conditional distributional treatment effects". In: AISTATS. 2023.
[41] Nathan Kallus and Angela Zhou. "Policy evaluation and optimization with continuous treat- ments". In: AISTATS. 2018.
[42] Nathan Kallus and Angela Zhou. "Residual unfairness in fair machine learning from prejudiced
data". In: /CML. 2018
[43] Diederik P. Kingma and Jimmy Ba. "Adam: A method for stochastic optimization". In: ICLR. 2015.
[44] Akshay Krishnamurthy, John Langford, and Slivkins, Aleksandrs, Zhang, Chicheng, "Con- textual bandits with continuous actions: smoothing, zooming, and Aaapting". In: Journal of Machine Learning Research 21.137 (2020), pp. 1-45.
[45]
Aviral Kumar et al. "Conservative Q-learning for offline reinforcement learning". In: NeurIPS. 2020.
[46] Aviral Kumar et al. "Stabilizing off off-policy Q-learning via bootstrapping error reduction". In: NeurIPS. 2019.
[47] Tianyi Lin, Chi Jin, and Michael I. Jordan. "On gradient descent ascent for noncomvex-concave minimax problems". In: ICML. 2020.
[48] Xihong Lin and Daowen Zhang. "Inference in Generalized Additive Mixed Models by Us- ing Smoothing Splines". In: Journal of the Royal Statistical Society. Series B (Statistical Methodology) 61.2 (1999), pp. 381-400.
[49] Haixu Ma, Donglin Zeng, and Yufeng Liu. "Learning individualized treatment rules with many treatments: A supervised clustering approach using adaptive fusion".. In: NeurIPS (2022).
[50] Haixu Ma, Donglin Zeng, and Yufeng Liu. "Learning optimal group-structured individualized treatment rules with many treatments". In: Journal of Machine Learning Research 24.102 (2023), pp- 1-48.
[51] Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. "Causal transformer for estimat- ing counterfactual outcomes". In: ICML. 2022.
12

---

[52] Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. "Normalizing flows for interven- tional density estimation". In: JCML. 2023.
[53] Milan Kuzmanovic, Tobias Hatt, and Stefan Feuerriegel. "Estimating conditional average treatment effects with missing treatment information". In: AISTATS, 2023.
[54] Razieh Nabi, Todd McNutt, and Ilya Shpitser. "Semiparametric causal cufficient dimension reduction of multidimensional treatments". In: UAI. 2022.
[55] Preetam Nandy, Marloes H. Maathuis, and Thomas S. Richardson. "Estimating the effect of joint interventions from observational data in sparse high dish-iimensional settings". In: The Annals of Statistics 45.2 (2017), Pp. 647-674.
[56] Lizhen Nic et al. "VCNet and functional targeted regularization for learning causal effects of continuous treatments". In: ICLR. 2021.
[57] Xinkun Nie and Stefan Wager. "Quasi-oracle estimation of heterogeneous treatment effects".
In: Biometrika 108.2 (2021), pp. 299-319.
[58] Sonali Parbhoo, Stefan Bauer, and Patrick Schwab. "NCoRE: Neural Counterfactual Repre-
sentation Learning for Combinations of Treatments". In: arXiv preprint (2021).
[59] Zhaozhi Qian, Alicia Curth, and Mihaela van der Schaar. "Estimating multi-cause treatment
effects via single cause perturbation". In: NeurIPS, 2021.
[60] Danilo Rezende and Shakir Mohamed. "Variational inference with normalizing flows". In:
ICML. 2015.
[61] Rachel P. Riechelmann et "Potential drug interactions and duplicate prescriptions among cancer patients". In: Journal of the National Cancer Institute 99.8 (2007), pp. 592-600.
[62] James M. Robins, Miguel A. Hernán, and Babette Brumback. "Marginal structural models and causal inference in epidemiology". In: Epidemiology 11.5 (2000), pp. 550-560.
[63] Paul R Rosenbaum. "Model-based direct adjustment". In: Journal of the American statistical
Association 82.398 (1987), pp. 387-394.
[64] Donald B. Rubin. "Bayesian inference for causal effects: The role of randomization", In: Annals of Statistics 6.1 (1978), pp. 34-58.
[65] Donald B. Rubin. "Estimating causal effects of treatments in randomized and nonrandomized studies". In: Journal of Educational Psychology 66.5 (1974), pp. 688-701.
[66] Donald B. Rubin. "On the limitations of comparative effectiveness research". In: Statistics in Medicine 29.19 (2010), pp. 1991-1995.
[67] Patrick Schwab et al. "Learning counterfactual representations for estimating individual dose- response curves".. In: AAAI. 2020, p. 5.
[68] Uri Shalit, Fredrik D. Johansson, and David Sontag. "Estimating individual treatment effect: Generalization bounds and algorithms". In: ICML. 2017.
[69] Claudia Shi, David M. Blei, and Victor Veitch. "Adapting neural networks for the estimation of treatment effects". In: NeurIPS. 2019.
[70] Adith Swaminathan and Thorsten Joachims. "Counterfactual risk minimization: learning from
logged bandit feedback",. In: ICML. 2015.
[71] Esteban G. Tabak and Eric Vanden-Eijnen Eijnden. "Density estimation by dual ascent of the log-
likelihood".. In: Communications Mathematical Sciences 8.1 (2010), pp. 217-233.
[72] Shengpu Tang et al. "Leveraging factored action spaces for efficient offline reinforcement learning in healthcare". In: NeurIPS (2022).
[73] Akira Tanimoto al. "Regret Minimization for Causal Inference Large Treatment Space".
In: AISTATS. 2021.
[74] Phillip S. Thomas and Emma Brunskill. "Data-efficient off-Policy policy evaluation for rein- forcement Learning". In: ICML. 2016.
[75] Brian L. Trippe and Richard E. Turner. "Conditional density estimation with Bayesian normal- ising flows". In: arXiv preprint arXiv: 1802.04902 (2018).
[76] Daniel Tschernutter, Tobias Hatt, and Stefan Feuerriegel. "Interpretable off-policy learning via hyperbox search". In: ICMI.. 2022.
[77] Xin Wang et al. "Generalization bounds for estimating causal effects of continuous treatments".
In: NeurlPS. 2022.
[78] Yixin Wang and David M. Blei, "The blessings of multiple causes", In: Journal of the American Statistical Association 14..528 (2019), pp. 1574-1596.
13

---

[79] John N. Weinstein et al. "The Cancer Genome Atlas Pan-Cancer analysis project". In: Nature Genetics 45.10 (2013), pp. 1113-1 1220.
[80] Christina Winkler et al. "Learning likelihoods with conditional normalizing flows". In: arXiv preprint arXiv: 00240004222220 (2019).
[81] Simon N. Wood. "Low-rank scale-invarian tensor product smooths for generalized additive mixed models". In: Biometrics 62.4 (2006), pp. 1025-1036.
[82] Debra Wu et al. "Design principles of drug combinations for chemotherapy". In: Journal of Controlled Release 323 (2020), pp. 36-46.
[83] Liyuan Xu et al. "Learning deep features in instrumental variable regression". In: ICLR. 2021.
[84] Yao Zhang, Alexis Bellot, and Mihaela van der Schaar. "Learning overlapping representations for the estimation of individualized treatment effects". In: A/STATS. 2020.
[85] Zhengyuan Zhou, Susan Athey, and Stefan Wager. "Offline Multi-Action Policy Learning: Generalization and Optimization". In: Operations Research 71.1 (2023), pp. 148-183,
[86]
Hao Zou al. "Counterfactual prediction for bundle treatment". In: NeurIPS. Vol. 33. 2020.
[87]
Hao Zou al. "Counterfactual prediction for outeome-oriented treatments". In: JCML. 2022.
14

---

A Additional Background
Estimating the dose-response function under dosage bias: Estimating the dose-response function becomes challenging with increasing dosage bias. Although the unbiasedness of counterfactual estimation can be achieved through the minimization of the prediction loss on the factual data, the inherent scarcity of data in continuous treatment settings leads to increased variance in the estimation process (see, e.g., [2]), especially in regions with limited overlap. Consequently, the elevated variance hinders accurate generalization over the whole covariate- treatment space and contributes to larger generalization error, which, in turn, impedes the reliable estimation of treatment effects. Effectiivly addressing this issue is crucial to improve the robustness and validity of counterfactual predictions.
covariate-treatment
In Sec. 2, we introduced various existing methods for estimating the dose-response function, e-g.. [5, 26, 28, 29, 57, 67]. Besides a custom neural architecture, some methods use additional techniques to reduce the variance of counterfactual predictions for discrete or continuous treatments. Examples include reweighting [63, 77], importance sampling [22], targeted regularization [56], or balancing representations [2. 33, 67]. In our DCNet, we build on top of the neural architectures of [57, 67] and then develop method that is capable of modeling the joint effect of dosage combinations.
In theory, we could also adapt additional techniques for variance reduction of the dose-response prediction to our setting of dosage combinations and leverage them in combination with DCNet. However, this may not heneficial for our goal of reliable policy learning for dosage combinations under limited overlap, which is different from the goal of accurate dose-response estimation over the whole covariate-treatmen prediction in scarce regions (i.e., with limited overlap), which, consequence, shifts the focus away from regions with sufficient overlap. As we constrain the policy to regions with sufficient overlap anyway by using our reliable policy optimization, we aim for adequate dose-response estimation in these regions, while regions with limited overlap can be neglected. Hence, in our method for reliable policy learning, techniques for tackling dosage bias during dose-response estimation can be considered more counterproductive than useful. We demonstrate this empirically in Appendix D.
space. These techniques aim to reduce the variance of the dose-response
Normalizing flows: Normalizing flows were introduced for expressive variational approximations in variational autoencoders [60, 71] and are based on the principle of transforming a simple base distribution, such Gaussian, through series of invertible transformations to approximate the target data distribution. Let us denote the base distribution as pa (z), where z is a latent variable, and the target data distribution as (x)) where represents the observed data. The goal is to find mapping f such that x = fz)).
The core idea behind normalizing flows is to model the transformation f as a composition of invertible functions f fK fk-1 o .... f1. where K is the number of transformations. The last transformation f& is typically modeled as affine transformation, and others are invertible nonlinear transformations. The inverse transformation can be easily computed since each fa is invertible.
The change of variables formula allows us to compute the probability density of a in terms of z. If assume the base distribution pa (=) is known and has a simple form (e.g., Gaussian), we can compute the density of æ via
@s 0z
Pa =p()= det
(10)
where det. 0/ is the determinant of the Jacobian matrix of the transformation with respect +-)
to 2. This determinant captures the stretching and squishing of the latent space induced by the transformation, thus allowing us to adjust the density of 2 match the density of .r.
To perform inference, such as density estimation or sampling, we need to compute the log-likelihood of the observed data x. Given a dataset D (.i,,.., J. the log-likelihood can be computed as the sum of the log-densities of each data point
n
log (D) = log (x:))(x)).
Σ
(11)
Conditional normalizing flows: In our method, we leverage conditional normalizing flows (CNFs) [75, 80] to estimate the generalized propensity score (GPS). CNFs model conditional densities
15

---

P(y z) by transforming simple base density p(2) through an invertible transformation with parameters (x) that depend on the imput
In our method, we use neural spline flows [15] in combination with masked auto-regressive networks [13]. Here, masked auto-regressive networks used to model the conditional distribution of each variable in the data given the conditioning variable. The network takes as input the conditioning variable and generates each variable in the data sequentially, one a time, while conditioning on the previously generated variables. This autoregressive property allows for efficient computation and generation of samples from the conditional distribution. For the motivation and advantages of our modeling choice, we refer to Sec. 4.2.
Offline reinforcement learning: A related literature stream covers off-policy evaluation (OPE) in an offline reinforcement learning (ORL) setting. Some works leverage value functions and/or logging policies achieve reliable policy learning (e.g., [19, 45, 46]). The main differences that ORL. assumes Markov decision process and sequential decision- decision-making. In contrast, we focus on non-sequential, off off-po learning from observational data. Also, unlike ORL or standard OPE, we leverage the causal structure of treatment assignment and the dose-response function to return causal estimands. This allows not only learn decisions but also causal estimates for potential outcomes. This is crucial for medical practice, where physicians would like to reason over different treatments rather than blindly following fully automated system.
setting
16

---

B Data Description
MIMIC-IV: MIMIC-IV is the latest version of the publicly available dataset of The Medical Information Mart for Intensive Care [35] and contains de- de-identified health records of patients admitted intensive care units (ICUs). The dataset includes comprehensive clinical data, such as vital signs, laboratory measurements, medications, diagnoses, procedures, and more. In line with the approach in [67], our primary objective is to acquire knowledge about the optimal configurations ventilation that can effectively maximize patient survival (Y). Note that some works (e.g., [5, 67]) used the previous version of the MIMIC dataset (MIMIC-III [34]), whereas we use the most current one to offer realistic, comparable, and meaningful findings.
mechanical
As treatments (1), we select the specific settings (dosages) utilized for mechanical ventilation, encompassing parameters such as respiratory rate and tidal volume. By filtering the data for the last available measurements per patient before ventilation, we are able to compile a dataset consisting of 5476 patients. We carefully selected 33 patient covariates (X)) that encompass factors of medical relevance such as age, sex, and various respiratory and cardiac measurements. For detailed identifiers of the selected covariates, we refer to our code.
While [67] adopted framework in which treatments T were modeled dosages, we opted for a more realistic approach, considering that the configuration of mechanical ventilation involves dosage combinations with a joint effect. This decision aligns closely with existing medical literature, which emphasizes the interdependence of mechanical ventilation parameters and underscores the importance of simultancous control [20].
of p mutually exclusive
TCGA:
The Cancer Genome Atlas (TCGA) dataset [79] represents a comprehensive and diverse collection of gene expression data derived from patients diagnosed with various types of cancer. Our research objective revolves around acquiring insights into the optimal combination of dosages for chemotherapy (7). In the domain of cancer treatment, the concurrent administration of different drugs has emerged as a prominent clinical approach [82]. In this context, patient survival serves as the designated outcome variable (Y).
We utilize the dataset as employed in previous works [5, 67]. Specifically, we use the version from [5], which includes gene expression measurements of the 4,000 genes with the highest variability, serving as the features (X), obtained from a cohort of n = 9659 patients. Diverging from the approach of the aforementioned works, once again model T as combination of different dosages that simultaneously assigned, taking into account their joint effect. This modeling choice closely aligns with medical practice, where the optimization of drug combinations for cancer therapy necessitates holistic consideration of each patient's 's unique profile [16, 82].
We summarize the characteristics of the data in Table 3.
MIMIC
TCGA
Variable
Dim.
Meaning
Dim.
Mcaning
X
33
patient covariates (age, sex, respira- tory and cardiac measurements)
4000
gene expression measurements
T
2
respiratory rate, tidal volume
2
chemotherapy drugs (c.g.. doxoru- bicin, cyelophosphamide)
chance patient survival
chance of patient survival
Table 3: Data characteristics of our standard settings.
Semi-synthetic data simulation: Our data simulation process is inspired by [5]. We use the observed features x from the real-world datasets. Then, for each treatment dosage 1,..., ..... we
randomly draw two parameter vectors u(()) (i) from normal distribution N 1) and normalize them via of = v /||of11 and uy
(f) = 1/2 /1lug) (5) respectively. We define n7(3) -000"
and
2000
'Available at: https://mimis mit.i.....i.eeee....
5 Available at: https://wur cancer gov/tcga
Availlble at: https://gitt com/Looaaabica/SCIGGN
17

---

[o (20 + 20 20exp(-n(1)) + 0.2. combination / via
We then sample the j-th dosage of the assigned dosage
( Beta(a, ) where á = a = (1) ā + 2,
-0)==
(12)
with dosage bias a 2, not specified otherwise. Hence, 10) E [0, 1] can be interpreted as percentage normalized version of the modeled dosages.
We define the dose-response function
p(t, 2)) -2
2
Σ
(1+exp(-n)))) + 0.55 cos (0((1()-807) 00.01 ((0) - -
P
P -0.1x II
(10)-801)
(13)
where y is an interaction parameter indicating the strength of the joint effect of the different dosages. Finally, sample the observed outcome from y ~ 2)),, 0.5). In data simulation, we ensure that 8 ( j=1 the optimal dosage combination and that additionally the mode
of the joint conditional dosage distribution of with the generalized propensity score as density given by ftt, x²) = 11; Beta(a, (()))
Taken together, we adapt to a realistic setting medicine where the dosage assignment is already informed to certain degree but still can be further optimized. With increasing a, the probability mass around the optimal dosage combinations increases, while, areas further away from the optimum, overlap becomes more limited, and, hence, the estimation uncertainty of the dose-response function increases. We can interpolate between random uniform distribution for a and a completely deterministic dosage assignment for 30.
18

---

C Additional Results
Robustness checks: We now repeat the robustness checks from Sec. 5 also for the MLP baselines. The results are shown in Fig. 4 and Fig. 5. We observe that our method (DCNet+reliable) performs constantly well with respect to low regret and low (to no) variability. This holds true across the different settings. Also, we see that the baseline MLP clearly benefits from our reliable optimization during policy learning. Hence, the constrained optimization that we formulated in Steps 2 and 3 of our method are also a source of gain for existing baselines.
(a) MIMIC-IV
(b) TCGA
400
35
NLP
30
DCHel
DC76) sobethee
Reggrtt
2.0
15
10
85
0.0
Dasage bios
Dasage bize
Figure 4: Robustness for dosage bias a. Shown: selected policy (line) and the range over 5 runs (area). The naive baseline has large variability across runs while ours is highly robust as expected (i.e., we see no variability).
(a) MIMIC-IV
(b) TCGA
Cloeveed
Grservss
M&P novo
35
M.P nave
MUST rekache
Mi rolalee
DCNd nusee
30
DONK -rar
DChol na lubre
DCNE raaaa
25
25
Regret
20
20
15
1.5
1.0
06
0.
0.0
Dosago dimensions
Desage dimensians
Figure 5: Robustness for number of dosages p. Shown: selected policy (line) and the range over 5 runs (area). The naive baseline has a large variability across runs while ours highly robust as expected (i.c., we see variability).
High-dimensional dosage setting: We perform additional experiments to examine how our method scales to treatments of much higher dimensions. To improve the scalability of our method, we apply multiple prediction heads, each taking 3 treatment dimensions as input. Thereby, we fix the maximum number of jointly interacting dimensions to 3. Our motivation is that applying our method without adjustments to high treatment dimensions would lead to high computational complexity of our prediction head because we would need to train weights for all possible interactions in the tensor product bases. However, it is often reasonable for real-world medical settings that the effective number of dimensions with joint effects is usually lower-dimensions tailored approach. We report the results in Fig. 6. We observe that, unlike the other basclines, our method (DCNet+reliabll) stays robust with low regret and low variation even for high dimensions.
which is thus captured by our
Performance across multiple re-runs: To better assess the robustness of our method, we re-run our experiments for multiple train/val/test splits. Here, we also retrain the nuisance functions j and f for each split. We display the aggregated results for (i) data generated by our standard semi-synthetic
19

---

(a) MIMIC-IV
(b) TCGA
MLP nalve
MLP naive
MUP relibbee
MLP resable
DONet
Bagest
retable
DCNet retable
Dosage dinersioro
Dessge
Figure 6: Robustness for high-dimensional number of dosages p. Shown: selected policy (line) and the range over runs (area). Lower regret better.
data simulation (Appendix B) in Table 4, and (ii) more complex setting with a multimodal GPS in Table For (ii), we replace the dosage combination assignment from Eq. (12) with a multimodal mixture density, i.e., { w() Beta(á, (0)) + (1 wll) Beta(a, u v.))),
(14)
with
w (ernoulli(O.5). ã ā=a+l, g á- ((3)
ā 2, 2(
á + 2.
(0)
Methods
MIMIC-IV
TCGA
selected
mean
selected
mean
sld
Oracle (N)
observed
1.089 + 0.067
0.952 + 0.061
MLP
naive
1.553 + 0.990
1,391 + 0.663
0.797 + 0.430
0.832 + 1.067
1,063+ 1.176
0,303 + 0.224
MLP
reliable
0,748 = 1.516
0.849 1.539
0.223 = 0.375
1.080 + 1.542
1.089 + 1.542
0.022 0.028
VCNe
2.715
2.322
3.169 0.588
2.153
0.242
VCNet
reliable
0.033 + 0.027
0,110 + 0,183
0,171 + 0.377
0.044 + 0.017
0.114 + 0.145
0,159 + 0.354
DCNET naive
2.512 + 0.787
0.9744 0.700
1.134 0.336
2.048 + 0.775
1.682 0.478
0.776 0.263
DCNET reliable (ours)
0.025 0.021
0.024 0.020
0.001 0.001
0.025 + 0.010
0,263 0.540
0.337 0.750
Reported:
selected/ mean standard deviation (lower better)
Table 4: Regret of 5 re-runs (mean + std) with different train/val/test splits with each k restarts for policy learning.
Methods
MIMIC-IV
TOGA
1
selected
meam
std
selected
mean
std
(j))
observed
1.294 0.156
1.624 0.071
MLP
naive
2.139 1.539
1.860=005500
0.993 0.447
1,745 + 1.270
1.444 + 1.432
0.640 + 0,166
MLP
reliable
1.069 = 1,518
0.544 1: 0,509
0.786 : 0.656
0.465 = 0.902
0.720 = 1.488
0.157 :: 0.328
VCNet
naive
1.885 + 0.258
1.627 0.149
0.443 = 0.152
L.728 + 0.306
1.659 + 0.176
0.368 0.130
VCNet
reliable
0.126 + 0.065
0.192 : 0.140
0.1472 0.193
0.029 +: 0.027
0.119 1 0.117
0,201 0.279
DCNET
naive
0.713 + 0
0,389 + 0,256
0.515 + 0.347
0.260 + 0.409
0.141 + 0.125
0,209 + 0.194
DCNET
reliable (ours)
0.077 0.059
0.065
0.061 0.058
0.014 0.010
0.028 0.026
0.031 0.060
Reported: selected mean standard deviation (lower better)
Table Regret of 5 re-runs (mean std) with different train/val/test splits with each restarts for policy learning in an additional setting simulated with a multimodal GPS.
We can interpret the results as follows:
(1) The column "selected" displays the final performance of our method when using the finally recommended policy. In this column, the mewn + sld can considered as the expected regret with Monte-Carlo-CC confidence intervals (including retraining the nuisance functions) and should primarily be used for evaluating the performance and for comparison with different policy learning methods. We observe that our method significantly outperforms the basclines.
20

---

(2) The column "mean" represents the means within the l restarts of the policy learning (i.e., Step our method). As such, values can indicate how a randomly selected policy out of the restarts would perform compared to the "selected one". For example, we observe that, for the "naīve" baselines using& na've MSE loss for selection, suboptimal policies are oftentimes selected, whereas our method shows robust behavior.
(3) The column "std" shows the standard deviation within the k restarts of the policy learning (i.e., Step 3 in our method). As such, the values indicate how often similar performing policies are learned. thus refers to the robustness within the k runs. As a consequence, lower values imply that (i) probably fewer restarts k are needed to find the targeted optimum, and (ii) fewer local optima are learned, which reduces the risk of selecting a suboptimal policy.
Hence, we suggest using the "selected" column (mean + sld) for evaluating the final performance of our method, and the other two columns ("mean" and "std"" to show the inner robustness of our method compared to ablation baselines.
Overall, we demonstrate that the performance of our framework is robust across the tested settings and has low regret and low variation across different re-reruns and different restarts, including the more complex setup.
Sensitivity analysis of the reliability threshold è: E to the 5%-quantile of the estimated GPS =t,,x) of the train set. This is due the reason that the GPS is a probability density function over p-dimensional continuous space and, hence, is hard to interpret intuitively. Unlike discrete probabilities, which are bounded between and and can be interpreted intuitively by humans, the GPS has upper bound. To this end, we suggest heuristic to choose E dependent on x77--uantiles of the estimated GPS based on the train dataset. As result, we aim to learn optimal dosages, which have at least the same estimated reliability 2% of the observed data.
our experiments. we set the reliability threshold
In our heuristic, the selection of the quantile x is still subjective and cannot be optimized during hyperparameter tuning, as we assume to have no access to the ground truth yut,, ..). Nevertheless, we can leverage our semi-synthetic data to perform sensitivity analysis over for E and thus corroborate our choice. For this, we evaluate the final performance of our method on the test data for set to different quantiles of f over lt = 5 runs. The results are displayed in Fig. 7.
We observe that, depending on the dataset, different quantiles for selecting E can lead to differences in variability. However, we find that lower values tend to lead to less variability. Also, we find that, even when there is higher variability within the & runs, for every setting, run with low regret is selected. We attribute this our validation loss from Eq. 9. Together, the results are in line with our expectations and add to the robustness of our methods.
(a) MIMIC-IV
(b) TCGA
24
1.5
Regrat
Regrat
1.0
1.0
00
02
0.4
0.8
38
0.0
02
0.4
08
08
Quanalc opation
Quantle opotion
Figure 7: Sensitivity analysis of the reliability threshold for different quantiles of f of the train set. Shown: selected policy (line) and the range over 5 runs (area).
21

---

D Insights
Individualized dose-response estimates: In our experiments in Sec. 5 and in Appendix C, we showed a comparison to naive baselines. Therein, in particular, we mainly demonstrated that the reliable optimization step in our method an important source of performance gain. We now give additional intuition as to why also our DCNet is source of performance gain. In the following, we first demonstrate that our DCNet is able to estimate the dose-response function more accurately than the baseline in regions with sufficient overlap. We then discuss why this brings large gains for our reliable optimization. Thereby, we offer an explanation why both DCNet and reliable policy optimization are highly effective when used in combination.
In Table 6, we display the performance of the dose-response estimators. In detail, we report: (i) the mean squared елгоr ("Biased MSE"), calculated on a holdout test dataset which follows the same biased dosage assignment as the train data; (ii) the mean ("MISE", see e.g., [67]) of the integrated squared error calculated on areas with sufficient overlap, i.e. Σ Ja.jj- (u(t, 22)
a()d dt| f(t,,)) > ε.; and (iii) the standard deviation of the integrated squared error on areas with sufficient overlap ("SD-ISE"). Intuitively, the MISE and SD-ISE are the expected value and variation of the prediction error of the dose- reseores estimator over the treatmmet space with sufficient overlap. Hence, low value implies that all suitable dosage combinations with sufficient overlap are estimated properly and that certain, less frequent non-dominant combinations not neglected. We observe that our DCNet also performs clearly best wrt. to all metrics, undermining its ability to successfully account for the different drug-drug interactions under dosage bias.
Methods
MIMIC-IV
TCGA
Biased MSE
MISE
SD-ISE
Biased MSE
MISE
SD-ISE
MLP
0.293
0.049
0.069
0.438
0.374
0.745
VCNet
0.691
0.558
0,749
0,551
0.406
0,474
DCNet (ours)
0.260
0.013
0.032
0,257
0.006
0,010
Table 6: Estimation error of dose-response estimators.
To give further intuition to the above, we select one random observation per dataset and plot the following functions: (i) the estimated GPS, (ii) the oracle individualized dosc-response surface, (iii) the estimates using DCNet, and (iv) the estimates using the baseline MLP. We compare two settings, namely with no dosage bias (a = 0) and thus sufficient overlap (in Fig. 8) and with dosage bias (a= 2) and thus clearly limited overlap (in Fig. 9).
The plots can be interpreted as follows. With no dosage bias (Fig. 8), both DCNet and MLP seem to approximate μ adequately (with slightly better estimates of DCNet at the modes at the boundary areas) and are thus suitable for naïve policy learning. However, the findings change in a setting with limited overlap (Fig. 9). Here, the limited overlap is indicated by areas with low estimated GPS/ f. Both estimators for j show clear deviations to the oracle μ.. However, the deviations are smaller for large parts in the case of our DCNet (as compared to the MLP). Evidently, the dose-response surface predicted by the MLP loses its expressiveness, especially for the nigh-dimensional TCGA dataset, and fails to model the maximum adequately. This leads to erroneous predictions when used plug-in estimator for policy learning, which can not be compensated by constraining the search space to regions with sufficient overlap. DCNet, on the other hand, is able to model the dose-response surface more adequately in areas with sufficient overlap due to its custom prediction head ensuring expressiveness. However, in regions with limited overlap, the splines of the prediction head extrapolate and lead to strongly incorrect predictions. Anyway, as our goal is learn reliable policy, we aim to exclude the regions with limited overlap and thus high uncertainty (yellow regions) such that they do not affect our final predicted policy.
In conclusion, when now applying our reliable policy optimization on top of our DCNet, we constrain the search space to the reliable regions, in which DCNet outperforms the MLP. Thus, we ensure reliable and accurate predictions for policy learning by using our method.
22

---

(a) MIMIC-IV
Extimsted GPS 7
35
30
Dosage
25
05
20
3.5
00
0.0
012
0.4
06
010
10
0.0
0.2
c4
06
08
1.0
0.0
022
0.4
0.6
C.8
1.0
0.0
0.2
0.4
0.6
0.
1.0
30
Doeage
Dosage 1
Dosage
Dosage
(b) TCGA
Estimated GPS
Dossge
05
222
1.84
14
104
00
064
0.0
0.2
0.4
06
08
1.0
0.0
022
0-4
06
08
1.0 00
022
0.4
0.5
0.0
1.0
90
02
0.5
0.5 0.8
1.0
Dosege
Dosage
Dosage
Dosage
Figure 8: Insights for one randomly sampled observation in the setting with dosage bias = 0. We show (i) the estimated GPS and, additionally, the individualized dose-response surface through (ii) the true oracle function, (iii) the estimates using our DCNet, and (iv) the estimates using the baseline MLP. The optimal dosage combination shown as red point.
(a) MIMIC-IV
Estimated GPS 7
UCNes
MPP
30
Dosage
25
0.6
20
33
00
00
02
B4
06
06
10
60
89
64
06
08
06
0.
6.4
06
C88
10
0.0
0.4
0.
088
1.0
000
Doeage
Dosege 1
Dosage
Dosege
(b) TCGA
Estrrated GPS i
Dosage
0.5
1.54
144
101
00
064
00
044
06
08
10
00
02
0-4
08
08
1.0 n0
02
0.4
08 0.5
1.0
00
09
0.4
08 0.8
10
Dosege
Dosege
Dosage
Dosege
Figure 9: Insights for one randomly sampled observation in the setting with dosage bias a = 2. We show (i) the estimated GPS and, additionally, the individualized dose-response surface through (ii) the true oracle function, (iii) the estimates using our DCNet, and (iv) the estimates using the bascline MLP. The optimal dosage combination shown as red point.
23

---

Computational complexity of policy optimization: To ensure scalability at inference time, we train additional policy neural network to directly predict the optimal policy, instead of performing optimization for new patients. We explain our motivation in the following: The complexity of direct optimization at inference time is non-trivial and depends on the complexity of the forward pass of DCNet a and the CNFs f, well the selected constrained optimization algorithm, and the number of patients to evaluate nevall While the exact complexity of the direct optimization depends on the specific solver cost solve. which, for most algorithms, scales heavily with the dosage dimension p. is at least O (Neva) (a + af) x Jsolve). This becomes costly for large-scale datasets in practice. In contrast, operations for evaluation, which is at most ( X duet), where net is the complexity of only the forward pass of the policy network. Hence, it usually holds net solve.
policy network is trained only once. Hence, upon deployment, it simply needs
24

---

E Modeling and Training Details
Dose-response estimator: For training our DCNet fi(t, we follow previous works from the baseline VCNet [2, 56]. We set the representation network ø to multi-layer perceptron (MLP) with two hidden layers with 50 hidden neurons each and ReLU activation function. For the parameters of the prediction head h. we use the same model choices as for Φ. Additionally, we use B-splines with degree two and place the knots of the tensor product basis at (1/3, 2/3 For the baseline MLP from Sec. 5, we ensure similar flexibility for fair comparison, and we thus select MLP with four hidden layers with 50 hidden units each and ReL.U activation. We train the networks minimizing the mean squared error (MSE) loss C2 = 1 E== (i(t,, xx2) - 3/5). For optimization, we use the Adam optimizer [43] with batch size 1000 and train the network for maximum of 800 epochs using early stopping with a patience of 50 on the MSE loss on the factual validation dataset.
We tune the learning rate within the search space {0.0001, 0.0005, 0.001, 0.005, 0.01}. and, for evaluation, we use the same criterion as for early stopping.
Conditional normalizing flows: For modeling the GPS f(t, we use conditional normalizing flows (CNFs). Specifically, we use neural spline flows [15] in combination with masked auto-regressive networks [13]. We use flow length of with 5 equally spaced bins and quadratic splines. For the autoregressive network, we use MLP with 2 hidden layers and 50 neurons each. Additionally, we use noise regularization with noise sampled from (O, 0.1). We train the CNFs by minimizing the negative log- log-likelihood (NLL) loss D =
Σn= log (f)) For optimization, we use the Adam optimizer with batch size 512 and train the CNFs for a maximum of 800 epochs using early stopping with a patience of 50 on the NLL loss on the factual validation dataset.
We tune the learning rate within the search space {0.0001, 0.0005, 0.001, 0.005, 0.01} and for evalu- ation we use the same criterion as for early stopping-
Policy learning: After training the nuisance models ja and f. we use them as plug-in estimators for our policy learning. For the policy network ft. we select a MLP with 2 hidden layers with 50 neurons each and ReLU activation. We set the reliability threshold
the 5%-quantile of the estimated GPS ftt, .)) of the train set, when not specified otherwise. We use Adam optimizers for updating 9 and A cach, with batch size 512, and train the network using the gradient descent-ascent optimization objective from Eq. (7) for a maximum of 400 epochs using early stopping with a patience of 20 on the validation loss from Eq. (9) on the factual validation dataset.
We set the learning rate for updating to 7a 0.01 and use random search with 10 con- figurations to tune the learning rate for updating the parameters of the policy network ne E (0.0001, 0.0005, 0.001, 0.005, 0.01} and the initial value of the Lagrangian multipliers X; E 51. To evaluate the performance during hyperparameter search, we use the same criterion as for early stopping. Then, after determining the hyperparameters, we perform policy.
runs to find the optimal
Our above procedure was chosen to ensure a fair and direct comparison to previous literature and to the naïve baselines, which enables us to show the sources of performance gain in our method (see Sec. 5). Note that when applying our method for reliable policy learning for dosage combinations in production, one might consider optimizing more exhaustively also over the different modeling choices of the nuisance estimators and the policy network.
25


Reviews:
[{"review_id": "sudIeLVrYw", "reviewer": "Reviewer_LeHo", "summary": "The paper introduces a new methodology to derive a policy to assign individualized dosage combinations relying on retrospectively collected observational data. The method relies on off-policy learning and try to account for the joint effect of multiple dependent dosages. A neural network is used to estimate individualized dose-response functions, accounting for the joint effect of multiple dependent dosages. The policy is then trained, avoiding regions with limited overlap to ensure reliable policy evaluation.\n\nThe developed method is then evaluated using semi-synthetic data relying on MIMIC-IV and TCGA. The robustness of the methodology is also assessed testing different levels of overlap for patients under different treatment regimens. The policy derived using the proposed method achieves low regret and low variance on all the experiments performed.", "questions": "As mentioned above, the results are only displayed for one split, being potentially prone to seed tuning, it would be interesting to repeat the split into train, validation and test sets and take this variation into account.\n\nGiven that you have access to the ground truth for the dose response function and GPS, could you also provide the error to estimate them in your experiments. This could provide insights on why the observed std is a 0 and motivate a more realistic dose-response function.\n\nIn the TCGA dataset and MIMIC-IV, the outcome Y is taken as the patient's survival. It's very uncommon to get a dataset with full observation of Y and the outcome is often right censored. Did you think of a way to take this into account?", "limitations": "T is the assigned p-dimensional dosage combination. In practice most of the time it's a sequence of treatment, varying dosage and number of cycles. The setting in the paper is an advancement over even more simple settings but is far from reflecting the data you can access in clinical routine.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper combines the idea of several recently published papers to derive the global method accounting for joint effects of dosage combinations. \n- A tensor product basis is used to estimate interaction effect and integrated into a model similar to [VCNet](https://arxiv.org/abs/2103.07861);\n- The generalized propensity score is estimated using conditional normalized flows, increasing reliability and avoiding discretization of the problem;\n- A NN is used for policy learning relying on the estimated dose response function and GPS;\n\nA nice protocol is then performed, assessing the performance of the model in the presence of dose combinations with a joint effect. \n", "weaknesses": "In the optimization problem of equation (6) (line 237),  the objective function and constraints can be non convex. There is no reason when applying the method of Lagrangian multipliers to have a function that is concave in the Lagrangian multipliers $\\lambda$. The method you rely on, [gradient descent ascent](https://arxiv.org/pdf/1906.00331.pdf) explicitly requires the concavity in $\\lambda$. This can lead to sub optimality and all the nice properties of the method are lost. \n\nGiven this, we can expect that the methodology will induce a lot of variability. If part of it is captured with the random restart proposed, we still expected it to be highly unstable. The results don't reflect that with a surprising std at 0 for the policy derived, whatever the task is. \nThis might  be due to the overly simple ground truth dose response function and GPS. Elements to assess this hypothesis are provided in the Questions. This also might be due to seed tuning. It would be interesting to repeat the split into train, validation and test sets and take the variation in the best underlying regret into account.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Answer:\nThank you for your detailed and constructive feedback. We improved our paper in the following ways:\n\n## Responses to “Weaknesses”\n- Thank you for giving us the opportunity to clarify the optimization part in our paper. We agree that our final objective in Eq. (7) is non-concave in the neural network parameters $\\theta$. However, for fixed $\\theta$, Eq. (7) is a linear function in terms of the Lagrange multiplier $\\lambda$ and thus **concave**. We therefore fulfill the assumption of the mentioned paper on gradient descent-ascent (Lin et al. 2021). Moreover, **we follow established literature** (e.g., Bui et al. 2022) on adversarial learning that uses gradient descent-ascent for non-convex concave objectives. The fact that Eq. (7) is non-convex in $\\theta$ means that we may converge to local minima, yet which must not be attributed to the optimization but to the neural network approach. In our paper, we thus address this by re-running the optimization with different initializations of the learnable parameters and find that the performance remains stable (next bullet item). \n\n- We thank you for the opportunity to explain why the performance of our framework method between the $k$ different re-runs is stable and of low variability. To improve our paper, we have thus added new results: (1) We show the performance across different splits and find that there is low variability ( **PDF, Table 3**). (2) We added an additional semi-synthetic experiment with a more complex setting. Therein, we again find that the performance is stable and of low variability ( **PDF, Table 4**).\nTo this end, our performance is **not** the result of seed tuning or overly simplistic nuisance functions. Rather, we exploit the properties of our medical setting (see Appendix B), as we have a complex multimodal dose-response function, where the dosage assignment is already informed to a certain degree by domain knowledge but must then be further optimized. This means, with increasing dosage bias $\\alpha$, the probability mass around the optimal dosage combinations increases, while, in areas further away from the optimum, overlap becomes more limited. This is a reasonable assumption in medicine, as clinical practice is based on expert knowledge for deploying drug-dosage prescriptions, which minimizes the risk of observing harmful treatment assignments frequently.\n\n**Action:** We will add additional numerical results that demonstrate that our performance is stable to our paper. As shown above, the results confirm the effectiveness of our framework.  \n\n## Responses to “Questions”\nWe thank the reviewer for the suggestions and added the following experiments.\n\n- **Performance across different splits (PDF,  Table 3&4)**: We find that our novel method consistently outperforms the baselines, yielding a consistently lower regret and lower variability. Note that this is *between* the different seeds as well as *within* the seeds between the k=5 restarts. As such, we conclude that the performance of our framework is robust.  \n\n- **Results for nuisance estimators (PDF, Table 1)**: We further report the performance of our nuisance estimators. Here, for the dose-response estimators we show: (1) the performance on the test dataset measured by the mean integrated squared error (MISE) on the whole covariate treatment space; and (2) the MISE only in regions with sufficient overlap. Our results demonstrate that our DCNet is **highly effective** in learning the dose-response function. Compared to the baselines, we observe that our DCNet does not try to learn the whole complex dose-response surface (which includes low-trust regions) but focuses strategically on the areas with sufficient overlap (as desired). This is key for our framework as it ensures accurate targeting of the optimal policy. We provide further intuition in Appendix D.\\\nWe also report results from a more complex setting **(PDF, Table 4)** to show that our method generalizes to other complex scenarios. For this, we model the GPS as a multimodal conditional density and ensure that the optimal dosage combination is not necessarily located in regions around the maximal density. Importantly, our framework outperforms the baselines by a clear margin (i.e., lower regret and lower variability). This confirms the robustness of our framework.\n\n- **Extensions of our experimental setup.** Thank you for this important question. We designed our analysis around patient survival as it is often used for benchmarking in medical practice. Needless to say, our method can be adapted to other metrics of interest and even to right-censored data. Here, an interesting direction is to extend our framework with, for example, the approach in Curth et al. (2021). Importantly, our framework consists of three steps (dose-response estimation, GPS estimation, constrained optimization) which are not restricted to our specific setting of observed outcomes but are general. In particular, our framework can thus be extended to survival analysis with right-censored data in a straightforward manner. \n\n**Action:** We will add the above experiments to our revised paper. We will also add a discussion on how to handle right-censored data. \n\n## Response to Limitations\nWe agree that medical data often is even more complex than in our scenario, and we will add this to our discussion of limitations. Nevertheless, we would like to emphasize that our paper makes important contributions over existing literature in order to advance personalized treatment design in real-world clinical applications.\n\n## Remark to Contribution\nSince the contribution was evaluated as “1 Poor”, we would like to emphasize the novelty of our work and refer to our General Author Rebuttal. \n\nBui, Tuan Anh, et al. \"A unified wasserstein distributional robustness framework for adversarial training.\" arXiv (2022)\\\nCurth, Alicia,et. al. \"Survite: Learning heterogeneous treatment effects from time-to-event data.\" NeurIPS (2021)"}, {"review_id": "sufrTiZfhu", "reviewer": "Reviewer_bKrg", "summary": "The paper deals with an important and interesting problem in personalized decision making. In particular, the paper considers the setting when there are multiple continuous treatments available and models the joint effect of dosage combinations. A novel off-policy learning algorithm is proposed to solve the problem using the well-constructed network architecture. Specifically, the paper also considers the issue when the (strong) positivity assumption of the propensity score does not hold. To solve this problem, the paper modifies the proposed algorithm to achieve a reliable estimation of the policy value by avoiding regions with limited overlap. Finally, several experiments and real data analysis are provided to demonstrate the performance of the proposed method. ", "questions": "1. In contrast to the approaches mentioned in the provided references, which focus on combining similar treatments and estimating treatment structures, this paper tackles the limited overlap problem in the context of continuous dosage by constraining the search space and shrinking the entire searching space. While the methods may differ in their specific techniques, there could be interesting connections between the two approaches. Exploring and discussing these potential connections could provide valuable insights into how different strategies can address the limited overlap problem across different treatment settings. Such a discussion would contribute to a broader understanding of the problem. In particular, are there any connections between these two methods? That would be interesting to discuss.\n\n2. When reducing the searching space to address the limited overlap problem, there may be concerns about whether the estimated policy with the proposed method leads to sub-optimal values at the population level. Understanding the potential trade-offs between reducing the search space and achieving optimal population-level outcomes may be valuable for both theoretical considerations and practical applications. If reducing the searching space, will the estimated policy with proposed method leads to a sub-optimal value in the population level?\n\n", "limitations": "Please see it in Weaknesses section. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper has a good presentation to illustrate the target problem, underlining causal framework, proposed methods, and related advantages. The authors have skillfully laid out the paper, providing a clear and concise overview of the problem at hand. The methods proposed by the authors are well-described and thoroughly explained, ensuring that readers can easily grasp the technical aspects of the approach. Furthermore, the paper showcases the advantages of the proposed methods, demonstrating how they address the limitations of existing approaches and offer novel contributions to the field. The well-designed format of presenting the problem, framework, methods, and advantages in a cohesive and coherent manner enhances the overall quality of the paper.\n\n2. The paper provides real data analysis in multiple dosage settings. This empirical analysis adds substantial value to the research as it validates the proposed methods and demonstrates their effectiveness in practical scenarios. Moreover, the multiple dosage settings explored in the analysis indicate the robustness of the proposed methods, making the findings applicable to a wider range of scenarios. ", "weaknesses": "1. The paper addresses a specific issue related to limited overlap, which can lead to unreliable estimations of policy value. However, it is worth noting that similar and related problems have been discussed in the literature. To the best of our knowledge, the following papers have explored these problems:\n\n(1). Ma et al. (2023) Learning Optimal Group-structured Individualized Treatment Rules with Many Treatments. Journal of Machine Learning Research.\n\n(2). Ma et al. (2022) Learning Individualized Treatment Rules with Many Treatments: A Supervised Clustering Approach Using Adaptive Fusion. NeurIPS.\n\nAlthough the above papers primarily focus on settings involving discrete treatments, they have also carefully analyzed the limited overlap problem. Specifically, they examine cases where the number of treatments approaches $+\\infty$, which can be treated as one of the special cases of continuous treatment, and propose methods for identifying latent treatment grouping structures (combine treatments or combine levels of doses) to address issues of unbalanced treatment assignment and limited overlap. It would be beneficial to include further discussion and comparisons with these methods in the paper, as they may offer valuable insights and contribute to the overall understanding of the problem.\n\n2. The paper mentions an important term, $\\bar{\\epsilon}$, which acts as a pre-specified threshold controlling the minimum overlap required for the policy estimation when shrinking the treatment-covariate space. It is interesting and important to investigate and provide guidelines for selecting an appropriate value for $\\bar{\\epsilon}$ in the experimental analysis. This could involve sensitivity analysis or exploring different threshold values to evaluate their impact on the estimated policies and their performance metrics. Assessing the robustness of the proposed method to the choice of $\\bar{\\epsilon}$ would provide insights into its practical applicability and generalizability across various scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed, constructive, and positive feedback!\n\n### Response to “Weaknesses”\n1. **Importance of continuous vs. discretized treatments.** Thank you for introducing this very interesting and related additional literature to us!\\\nWe understand the importance of benchmarking our framework for continuous treatments against methods that work instead on discretized settings. In principle, translating our continuous setting into a discretized setting would allow us to use a variety of established methods (such as the works that you cited above). We thus added experiments where we discretize the setting and then evaluate the solution quality (see **Response PDF, Table 4&5**). We find that the discretized approaches are consistently inferior compared to our approach which directly operates on continuous values. In particular, we also benchmark an “oracle” discretized baseline (where we assume perfect knowledge of the dose-response function and perfect individual treatment assignment). This serves as an upper bound for all policy learning methods with discretization. Still, that discretization leads to a higher regret than our proposed method, which confirms the advantages of leveraging continuous dosage information. \\\n**Action:** We will cite the papers. We will add further discussions and comparisons between our method and the proposed papers on learning individualized treatment rules with many treatments. Specifically, we will discuss the connections and differences of the methods for tackling the challenges of limited overlap and biased treatment assignment, as well as the role of discrete and continuous treatment. We will also add the numerical results from the discretization baselines to our revised paper. \n\n2. **Guidelines to select the reliability threshold.** We thank the authors for raising the important point of how to select the reliability threshold $\\overline{\\varepsilon}$ and for giving us the opportunity to elaborate on the approach we used in our work. We offer a detailed description in **Appendix C** and summarize key points in the following. For our framework, we developed a heuristic guideline to choose $\\overline{\\varepsilon}$ dependent on $x\\%$-quantiles of the estimated GPS on the train dataset. As a result, we aim to learn optimal dosages, which have at least the same estimated reliability as $x\\%$ of the observed data. In our experiments, we set $x=5$. Furthermore, we performed a series of sensitivity analysis where we explore different threshold values and then evaluate the impact of $\\overline{\\varepsilon}$ on the regret of the estimated policies. We show the results in **Response PDF, Figure (1)**. We find that, depending on the dataset, different quantiles for selecting $\\overline{\\varepsilon}$ can lead to small performance differences. However, we find that lower values tend to lead to less variability. Also, we find that, even when there is higher variability within the $k$ runs, for every setting a run with low regret is selected. This demonstrates that, by using our proposed guideline, our method yields **robust results also over different values of the reliability threshold**. \n\n### Responses to “Questions”\n1. We fully agree that discussing potential connections between the provided references and our method will contribute to our paper in terms of providing an even more extensive background of reliable off-policy learning under complex treatments with limited overlap. \\\n**Action:** We will cite the papers and add a discussion to our revised paper. We will also benchmark the original methods and our “oracle” approach from above to show the importance of dealing with continuous treatments in medicine. \n\n2. We thank you for the interesting question and the opportunity to discuss the effects of reducing the search space on the population-level outcomes. Importantly, even though our framework reduces the search space, it does so in a way that we still optimize against the optimal solution.\n- **Theoretical explanation.** We would like to emphasize that our method reduces the search space of the treatments during policy learning on an individualized level (conditioned on the patient covariates), but not necessarily on the population-level. This ensures that the treatment recommendation is still flexible across different patients and thus enables optimal individualized policy learning. Formally, recall that the empirical policy value $\\hat{V}(\\pi) = \\frac{1}{n}\\sum_{i=1}^{n} \\hat{\\mu} \\left( \\pi(x_i), x_i \\right)$ is a metric for the population-level performance of a policy. It is then the same as the mean of the individual predicted outcomes. Hence, optimizing over the individual level and the population level leads to the same optimal policy. Hence, our framework which constrains the search space still targets the optimal policy at the population level. \n- **Empircial confirmation.** We also demonstrate the above empirically in our experiments. Here, we measure the regret which is the difference between the policy value of the optimal policy and the true policy value of our learned policy. We observe that our framework constantly outperforms the policies with unrestricted search spaces (called “naive” in our paper) wrt. to the regret. As a result, this provides empirical evidence that demonstrates the strong performance of our framework at a population level.\n"}, {"review_id": "NgGpuOt40v", "reviewer": "Reviewer_aVCP", "summary": "(I have more background in offline RL than causal inference, and my description below may include some RL terminologies as the two are closely related.) \n\nThis submission studies the problem of off-policy learning of dosage recommendations in personalized medical decision making, using a causal inference framework. The proposed approach operates in three stages: the first two stages aim to learn the dose-response function (similar to the Q-function) and the logging policy (used to identify poor overlap), whereas the third stage combines the two to learn a reliable policy using constrained optimization (max performance while avoiding poor overlap), solved via a minimax gradient optimization of the Lagrangian form of the objective. Experiments are conducted using semi-synthetic datasets for ventilation (MIMIC-IV) and gene expression (TCGA).", "questions": "- For medical applications, how common is it that practitioners use exact dosages vs dosage levels? What's the practicality of an exact dosage specification in real-world problems? More discussion about this is appreciated. \n- L131 Why is ε = 0 considered weak overlap instead of no overlap? \n- L183 mentions “polynomial spline basis functions”, a possibly unfamiliar concept to many readers. Can you provide the precise definition? \n- Sec 4.1 is very dense, I still don't quite understand it after several read throughs. It requires a lot of thinking to understand p-dimensions, may be easier to illustrate using  a simple example with two dimensions. \n- L274 training stability: \"as there is no guarantee for global convergence in the non-convex optimization problem\", do you observe any training instability in the experiments and how much variation is there? When selecting the best run over K random restarts, what is K set to in the experiments? \n- L95, L247, Sec 4.3: a neural network policy is trained to approximate the individualized optimal dosage, and is claimed to have a computational advantage. However, no empirical results (runtime of two implementations) or analysis (big-O complexity) is shown for compute scalability. Also, is there any performance loss from using this neural network policy compared to an exact argmax? \n\nMinor:\n- Figures: fonts are too small in all figures\n- L95 \"unfeasible\" -> \"infeasible\"\n- L116 “Ω denotes the set of all possible policy classes” is confusing, I think Ω should be the set of all policies. \n- L246 I think \"max\" in the equation should be argmax\n- L289 \"when filtering for T\" -> what does this mean? \n\n---\n\n**Updates after rebuttal and discussion:**\n\nThank you authors for the response, it resolved most of my questions. I have raised my rating from 5 to 6. I would like to see the final version incorporating the updated limitation section and the details on dataset characteristics.", "limitations": "Sec 6 Conclusion separately discusses limitations and broader impacts. Though the proposed approach aims to avoid unreliable treatments, I think it is still necessary to acknowledge potential failure scenarios of the proposed approach and discuss what are the possible downstream impacts, especially since this work is focused on the high-stakes decision making of treatment dose recommendation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Writing is clear with very organized descriptions of each stage of the proposed approach. \n- Well-motivated problem by real-world medical applications where one needs to determine multiple continuous treatments with dependent effects. \n- Code is provided as part of the supplement (I did not check or run the code).", "weaknesses": "- The contribution of this work should be further clarified. As \"several methods for off-policy learning aim at discrete treatments\" (L27), it may be worth pointing out whether solutions exist for multiple *discrete* treatments (instead of continuous dosages) and explain why those approaches don't directly apply, and how well do we expect those methods to perform for dosage combinations if we discretize the dosages. Furthermore, specific architectures are used for estimating dose-response and GPS, and I'm not entirely convinced why simpler neural networks do not work. \n- Furthermore, there are many recent works on offline RL and OPE that make use of value functions and logging policies to achieve reliable policy learning (e.g., references below, including many for continuous actions). Some discussion about how this work is related and/or different from those approaches would make the paper more complete and resonate with a wider audience. \n- In the experiments, the baselines considered may be more accurately described as ablation studies. These are certainly important for understanding the sources of performance gains, but I think it is important to include more direct baselines. Since \"there are no existing methods tailored to off-policy learning for dosage combinations under limited overlap\" (L329), how well does a standard causal effect estimation method work here? \n\nReferences:\n[1] Off-Policy Deep Reinforcement Learning without Exploration. https://arxiv.org/abs/1812.02900\n[2] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. https://arxiv.org/abs/1906.00949\n[3] Conservative Q-Learning. https://arxiv.org/abs/2006.04779", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to “Weaknesses”\n- **Importance of modeling _continuous_ treatments.** Many applications in medicine rely on continuous treatments where discretization would often miss the optimal solution for patients (e.g., chemotherapy).\\\nWe performed additional experiments to demonstrate the gain of our continuous setting over discretized settings **(PDF, Table 5&2)**.  Specifically, we demonstrate the shortcomings of discretization, where we evaluate an oracle performance of different grid discretizations, assuming perfect knowledge of the dose-response function and perfect treatment assignment. We thus interpret our “oracle” as an upper bound for all policy learning methods with discretization. We find that discretization itself leads to a higher regret than our proposed method. To this end, our results confirm the advantages of modeling continuous dosages.  \\\n**Simple neural network baseline.** To show the necessity of our specifically developed network architectures, we performed **additional experiments (PDF,  Table 1&2)**. We find that our proposed framework is consistently superior:   \nAblation (1) compares a simple MLP in Step 1 of our method, which is especially inferior for settings with highdimensional confounders (as in the TCGA dataset). (Intuition is given in Appendix D.)\nAblation (2) compares the SOTA network for estimating dosage effects VCNet, which is designed for *independent* dosages and not for *dependent* dosage combinations. Different from VCNet, our DCNet is able to model non-linear interactions between the $p$ dosages. This results in clearly superior performance. \nAblation (3) varies the estimation in Step 2 of our framework. Therein, we seek to model the GPS, which requires modeling a multidimensional conditional density rather than a simple conditional mean prediction. Hence, a simple neural network is *not* applicable. Instead, we compare  our CNFs to mixture density networks (MDN). Evidently, MDNs lead to a very large standard deviation across runs and thus fail to achieve reliable policy learning. As such, our CNFs are preferred.\n- **Difference to other literature streams.** Thank you for the suggestion to compare our setting against ORL for continuous actions. The main **differences** are: ORL assumes a Markov decision process and sequential decision-making. In contrast, we focus on non-sequential, off-policy learning from observational data.\nAlso, unlike ORL or standard OPE, we leverage the causal structure of treatment assignment and the dose-response function to return **causal** estimands. This allows us not only to learn decisions but also causal estimates for potential outcomes. This is crucial for medical practice, where physicians would like to reason over different treatments rather than blindly following a fully automated system.\n- **Additional naive baselines.** We would like to emphasize that the ablation studies (first learning a dose-response estimator like the MLP and then optimizing over it) **also** are standard baseline causal effect procedures for off-policy learning, often referred to as the “direct method”. To even better assess our method, we added two further causal baselines to our results **PDF, Table 1, 2**: (1) VCNet as part of the direct method (2) DR Policy Forest as a SOTA doubly-robust method for discretized dosage combinations (5x5) (using logistic regression for propensity score modeling and a weighted lasso regression for outcome modeling). Note that both baselines ignore specific characteristics of our setting (e.g., dependency among dosages). Altogether, we observe that our framework consistently performs best.\n**Action:** We will include our additional above experimental results in our final paper. We will also discuss more extensively the proposed literature streams.\n## Answer to “Questions”\n- Continuous dosages are common for instance, in chemotherapy, where current guidelines are based on different multiplicative formulas using continuous patient features.\n**Action:** We will expand our motivation to highlight the need for modeling dosages as a continuous variable.\n- We define overlap as $f(t, x) > \\varepsilon$, meaning that $\\varepsilon=0$ still implies a non-zero density. (“weak overlap”).\n- We will add a more detailed background about tensor product splines to our revised paper.\n- We will add an intuitive example to our revised paper.\n- We set $K=5$ in our experiments. The variation within the $K$ runs is displayed in the column “std”. We also included additional results for different restarts of our method with different seeds and train/val/test splits in **PDF, Table 4&5**. Overall, our framework has the lowest variation and is thus highly **stable**.\n- The complexity of a direct optimization is non-trivial and depends on the complexity of the forward pass of DCNet and CNFs, and the number of patients to evaluate. Also, the exact complexity of the direct optimization depends on the solver cost $\\sigma$, it is at least $O(n \\times k \\times \\sigma)$. This becomes costly for large-scale datasets. In contrast, our policy network is trained only once. Hence, upon deployment, it is constant (i.e., $O(1 \\times \\sigma)$ when measured in terms of $\\sigma$). \n- The argmax cannot be determined analytically, because it depends on the learned nuisance parameters. However, when comparing our predicted policy against directly optimized dosaging, we observed barely any differences. One reason could be that the policy network introduces additional regularization for observations in low overlap regions. \n## Response to “Minor” \nThank you! We will add the suggested changes to our paper. \n“Filtering for T (L289)”: MIMIC-IV is a large dataset of patients in intensive care units for different causes, where we filter for patients who are ventilated.\n## Response to “Limitations”\nThank you! We will expand our discussion around further potential failure scenarios and possible downstream impacts when applying our method in clinal practice."}, {"review_id": "WWrA2l4yPe", "reviewer": "Reviewer_5Gdr", "summary": "The paper proposed a novel method for off-policy learning for continuous dosage combinations that aims to be robust to action space sparsity. To account for drug-drug interactions, they proposed a dosage combination network (DCNet) which leverages a tensor product basis to estimate smooth interaction effects in the dose-response surface, incorporating all dosages simultaneously into parameterization by only one head. The DCNet is updated by minimizing the MSE between the individualized dose-response function (the expected patient outcome of a given treatment in given patient covariates) and patient outcome. Then they use conditional normalizing flows to model the stochastic and multimodal behaviour policy which they call the generalized propensity score (GPS). Finally, they learn a parametric policy in a constrained optimization problem that maximizes individualized dose-response function under the learned policy subject to the probability of the proposed treatment in the GPS being larger than a threshold. This is approximately solved as an unconstrained Lagrangian problem through stochastic gradient descent-ascent with random restarts. Experiments were conducted on two real healthcare datasets. Comparisons to ablated versions (MLP instead of DCNet, without overlap constraint) suggest that the proposed components reduce regret and improve stability.", "questions": "1. do you have to learn an analytical form of the GPS if it is only to indicate overlap?\n2. what is the harm of assuming independent dosage dimensions, which is mentioned several times throughout the paper but never clearly explained\n3. why the single-head individualized dose-response function trained against the overall treatment outcome can account for drug-drug interactions? Is it possible that one drug dominates the others in the model training, or one common combination of dosages dominate less frequent combinations that actually entail more subtle drug-drug interactions? As I understand, the tensor product basis of the proposed DCNet is all inner work of the model, whereas the input, output and training objective are all subject to non-individualized information.", "limitations": "please see above", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. proposed a novel method for off-policy learning for continuous dosage combinations\n2. a viable design for constrained policy optimization", "weaknesses": "1. dosage dimensionality in experiments is just 2 or 3, it is hard to say wether the obtained results can generalize to much higher dimensionalities, or treatments intended for different diseases simultaneously\n2. the individualized weights in the proposed DCNet are only trained in an unsupervised fashion, i.e. the training objective does not reflect individual dosages\n3. illegible fonts in figures", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for giving us the opportunity to clarify certain aspects of our paper.\n\n### Response to “Weaknesses”\n1. Thank you for raising this important point. Our framework can be easily extended to much higher dimensions. To demonstrate this, we added new results for higher dimensions (see **PDF, Fig. 2**). We observe that our framework scales well with higher dimensions and clearly outperforms the other baselines. As such, our framework is directly applicable to multi-disease settings, each requiring different treatments. \n2. We would like to clarify that our DCNet is indeed trained in a **supervised** fashion. We further would like to clarify that the training objective _does_ reflect individual dosages. We consider an observational dataset for $n$ patients with i.i.d. observations $\\{(x_i, t_i, y_i)\\}{_{i=1}^{n}}$ sampled from a population $(X, T, Y)$, where $X \\in \\mathcal{X} \\subseteq \\mathbb{R}^{d} $ are the patients' covariates, $T \\in \\mathcal{T} \\subseteq \\mathbb{R}^{p} $ is the assigned $p$-dimensional dosage combination per patient (i.e., multiple continuous treatments), and $Y \\in \\mathbb{R}$ is the observed outcome per patient. Each patient $x_i$ is then assigned one individualized dosage combination $t_i$ consisting of one dosage per each of the $p$ drugs leading to the outcome $y_i$, reflecting individual dosages and supervised data. The challenge here is that we only observe _one_ individual dosage combination per patient, but we want to estimate the individualized dose-response function for _all_ potential dosage combinations in $T$ for an individual patient $x$, i.e. $\\mu(t, x) = \\mathbb{E} [Y(t) \\,\\mid\\, X=x ]$. However, when the standard assumptions of causal inference hold (i.e., consistency, ignorability, and overlap), we can estimate the function for _all_ potential dosage combinations by regressing $\\mu(t, x) = \\mathbb{E} [Y\\mid T=t, X=x ]$. Furthermore, our DCNet will then **generalize** to different values of $t$ and $x$, that is, to different dosages and different patients. As a result, we thus estimate $\\mu(t, x)$ for individualized regions. This is a major advantage of our method, and it is the reason why we build upon causal inference. Still, our framework avoids regions of the policy with large uncertainty to ensure reliable decisions. \n3. We will increase the font size in our plots.\n\n### Responses to “Questions”\n1. We need to learn the GPS, i.e. the conditional density $f(T, X)$, over the whole covariate-treatment space for our Lagrangian optimization objective. The reason is that the learned GPS $\\hat{f}\\left( \\hat{\\pi}(x_i), x_i \\right)$ needs to be evaluated at every training step for every proposed dosage combination $\\hat{\\pi}(x_i)$, so that we can calculate the training loss $\\mathcal{L}_{\\pi}(\\theta, \\lambda)$ and its gradients. Intuitively, it is necessary to have a fully learned form of the GPS in order to update the policy network parameters to shift the suggested dosage combinations into regions with sufficient overlap, when the previous suggestions target regions with an estimated GPS below the reliability threshold $\\overline{\\varepsilon}$. To clarify this, we will change the wording from “analytical form” to “fully estimated form”. \n2. We thank you for the opportunity to spell out the importance of modeling the _dependency_ of different drugs. We proceed in two ways: \\\n(i) **Numerical explanation.** We added a comparison against VCNet where drug-drug interactions are **not** explicitly modeled but rather taken as **independent**. The results are in **PDF, Table 2**. Evidently, our proposed framework _with dependency_ outperforms the ablation _without dependency_ by a clear margin. \\\n(ii) **Theoretical explanation.** Let us assume a framework _without_ dependency. Then, non-linear interaction effects between the drugs can *not* be captured and the dose-response function can *not* be estimated sufficiently but will be lost. In medical applications, this can have severe negative and even harmful consequences and should be avoided. For example, the VCNet baseline (as opposed to our DCNet) will underestimate synergistic effects of different drugs. Instead, a framework is needed where drug-drug interactions are carefully modeled. To the best of our knowledge, our DCNet is the first framework for causal inference in such setting.  \n3. Our proposed DCNet can successfully model drug-drug interactions. The reason is that the dosage information is fused through a tensor product basis, which allows different drugs to interact in complex, non-linear, and non-additive ways. \\\n We carefully checked whether certain drugs may dominate others but did not find such evidence. Instead, we performed a simulation where we simply set certain drug dosages to 0 for some patients and then examined the performance. Here, we find that our results are robust. Still, we find that some drugs that are highly influential in changing patient health, also explain larger parts of the treatment effect, in line with what is expected from medical practice. This demonstrates the effectiveness of our framework. \\\nFinally, we would like to emphasize again that our data, our model, and our training objective carefully consider _individualized_ information from patients. Importantly, we do neither learn aggregated nor average effects; instead, we learn individualized effects at the individual, patient-level, so that the estimated effects vary from patient to patient (we kindly refer to our second answer from our response to “Weaknesses” for technical details). If there is still something unclear, we are happy to clarify this in the discussion period.\n\nReferences:\n[1] Nie, Lizhen, Mao Ye, and Dan Nicolae. \"VCNet and Functional Targeted Regularization For Learning Causal Effects of Continuous Treatments.\" International Conference on Learning Representations. 2020.\n"}]

Author rebuttal:
Thank you very much for the constructive and positive evaluation of our paper and your helpful comments! We addressed all of them in the comments below and uploaded additional results as a PDF file. 

Our main improvements are the following:
- We provide further **extensive experiments**: \
(1) We report the performance of our nuisance estimators and show our framework is superior. \
(2) We report results for multiple train/val/test splits for (a) our existing setting and (b) for an even more complex, multimodal setting. Here, we demonstrate that the performance of our framework is robust across all settings. \
(3) We include additional causal baselines for (a) dose-response estimation, (b) GPS estimation, and (c) policy learning. We observe that all components of our framework lead to performance gains. \
(4) We show that our framework for continuous treatments outperforms baselines based on discretized treatments. \
(5) We compare additional, standard causal baselines and find that our framework still perfoms best. \
(6) We demonstrate the scalability of our framework to higher dimensions.

- We discussed the **connections of our method to further related work** (e.g., offline reinforcement learning, causal inference for discrete treatments) and clarify the necessity and advantages of using our method for reliable off-policy learning for dosage combinations.
- We elaborate on the **role of the reliability threshold** $\overline\varepsilon$ for reliable policy learning in our method. We further propose a guideline for selecting $\overline\varepsilon$. We also show that our framework is robust to different choices of $\overline\varepsilon$.

As a summary, we would like to emphasize our contribution again: To the best of our knowledge, we are the first to tackle the problem of learning optimal individualized dosage combinations from a causal perspective. For this, we adjust for the naturally occurring but non-trivial problems of drug-drug interactions and limited overlap. Crucially, our study setting is highly relevant in medical practice and present the foundation for personalized decision-making in cancer therapy and critical care.  Even more, we implemented additional baselines and show that our method clearly outperforms existing methods. Rather, our problem requires a tailored and non-trivial framework as ours. As such, we are confident to provide a valuable contribution toward learning reliable policies for dosage combinations in medicine.

We will incorporate all changes (labeled with **Action**) into the camera-ready version of our paper. Given these improvements, we are confident that our paper will be a valuable contribution to the machine learning for healthcare literature and a good fit for NeurIPS 2023.


Meta-review: This paper proposes off-policy learning for drug-drug combinations. The method trains treatment specific response functions followed by policy learning that restricts to learning under conditions of overlap to extrapolate drug-drug combination effects. Experiments in TCGA and MIMIC on improving survival are demonstrated.

Overall the reviewers find the contribution valuable and the authors have made a significant effort in addressing clarifying questions and concerns from reviewers. I do believe all additional analysis should be incorporated in the main paper. Particularly the sensitivity analysis.

Otherwise I do not have concerns about acceptance.
Decision: Accept (poster)