SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \hat {V} _ {\tau} (\boldsymbol {s}, l, t) - \hat {V} _ {\tau} \left(\boldsymbol {s} ^ {\prime}, l, t\right) \right| \leq \left| R \left(\boldsymbol {s}, a ^ {*}\right) - R \left(\boldsymbol {s} ^ {\prime}, a ^ {*}\right) \right| \\ + \left| \hat {V} _ {\tau} \left(\tilde {\boldsymbol {s}}, l _ {a ^ {*}}, t + 1\right) + L _ {t + 1} \| \tilde {\boldsymbol {s}} - \boldsymbol {s} _ {a ^ {*}} \| \right. \\ - \hat {V} _ {\tau} (\tilde {s}, l _ {a ^ {*}}, t + 1) + L _ {t + 1} \| \tilde {s} - s _ {a ^ {*}} ^ {\prime} \| \\ = \left| R \left(\boldsymbol {s}, a ^ {*}\right) - R \left(\boldsymbol {s} ^ {\prime}, a ^ {*}\right) \right| + L _ {t + 1} \| \tilde {\boldsymbol {s}} - \boldsymbol {s} _ {a ^ {*}} \| - \| \tilde {\boldsymbol {s}} - \boldsymbol {s} _ {a ^ {*}} ^ {\prime} \| \| \\ \stackrel {(*)} {\leq} C _ {a ^ {*}} \left\| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right\| + L _ {t + 1} \left\| \boldsymbol {s} _ {a ^ {*}} - \boldsymbol {s} _ {a ^ {*}} ^ {\prime} \right\| \\ \stackrel {(*)} {\leq} C \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| + L _ {t + 1} K _ {\boldsymbol {u} _ {t}} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| = L _ {t} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \|, \\ \end{array}
$$
where in  $(\ast)$  we use the triangle inequality and the fact that the SCM  $\mathcal{C}$  is Lipschitz-continuous, and  $(\ast\ast)$  follows from Lemma 5.
Algorithm 2: Graph search via  $A^{*}$
Input: States S, actions A, observed action sequence  $\{a_{t}\}_{t = 0}^{t = T - 1}$ , horizon T, transition function  $F_{\tau ,t}^{+}$ , reward function R, constraint k, initial state s0, heuristic function  $\hat{V}_{\tau}$ .  
Initialize: NODE  $v_{0}\gets \{\text{"tuple":}(s_{0},0,0)\text{"rdw":}0\text{"par":}Null\text{"act":}Null\}$  STACK action_sequence  $\leftarrow []$  QUEUE  $Q\gets \{root\}$  SET explored  $\leftarrow \emptyset$   
while True do  
 $v\gets \mathrm{argmax}_{v^{\prime}\in Q}\{v^{\prime}.rwd + \hat{V}_{\tau}(v^{\prime}.tuple)\}$ ;  $Q\gets Q\backslash v$  ; /* Next node to visit */ if v.tuple  $= (*,*,T)$  then while v.par  $\neq$  Null do action_sequence.push(v.act); /* Retrieve final action sequence */ v=v.par end return action_sequence  
end explored  $\leftarrow$  explored  $\cup$  {v}; /* Set node v as explored */ if  $l = k$  then availabeproperties  $\leftarrow \{a_t\}$  else availabeproperties  $\leftarrow \mathcal{A}$   
end for a  $\in$  availableActions do  $(s,l,t)\gets v.tuple(s_a,l_a)\gets F_{\tau ,t}^{+}((s,l),a);$  /* Identify v's children nodes */  $v_{a}\gets \{\text{"tuple":}(s_{a},l_{a},t + 1)\text{"rdw":}v.rwd + R(s,a)\text{"par":}v,\text{"act":}a\}$  if  $v_{a}\notin Q$  and  $v_{a}\notin$  explored then  $Q\gets Q\cup \{v_a\}$ ; /* Add them to the queue if unexplored */ end end
E A\* algorithm
Algorithm 2 summarizes the step-by-step process followed by the  $A^{*}$  algorithm. Therein, we represent each node  $v$  by an object with 4 attributes: (i) the "tuple"  $(s, l, t)$  of the node, (ii) the total reward "rwd" of the path that has led the search from the root node  $v_{0}$  to the node  $v$ , (iii) the parent node "par" from which the search arrived to  $v$ , and (iv) the action "act" associated with the edge connecting the node  $v$  with its parent. In addition to the queue of nodes to visit, the algorithm maintains a set of explored nodes, and it adds a new node to the queue only if it has not been previously explored. The algorithm terminates when the goal node is chosen to be visited, i.e., the "tuple" attribute of  $v$  has the format  $(*, *, T)$ , where * denotes arbitrary values. Once the goal node  $v_{T}$  has been visited, the algorithm reconstructs the action sequence that led from the root node  $v_{0}$  to the goal node and returns it as output.
(a) MC-Lipschitz
(b) MC-Uniform
Figure 5: Computational efficiency of our method, measured by means of the Effective Branching Factor (EBF), under three different anchor set selection strategies against the size of the anchor set  $S_{\dagger}$ . In all panels, we set  $L_{h} = 1.0$ ,  $L_{\phi} = 0.1$  and  $k = 3$ . Error bars indicate  $95\%$  confidence intervals over 200 executions of the  $A^{*}$  algorithm for 200 patients with horizon  $T = 12$ .
(c) Facility-Location
F Experimental evaluation of anchor set selection strategies
In this section, we benchmark the anchor set selection strategy presented in Section 4 against two alternative competitive strategies using the sepsis management dataset and the same experimental setup as in Section 5. More specifically, we consider the following anchor set selection strategies:
(i) MC-Lipschitz: This is the strategy described in depth in Section 4, based on Monte Carlo simulations of counterfactual episodes under randomly sampled counterfactual action sequences. Notably, the time steps where each counterfactual action sequence differs from the observed one are sampled proportionally to the respective Lipschitz constant  $L_{t}$  of the SCM's transition mechanism. To ensure a fair comparison with other strategies, instead of controlling the number of sampled action sequences  $M$ , we fix the desired size of the anchor set  $S_{\dagger}$ , and we repeatedly sample counterfactual action sequences until the specified size is met.
(ii) MC-Uniform: This strategy is a variant of the previous strategy where we sample the time steps where each counterfactual action sequence differs from the observed one uniformly at random, rather than biasing the sampling towards time steps with higher Lipschitz constants  $L_{t}$ .
(iii) Facility-Location: Under this strategy, the anchor set is the solution to a minimax facility location problem defined using the observed available data. Let  $S_{o}$  be the union of all state vectors observed in all episodes  $\tau$  in a given dataset. Then, we choose an anchor set  $S_{\dagger} \subset S_{o}$  of fixed size  $|S_{\dagger}| = b$ , such that the maximum distance of any point in  $S_{o}$  to its closest point in  $S_{\dagger}$  is minimized. Here, the rationale is that counterfactual states resulting from counterfactual action sequences for one observed episode are likely to be close to the observed states of some other episode in the data. Formally,
$$
\mathcal {S} _ {\dagger} = \underset {\mathcal {S} ^ {\prime} \subset \mathcal {S} _ {o}: | \mathcal {S} ^ {\prime} | = b} {\operatorname {a r g m i n}} \left\{\max  _ {\boldsymbol {s} \in \mathcal {S} _ {o}} \min  _ {\boldsymbol {s} ^ {\prime} \in \mathcal {S} ^ {\prime}} \left\{\left| \left| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right| \right| \right\} \right\}. \tag {18}
$$
Although the above problem is known to be NP-Complete, we find a solution using the farthest-point clustering algorithm, which is known to have an approximation factor equal to 2 and runs in polynomial time. The algorithm starts by adding one point from  $S_{o}$  to  $S_{\dagger}$  at random. Then, it proceeds iteratively and, at each iteration, it adds to  $S_{\dagger}$  the point from  $S_{o}$  that is the furthest from all points already in  $S_{\dagger}$ , i.e.,  $S_{\dagger} = S_{\dagger} \cup s$ , where  $s = \max_{s' \in S_{o}} \left\{ \min_{s_{\dagger} \in S_{\dagger}} || s' - s_{\dagger}|| \right\}$ . The algorithm terminates after  $b$  iterations.
Results. We compare the computational efficiency of our method under each of the above anchor set selection strategies for various values of the size of the anchor set  $|S_{\dagger}|$ . Figure 5 summarizes the results. We observe that the Facility-Location selection strategy performs rather poorly compared to the other two strategies, achieving an effective branching factor (EBF) higher than 3. In contrast, the MC-Lipschitz and MC-Uniform strategies achieve an EBF close to 2, which decreases rapidly as the size of the anchor set increases. Among these two strategies, the MC-Lipschitz strategy, which we use in our experiments in Section 5, achieves the lowest EBF.
Table 2: Levels of vasopressors and intravenous fluids corresponding to the 25 actions in  $\mathcal{A}$
Vasopressors (mcg/kg/min)	Intravenous fluids (mL/4 hours)
0.00	0
0.04	30
0.113	80
0.225	279
0.788	850
G Additional details on the experimental setup
G.1 Features and actions in the sepsis management dataset
As mentioned in Section 5, our state space is  $S = \mathbb{R}^D$ , where  $D = 13$  is the number of features. We distinguish between three types of features: (i) demographic features, whose values remain constant across time, (ii) contextual features, for which we maintain their observed (and potentially varying) values throughout all counterfactual episodes and, (iii) time-varying features, whose counterfactual values are given by the SCM  $\mathcal{C}$ . The list of features is as follows:
- Gender (demographic)
- Re-admission (demographic)
Age (demographic)
- Mechanical ventilation (contextual)
-  $\mathrm{FiO}_2$  (time-varying)
$\mathrm{PaO}_2$  (time-varying)
- Platelet count (time-varying)
- Bilirubin (time-varying)
- Glasgow Coma Scale (time-varying)
- Mean arterial blood pressure (time-varying)
- Creatinine (time-varying)
- Urine output (time-varying)
- SOFA score (time-varying)
To define our set of actions  $\mathcal{A}$  we follow related work [6, 55, 57], and we consider 25 actions corresponding to  $5 \times 5$  levels of administered vasopressors and intravenous fluids. Specifically, for both vasopressors and fluids, we find all non-zero values appearing in the data, and we divide them into 4 intervals based on the quartiles of the observed values. Then, we set the 5 levels to be the median values of the 4 intervals and 0. Table G.1 shows the resulting values of vasopressors and fluids.
G.2 Additional details on the network architecture & training
We represent the location and scale functions  $h$  and  $\phi$  of the SCM  $\mathcal{C}$  using neural networks with 1 hidden layer, 200 hidden units and tanh activation functions. The mapping from a state  $s$  and an action  $a$  to the hidden vector  $z$  takes the form  $z = \tanh(W_s s + W_a a)$ , where  $a$  is a 2-D vector representation of the respective action. The mapping from the hidden vector  $z$  to the network's output is done via a fully connected layer with weights  $W_z$ . To enforce a network to have a Lipschitz constant  $L$  with respect to the state input, we apply spectral normalization to the weight matrices  $W_s$  and  $W_z$ , so that their spectral norms are  $\| W_s \|_2 = \| W_z \|_2 = 1$ . Additionally, we add 2 intermediate layers between the input and the hidden layer and between the hidden layer and the output layer, each one multiplying its respective input by a constant  $\sqrt{L}$ . Since it is known that the tanh activation function has a Lipschitz constant of 1, it is easy to see that, by function composition, the resulting network is guaranteed to be Lipschitz continuous with respect to its state input with constant  $L$ . Note
Figure 6: Goodness of fit of the Lipschitz-continuous SCM  $\mathcal{C}$ , measured by means of the percentage decrease in log likelihood of the data in comparison with an SCM trained without Lipschitz-continuity constraints. The x and y axes correspond to different enforced values for the Lipschitz constants  $L_{h}$ ,  $L_{\phi}$  of the location and scale networks  $h$  and  $\phi$ , respectively. Darker values indicate that the learned SCM achieves a significantly lower log likelihood than the unconstrained SCM.
that, since the matrix  $W_{a}$  is not normalized, the network's Lipschitz constant with respect to the action input can be arbitrary.
To train the SCM  $\mathcal{C}$ , for each sample, we compute the negative log-likelihood of the observed transition under the SCM's current parameter values (i.e., network weight matrices & covariance matrix of the multivariate Gaussian prior), and we use that as a loss. Subsequently, we optimize those parameters using the Adam optimizer with a learning rate of 0.001, a batch size of 256, and we train the model for 100 epochs.
We train the model under multiple values of the Lipschitz constants  $L_{h}$ ,  $L_{\phi}$  of the location and scale networks, and we evaluate the log-likelihood of the data under each model using 5-fold cross-validation. Specifically, for each configuration of  $L_{h}$  and  $L_{\phi}$ , we randomly split the dataset into a training and a validation set (with a size ratio 4-to-1), we train the corresponding SCM using the training set, and we evaluate the log-likelihood of the validation set based on the trained SCM. This results in the log-likelihood always being measured on a different set of data points than the one used for training. For each configuration of  $L_{h}$  and  $L_{\phi}$ , we repeat the aforementioned procedure 5 times and we report the average log-likelihood achieved on the validation set. In addition, we train an unconstrained model without spectral normalization, which can have an arbitrary Lipschitz constant.
Figure 6 shows the decrease in log-likelihood of the respective constrained model as a percentage of the log-likelihood achieved by the unconstrained model, under various values of the Lipschitz constants  $L_{h}$ ,  $L_{\phi}$ . We observe that, the model's performance is predominantly affected by the Lipschitz constant of the location network  $L_{h}$ , and its effect is more pronounced when  $L_{h}$  takes values smaller than 1. Additionally, we can see that the scale network's Lipschitz constant  $L_{\phi}$  has a milder effect on performance, especially when  $L_{h}$  is greater or equal than 1. Since we are interested in constraining the overall Lipschitz constant of the SCM  $\mathcal{C}$ , in our experiments in Section 5, we set  $L_{h} = 1$  and  $L_{\phi} = 0.1$ , which achieves a log-likelihood only 6% lower to that of the best model trained without any Lipschitz constraint.