SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchao Qin
Universitvity of Cambridge, UK
Alan Turing Institute, UK
Changhee Lee
Chung-Ang University, South Korea
Timely outcome prediction is essential in healthcare to enable early detection and intervention of adverse events. However, in longitudinal follow-ups to patients' health status, cost-efficient acquisition of patient covariates is usually necessary due to the significant expense involved in screening and lab tests. To balance the timely and accurate outcome predictions with acquisition costs, an effective active sensing strategy is crucial. In this paper, we propose a novel risk-averse active sensing approach RAS that addresses the composite decision problem of when to conduct the acquisition and which measurements to make. Our approach decomposes the policy into two sub-policies: acquisition scheduler and feature selector, respectively. Moreover, we introduce a novel risk-aversion training strategy to focus on the underrepresented subgroup of high-risk patients for whom timely and accurate prediction of disease progression is of greater value. Our method outperforms baseline active sensing approaches in experiments with both synthetic and real-world datasets, and we illustrate the significance of our policy decomposition and the necessity of a risk-averse sensing policy through case studies.
Timely decision-making through accumulated observation history has attracted significant attention in the machine learning community, with broad impact and applications in healthcare [8]. Consider a typical decision-making scenario in which an agent employs a sensing policy to actively collect diagnosis-related information from an underlying feature trajectory. In the presence of cost pressure, the goal of the agent is to achieve timely and accurate predictions of a time-varying outcome of interest based on the sensing history, i.e., feature observations accumulated over time, while maintaining a reasonable expense of feature acquisition. At each decision step, we refer to the problem of deciding when to make new acquisitions and which features to measure as active sensing [2, 24]. The optimal sensing policy can be achieved by negotiating the subjective trade-off between outcome prediction (accuracy and timeliness) and acquisition cost over time.
A special class of the active sensing problem has been extensively explored in the optimal stopping literature [2, 6, 15]. In optimal stopping, the agent focuses on the timely diagnosis of a terminal status, where observation ceases immediately after a confident diagnosis can be made [1, 5]. Thereby, the acquisition cost in optimal stopping is typically considered as the time span of the information collection process. Recently, Jarrett and van der Schaar [9] introduced a novel sensing framework to allow for the consideration of measurement cost of individual feature variables in optimal stopping, enabling better scheduling of the acquisition sequences. However, their approach still focuses on
a static terminal status and is unsuitable for general purpose active sensing where the outcome of interest varies over time.
(a) Active sensing process
(b) Long-tailed sensing deficiency distribution
Figure 1: Overview of active sensing strategies. Active sensing in the dynamical setting (a) involves two distinct decisions on: what features to measure; and when to perform next acquisition. The trade-off between diagnosis accuracy and acquisition cost usually yields a (b) long-tailed sensing deficiency distribution. The sensing deficiency  $Q^{\pi}(\mathbf{X})$  accumulates the weighted sum of the diagnosis error and measurement cost over time. Higher sensing deficiency values indicate failures in the sensing process.
In this paper, we focus on more general healthcare scenarios where timely diagnosis and intervention need to be achieved through active sensing under certain budget. The budget serves as the source of cost pressure and prioritize safe and low-cost measurements in the sensing process. For instance, the management of Alzheimer's disease (AD) requires continuous monitoring and staging of disease progression, which is crucial for early detection and intervention of high-risk patients [8]. However, certain measurements involved in diagnosing AD are notably expensive. A typical example is positron emission tomography (PET) scans. PET scans provide precise information about AD-associated proteins in the brain and are widely used for monitoring AD progression [3]. Nevertheless, the high accuracy of PET scans compared to cognitive tests is accompanied by greater financial costs as well as potentially harmful exposure to radiation [7]. A desirable active sensing policy should adaptively balance measurement costs and diagnostic accuracy at different disease stages to provide timely diagnosis and intervention under cost pressure. Such property is not only useful in healthcare but also other applications such as delay-sensitive wireless communication [14].
Another crucial but relatively unexplored aspect of active sensing is the consideration of tail risk. To illustrate this concept, let's examine the prediction of AD progression. In active sensing, the trade-off between diagnosis error and acquisition cost is usually achieved through the minimization of their weighted sum [9, 23, 24]. As depicted in Figure 1(b), the weighted objective  $(Q^{\pi}(\mathbf{X}))$  typically exhibits a long-tailed distribution with a notable concentration of high-value patients at the tail side. We consider these patients to be of higher value in the sensing task for two main reasons. Firstly, accurate diagnosis of many high-risk patients heavily relies on biomarkers obtained through expensive PET scans such as fluorodeoxyglucose (FDG) PET and florbetapir (AV45) PET [7, 10]. A vanilla sensing policy optimized for the entire population would easily fail to arrange enough PET scans for these high-risk patients due to the high costs, leading to degraded sensing performance (the long tail as shown in Figure 1(b)). In the meantime, the majority of patients tend to remain in stable conditions. Their disease stages can usually be diagnosed even with very sparse acquisitions over time. A biased sensing policy may overemphasize the importance of high-risk patients and order frequent PET scans for all patients to avoid false negatives in its diagnosis, which contributes to the long-tailed distribution from another perspective. To avoid such situations, an active sensing strategy must be risk-averse such that it can effectively cater to the needs of both the majority of stable patients and the high-risk subgroups requiring frequent follow-ups, ensuring timely and accurate detection of disease progression under cost pressure.
Contributions. In this paper, we propose a novel continuous-time active sensing approach to address the aforementioned desiderata. Our sensing approach adaptively balances measurement costs and diagnostic accuracy by factorizing sensing policy into two sub-policies: acquisition scheduler which
determines when to conduct feature acquisitions (i.e., the time interval between adjacent follow-ups), and feature selector that decides which observations to make. Furthermore, our sensing approach adopts the risk-averse strategy and focuses on the trade-offs happen at the long tail, where accurate diagnosis becomes particularly crucial. By dynamically identifying patient trajectories located at the tail side, we optimize our sensing strategy using the conditional value-at-risk (CVaR) loss to provide guarantees on timely and accurate prediction for high-risk patients while avoiding biased decisions for patients with stable conditions.
We validate our approach through experiments on synthetic and real-world datasets, where the sensing performance on high-risk patients is significantly impacted by their relatively small cohort size. With the consideration of acquisition costs, our method is able to adaptively identify the patient subgroups at the long tail of sensing deficiency distribution and achieves improved diagnostic performance compared to the state-of-the-art active sensing benchmarks.
2 Active Sensing in Practice
In this section, with a focus on healthcare applications, we introduce a new active sensing approach, which we refer to as Risk-averse Active Sensing (RAS), to solve the decision-making problem of when to conduct feature acquisitions and which features to observe considering prediction accuracy, timeliness, and cost pressure. We will start by providing a general framework for active sensing problems.
2.1 Preliminaries: Active Sensing
Notation. Let  $\pmb{x}(t) \in \mathbb{R}^d$  be the characteristic of a patient at time  $t \in [0, T]$  which is a collection of  $d$  (costly) observable time-varying covariates about an underlying disease progression of our interest.  $\pmb{X}_t = \{\pmb{x}(\tau) | \tau \in [0, t]\}$  represents a temporal trajectory of observable features until time  $t \leq T$ . For ease of description, we will occasionally abuse the notation and use  $\pmb{X}$  to denote the entire patient trajectory  $\pmb{X}_T$ . Let  $y_t \in \mathcal{V}$  be the outcome of our interest which represents the underlying health status of a patient (e.g., diagnosis, adverse outcome, etc.) at time  $t$ . We assume the joint distribution  $p(\pmb{X}_t, y_t)$  from which the trajectory  $\pmb{X}_t$  and the outcome  $y_t$  are drawn. Hereafter, we focus our description on  $K$ -class classification tasks, i.e.,  $\mathcal{V} = \{1, 2, \dots, K\}$ .
We denote an active sensing policy as  $\pi$ . In the sequential outcome prediction task, for each patient  $X$ , policy  $\pi$  arranges a set of follow-up times  $\{t_1, t_2, \ldots, t_I\} \subset [0, T]$  and determines the subsets of features  $m_i$  to be measured at each time step  $t_i$ , where  $m_i$  is a binary vector. The collected observations of patient trajectory  $X$  until time  $t$  form a sensing history  $X_t^\pi$ . It is worth noting that  $X_t^\pi$  is piece-wise constant over time as it is only updated at discrete follow-ups. At each step, the sensing policy  $\pi$  shall generate a diagnosis of the patient by approximating the conditional  $p(y_t | X_t^\pi)$ .
Dataset. For the active sensing task, we assume access to an observational dataset  $\mathcal{D}$  containing electrical health records (EHRs) of  $N$  patients, i.e.,  $\mathcal{D} = \{X_{n} = (t_{i},\boldsymbol {x}(t_{i}),y_{t_{i}})_{i = 1}^{L}\}_{n = 1}^{N}$ . The record of each patient consists observations collected from  $L$  follow-ups. It is worth noting that the record  $X_{n}\in \mathcal{D}$  is equivalent to a sensing history.
Following the convention in active sensing literature [9, 24], we consider the cost of measurement on each time-varying observable feature to be time-invariant and thus can be represented as a cost vector  $\pmb{c}$ , where the  $j$ -th element, denoted as  $(c)_j$ , is the cost for measuring the  $j$ -th time-varying feature in vector  $\pmb{x} \in \pmb{X}$ . The central goal of active sensing is then to balance between the outcome estimation accuracy based on sensing history  $X_{t}^{\pi}$  and the accumulated acquisition costs over time.
Challenges. In practice, there are two major challenges that confound the active sensing problem:
1. Adaptive follow-up intervals. Most existing active sensing policies assume a constant follow-up interval. However, in real-world healthcare settings, the desirable acquisition intervals usually change along with disease progression. Sensing policy with constant decision intervals creates redundant follow-ups and may induce extra costs for feature acquisition.
<sup>1</sup>In general, observation of a patient's underlying disease progression ends at some time  $T > 0$  due to death or lost to follow-up.
2. Failures at the long tail. Vanilla active sensing policies are optimized for the entire population. For diseases like AD, the imbalanced proportion of high-risk and stable patients may lead to over-conservative sensing policies that make no observations. Similarly, sensing policies targeted on high-risk patients may generate biased decision with higher acquisition costs for patients in stable conditions. Both situations lead to failures for samples at the long tail of sensing deficiency distribution.
In the following, we first introduce a generalized sensing policy  $\pi$  with the decomposition of decisions on follow-up interval and feature selection. Then, we formulate the active sensing task into a risk-averse optimization problem, the solution to which addresses the second challenge.
2.2 Decomposing the Active Sensing Policy
Vast research in active sensing has assumed a constant follow-up interval between two consecutive patient visits and has primarily focused on the sequential feature selection problem, determining which feature variables to measure [6, 24]. However, considering the dynamic and complex nature of disease progression, sensing policies with dynamic follow-up intervals are broadly desired in real-world applications. In this paper, we propose a generalized formulation of the sensing policy, i.e.,  $\pi = (\pi_{m},\pi_{\Delta})$ , where the acquisition scheduler  $\pi_{\Delta}$  determines the time for next follow-up, the feature selector  $\pi_{m}$  determines which features to measure during the subsequent visit.
Consider the  $i$ -th follow-up of patient  $\mathbf{X}$  at time  $t_i$ . The next follow-up time  $t_{i+1}$  is firstly determined by acquisition scheduler  $\pi_{\Delta}$  as  $t_{i+1} = t_i + \Delta_i, \Delta_i \sim \pi_{\Delta}(X_{t_i}^{\pi})$ . Then, at time  $t_{i+1}$ , the feature selector  $\pi_m$  determines which subset of patient covariates to measure using a binary vector  $\pmb{m}_{i+1} \sim \pi_m(X_{t_i}^{\pi}), \pmb{m}_{i+1} \in \{0,1\}^d$ . The new observation is calculated as  $\pmb{m}_{i+1} \odot \pmb{x}(t_{i+1})$ , where the  $j$ -th element
$$
(\boldsymbol {m} _ {i + 1} \odot \boldsymbol {x} (t _ {i + 1})) _ {j} = \left\{ \begin{array}{c l} (\boldsymbol {x} (t _ {i + 1})) _ {j} & \text {i f} (\boldsymbol {m} _ {i + 1}) _ {j} = 1, \\ * & \text {o t h e r w i s e}. \end{array} \right.
$$
2.3 Towards Risk-Averse Sensing Policy
Trade-off between accuracy and costs. Consider the  $i$ -th follow-up at time  $t_i$ . The feature selection is determined by  $\pmb{m}_i \sim \pi_m(\tilde{X}_{t_{i-1}}^\pi)$ . The acquisition cost is evaluated as  $r_m(\pmb{m}_i) = \pmb{c}^\top \pmb{m}_i$ , where  $\pmb{c}$  is the cost vector. The updated sensing history  $X_{t_i}^\pi = X_{t_{i-1}}^\pi \cup \{\pmb{m}_i \odot \pmb{x}(t_i)\}$  is then used to determine the acquisition interval  $t_{i+1} - t_i = \Delta_i \sim \pi_\Delta(X_{t_i}^\pi)$ . Assume a diagnosis error function  $r_y(X_{t_i}^\pi, \Delta_i)$  that evaluates the mismatch between  $p(y_t | X_t)$  and  $p(y_t | X_t^\pi)$  during the interval  $\Delta_i$ . The trade-off between diagnosis accuracy and measurement costs of the  $i$ -th interval is defined as the following reward signal
$$
r \left(X _ {t _ {i}} ^ {\pi}, \boldsymbol {m} _ {i}, \Delta_ {i}\right) = r _ {m} (\boldsymbol {m} _ {i}) + \lambda r _ {y} \left(X _ {t _ {i}} ^ {\pi}, \Delta_ {i}\right) \tag {1}
$$
where  $\lambda > 0$  is a coefficient chosen to balance the two terms.
Sensing deficiency. We define the sensing deficiency of policy  $\pi$  along a patient trajectory  $X$  as the expected cumulative reward as shown in (2).
$$
Q ^ {\pi} (\boldsymbol {X}) = \mathbb {E} _ {t _ {1}, t _ {2}, \dots , t _ {I}, \boldsymbol {m} _ {1}, \boldsymbol {m} _ {2}, \dots , \boldsymbol {m} _ {I} \sim \pi} \left[ \sum_ {i = 1} ^ {I} \gamma^ {i - 1} r \left(X _ {t _ {i}} ^ {\pi}, \boldsymbol {m} _ {i}, \Delta_ {i}\right) \right], \tag {2}
$$
where discount factor  $\gamma \in (0,1]$  is used to tackle long trajectories. In this paper, we take  $\gamma = 0.99$ .
Trajectories at the long tail. As mentioned earlier, one major challenge for active sensing in practice is the potential sensing failures for patients at the tail of sensing deficiency distribution. Since the distribution tail of  $Q^{\pi}(\mathbf{X})$  changes along with the updates of policy  $\pi$ , in this paper, we propose to optimize the sensing policy over a dynamic subset of patient trajectories located at the long tail. These tail trajectories can be identified through the upper  $\alpha$  quantile  $\rho_{\alpha}$ , where  $\mathbb{E}_{\mathbf{X}}[\mathbb{I}(Q^{\pi}(\mathbf{X}) \geq \rho_{\alpha})] = \alpha$ . Assume the marginal trajectory distribution  $p(\mathbf{X})$ , we denote with  $S_{\alpha}^{\pi} = \{ \mathbf{X} | \mathbf{X} \sim p(\mathbf{X}), Q^{\pi}(\mathbf{X}) \geq \rho_{\alpha} \}$  the set of tail trajectories under policy  $\pi$ .
Conditioned on the quantile factor  $\alpha \in [0,1]$ , the sensing deficiency of trajectories in  $S_{\alpha}^{\pi}$  is considered as the value at risk that needs to be improved. Based on the evaluation of CVaR in sensing deficiency distribution, we formulate our active sensing task as a risk-averse optimization problem.
$$
\underset {\pi} {\text {m i n i m i z e}} \mathrm {C V a R} _ {\alpha} \triangleq \mathbb {E} _ {\boldsymbol {X} \in S _ {\alpha} ^ {\pi}} [ Q ^ {\pi} (\boldsymbol {X}) ]. \tag {3}
$$
Solutions to problem (3) are considered risk-averse since they reduce the risk of sensing failures in the worst scenarios  $(S_{\alpha}^{\pi})$ . Our proposed risk-averse sensing problem has two major distinctions from the class of robust adversarial reinforcement learning (RL) problems (e.g., [18]): i) we directly optimize the value at risk  $\mathrm{CVaR}_{\alpha}$ ; and ii) the adversarial agent is replaced by the dynamic tail trajectory subset  $S_{\alpha}^{\pi}$ . As such, our proposed problem requires no online interactions with the data generation process and thus can avoid the sensing policy  $\pi$  from being biased toward unrealistic or out-of-distribution trajectories generated by simulators.
3 Method: Risk-Averse Active Sensing
Solution to the risk-averse active sensing problem in (3) given observational data is faced with three unique challenges. First, we only have access to observational data that consist of discrete measurements of (continuous) patient trajectories. Second, we need access to a baseline conditional outcome estimator,  $f_{P}(y_{t}|\mathbf{X}_{t})$ , to estimate the diagnosis error  $r_{y}$  in (1). Third, the solution of (3) heavily relies on the appropriate construction of the subset,  $S_{\alpha}^{\pi}$ , which, by definition, depends on the sensing policy itself.
Handling discrete and sparse observations. Dataset  $\mathcal{D}$  only contains discrete observations of patient trajectories. However, the adaptive acquisition intervals generated by sensing policy  $\pi$  may require measurements of the underlying patient trajectories at time points not included in  $\mathcal{D}$ . To solve this problem, we adopt a linear interpolator  $\mathcal{L}$  as a proxy to obtain estimations of these values. In the meantime, policy  $\pi$  by nature generates sparse sensing history with partial observations of patient trajectories. To properly encode the missingness and adaptive observation intervals in sensing history  $X_{t}^{\pi}$ , we include the neural CDE as the base encoders in our approach. As enlightened in [11], CDE is a continuous-time generalization to recurrent neural network (RNN) and offers a natural and elegant approach to encode the information in sparse and irregular measurements of a patient trajectory.
Specifically, given a discrete sensing history  $X_{t}^{\pi}$ , we first convert it into a continuous trajectory  $\pmb{u}(\tau) = \mathcal{I}(\tau, X_{t}^{\pi})$  via the interpolator  $\mathcal{I}$ . Then, CDE encodes the temporal changes in  $X_{t}^{\pi}$  as the dynamics of a latent variable  $\pmb{z}(t) \in \mathbb{R}^{l}$  via
$$
\boldsymbol {z} (t) = \boldsymbol {z} \left(t _ {0}\right) + \int_ {t _ {0}} ^ {t} f _ {\phi} (\boldsymbol {z} (\tau)) \mathrm {d} \boldsymbol {u} (\tau), \text {f o r} \tau \in [ t _ {0}, t ], \tag {4}
$$
where  $t_0 \geq 0$ ,  $f_{\phi} : \mathbb{R}^l \mapsto \mathbb{R}^{l \times d}$  is a map with learnable parameters  $\phi$ , initial state  $\mathbf{z}_0 = \mathbf{z}(t = 0)$  could be manually specified or learned from data.
Timely and accurate predictions. In this paper, we build CDE-based outcome estimator  $f_{P}(t,X)$  to approximate the density  $p(y_t|X_t)$ , where  $X$  is a sensing history of trajectory  $X$ . Details about  $f_{P}$  can be found in the Appendix. Given a patient with record  $X \in \mathcal{D}$ , to provide timely and accurate predictions about the underlying disease progression, our sensing policy aims to minimize the mismatch between baseline estimate  $f_{P}(\tau ,X)$  and sensing history-based prediction  $f_{P}(\tau ,X_{\tau}^{\pi})$ . For this purpose, in time horizon  $\tau \in [t_i,t_{i + 1})$ , the diagnosis error function for the  $i$ -th follow-ups is computed as the cumulative mismatch:
$$
r _ {y} \left(X _ {t _ {i}} ^ {\pi}, \Delta_ {i}\right) \triangleq \int_ {t _ {i}} ^ {t _ {i + 1} ^ {-}} D _ {\mathrm {J S}} \left(f _ {P} \left(\tau , X _ {t _ {i}} ^ {\pi}\right) \| f _ {P} (\tau , X)\right) \mathrm {d} \tau , \tag {5}
$$
where  $D_{\mathrm{JS}}$  is the JS divergence,  $t_{i + 1} = t_i + \Delta_i$
The distinction from previous works that utilize a single-step mismatch – such as at a pre-specified time interval [24] – is that (5) provides a precise proxy to evaluate how timely and accurate the predictions are by gauging the impact with respect to the acquisition interval  $\Delta_{i}$ , which is a desired property. Adverse clinical outcomes of the underlying disease progression are likely to occur when the outcome probability,  $p(y_{t}|X_{t})$ , hits a certain threshold, often accompanied by a sudden rise or fall. Making too early or too late detection of such a rise/fall based on the sensing history, i.e.,  $p(y_{t}|X_{t}^{\pi})$ , will drastically increase the cumulative mismatch in (5).
Penalty on invalid visits. We define the follow-up at time  $t_{i+1}$  as an invalid visit if no patient covariates are measured, i.e.,  $m_i = 0$ . Allowing invalid visits potentially yields degenerated active sensing policy and makes the policy decomposition meaningless. To prevent such situations, we
Figure 2: The model structure of RAS.
extend the acquisition cost function  $r_m$  as  $r_m(\pmb{m}) = \pmb{c}^\top \pmb{m} \cdot \mathbb{I}(\pmb{m} \neq \pmb{0}) + \nu \cdot \mathbb{I}(\pmb{m} = \pmb{0})$ , where  $\nu \geq c^\top \mathbf{1}$  penalizes the invalid visits scheduled by the sensing policy  $\pi$ .
Network structure. Similar to the predictor  $f_{P}$ , the neural CDE and interpolator  $\mathcal{I}$  are also employed in our proposed policy network  $\pi$  to construct an embedding of the discrete sensing history  $X_{t}^{\pi}$ . As illustrated in Figure 2, our sensing policy  $\pi$  (parameterized by  $\theta$ ) is constructed with the following three networks: