Towards Transparent Time Series Forecasting
Authors: Krzysztof Kacprzyk
Tennison Liu
Mihaela van der Schaar
Venue: ICLR 2024 poster (2024)
Keywords: transparency
interpretability
time series
Primary area: societal considerations including fairness, safety, privacy

Abstract: Transparent machine learning (ML) models are essential for ensuring interpretability and trustworthiness in decision-making systems, particularly in high-stakes domains such as healthcare, finance, and criminal justice. While transparent machine learning models have been proposed for classification and regression, time series forecasting presents some unique challenges for ensuring transparency. In particular, currently used bottom-up approaches that focus on the values of the time series at specific time points (usually regularly spaced) do not provide a holistic understanding of the entire time series. This limits the applicability of ML in many critical areas. To open up these domains for ML, we propose a top-down framework of bi-level transparency, which involves understanding the higher-level trends and the lower-level properties of the predicted time series. Applying this framework, we develop TIMEVIEW, a transparent ML model for time series forecasting based on static features, complemented with an interactive visualization tool. Through a series of experiments, we demonstrate the efficacy and interpretability of our approach, paving the way for more transparent and reliable applications of ML in various domains.

Published as a conference paper at ICLR 2024
TOWARDS TRANSPARENT TIME SERIES FORECASTING
Krzysztof Kacprzyk University of Cambridge kk75 c1mcam. ac. uk
Tennison Liu University of Cambridge t ac.uk uk
Mihaela van der Schaar University of Cambridge The Alan Turing Institute mv 4 72 cam. ac uk
ABSTRACT
Transparent machine learning (ML) models are essential for ensuring inter- pretability and trustworthiees in Decision-making systems, particularly in high- stakes domains such as healthcare, finance, and criminal justice. While transpar- ent machine learning models have been proposed for classification and regression, time series forecasting presents some unique challenges for ensuring transparency, In particular, currently used bottom-up approaches that focus the values of the time series at specific time points (usually regularly spaced) do not provide holistic understanding of the entire time series. This limits the applicability of ML in many critical areas. To open up these domains for ML, we propose a top-down framework of bi-level transparency, which involves understanding the higher-level trends and the lower-level properties of the predicted time series. Ap- plying this framework, we develop TIMEVIEW, transparent ML model for time series forecasting based on static features, complemented with an interactive visu- alization tool. Through a series of experiments, we demonstrate the efficacy and interpretability of our approach, paving the way for more transparent and reliable applications of ML in various domains.
INTRODUCTION
Why do we need transparent models? eXplainable Artificial Intelligence (XAI) methods are broadly divided into transparent models and post-hoc explanation techniques (Barredo Arrieta al.] 2020). Transparent (also called glass box) models are crucial in many settings involving high-stakes decisions such as healthcare or credit scoring (Rudin) 2019). As these models are interpretable by design, they are by themselves understandable. Such understanding, apart from being mandated by certain regulatory bodies (Goodman & Flaxman 2017). needed, for instance, to improve ro- bustness, detect biases, evoke trust, or certily model compliance with legislation (Barredo Arrietal jet al., 2020). Many transparent machine learning models that issue static predictions have been proposed. That includes Linear/Logistic Regression, Generalized Additive Models (Hastie & Tib- shirani 1986), and Decision Trees (Hu et al. 2019). By definition, these methods are not directly applicable to time series forecasting-win when we want predict whole trajectory rather than single label. Although they can be adapted, they exhibit poor performance (see Section 7
Challenges of time series forecasting: limitations of a bottom-up approach. contrast single-label output (as in classification or regression), understanding the change in the trajectory is more complicated as it is an entire function (described by many values). As time series forecasting remains an under-studied field of XAI (Barredo Arrieta et al. 2020). current techniques usually resolve to bottom-up approach. This means they focus on the values of the trajectory individual time points (usually regularly spaced). For instance, the importance scores in saliency methods are calculated for different prediction horizons (Leung et al. (023). This may be sufficient when we interested in particular time point (c.g., 5-year survival rate), but we often want comprehend the whole trajectory at once. For instance, when administering a drug, we may be less interested in the concentration of the drug every few hours but rather in understanding the entire curve, including properties like the peak plasma concentration and the time when it is achieved (Han et al.) 2018).
Bi-level transparency for time series forecasting: top-down approach. We propose top-down approach to trajectory comprehension and consequently two levels of transparency for time series forecasting: (level 1) understanding how the trend (the general shape of the trajectory) changes

---

Published as a conference paper at ICLR 2024
modify the input, and (level 2) understanding how the properties of the current trend (e.g., minimum value) change as we modify the input. To illustrate this, let us consider the following example.
Forecasting a tumor volume trajectory from patient's baseline covariates and drug dose
Understanding model like this may include answering questions such as: (Liao al. 2020
What if: "What would happen the model's prediction if specific covariate changes?"
How to be that: "How should the covariates be modified to get different prediction?"
How to still be this: "What range of drug dose values keeps the prediction the same?"
We characterize the difference between predicted trajectories two levels, which enables us to answer concrete questions about each level such as
Level (trends)
Level 2 (properties)
"Would the predicted tumor volume keep decreasing if we adjusted the treatment?"
"What feature changes would lower the minimum tumor volume?"
We explain trends and properties in detail in Section and then formalize them in Section It is worth noting that answering such questions with the current bottom-up approaches may often be futile since the notion of "different prediction" (based on individual time points or norms such as (P) may be non-interpretable (see Section 2 or simplistic. We demonstrate how our framework can enable answering such questions Figure and more thoroughly in Appendix E.
Time series forecasting based on static features. Our work focuses on understanding the change in the predicted trajectory. However, time series models take many types of inputs, including time series and exogenous features. To provide clear exposition of our framework, we focus one spe- cific input type: static features. Time series forecasting based on static features has applications in many domains ranging from finance through medicine and pharmacology physics (see Section B). In Section 5 we introduce TIMEVIEW a transparent ML model for time series forecasting based on static features. As with many transparent models (e.g., GAMs, decision trees), model visual- ization is crucial for its interpretability. In Section 7 we demonstrate a visualization tool based on interactive plots that allows for the examination of both the higher and lower-level features of the predicted trajectories and how they change based on the input (Figure I
Level 1 (trends)
Level (properties)
Predicted tumor volume
Trectton point
wiigh
0.45
0.40
0.30
0.24
0.30
0.25
0.35
0.60
0.45
0.50
feature
0.20
0.0
0.2
0.4
0.8
1.0
y or coordinate
Transition points
Figure 1: Snapshot of our dynamical visualization of TIMEVIEW. Our model adheres to bi-level transparency- top-down approach that focuses on the trend of the trajectory (Level 1) and the properties of the particular trend, e.g.. transition points (Level 2). The left panel shows how the trajectory trend changes when one of the features is perturbed. For instance, the tumor will increase if we lower the dose and decay if we increase the dose. The right panel investigates the position (g-coordinate) of the second transition point (local minimum) as the initial tumor volume changes.
Contributions. We introduce bi-level transparency, a novel top-down framework for time series forecasting that allows for holistic understanding of the entire trajectory through trends and prop- erties (Section 2). We formalize it by introducing the notions of motifss and compositions (Section Based on the new formalism, we develop TIMEVIEW, Time series Interpretable Model with Effec- tive VIsualization (5) We demonstrate how its visualization aids in model comprehension while exhibiting only minor performance drop compared to black-box models (Section
2

---

Published as a conference paper at ICLR 2024
2 TRANSPARENCY FOR TIME SERIES FORECASTING
2.1 SETUP
Time series forecasting. A general ML model is function f mapping samples from the input space X to the output space y. We say that f issues static predictions when ) is a subset of R (a regression model) a finite set of labels {1.... .I...... K} (a classification model) In contrast, we define f to be a time series forecasting model (or just forecasting model) when ) is a space of trajectories. A trajectory is function y T R. where T R is set of time points. A Although the conceptual framework in Section is is agnostic to the nature of TT our work focuses on settings where T is an interval [0, T] C. R. where T R is a time horizon, and the underlying trajectory is continuous. Note, in practice, we only observe discrete samples of y, which may be irregular and noisy.
Transparency. We assume the following general definition of transparency: ML model transpar- ent if we can understand the impact of the inputs on the prediction. In particular, how changing one (or few) of the features would impact the output. This crucial for counterfactual reasoning (e.g., "What would the model predict if the patient was female?"" or detecting anomalies (c.g., "Why does the model assign significantly higher risk score if the patient's s age is changed from 64 to 65?").
Comprehending the change in the output. As discussed in Section I understanding the change in the output crucial for answering important questions about the model (e.g., "What would happen to the model's prediction if a specific feature changes?"). In a static setting, when the prediction a single label, understanding the change in the output is relatively straightforward as only a few things happen. In regression, the target variable can decrease, increase, remain constant. In classification, the target variable can change from one option to another among a finite number of classes. In time series forecasting, when the prediction is trajectory, understanding the change in the output is challenging because there are numerous ways a function can change (discussed further in Appendix E Morcover, these changes need to be interpretable for humans.
2.2 BOTTOM-UP: CURRENT XAI APPROACH TO TRAJECTORY COMPREHENSION
As trajectory is function y T + R. current XAI techniques for time series forecasting focus on understanding the impact of the inputs on y(t for a particular E T. For instance, the values in saliency methods are calculated independently for different prediction horizons (Leung et al., 2023 (and might be later aggregated). Inspired by the motivations of rough path theory (Lyons) 2014 Fermanian et al. 2023). we call the current comprehending strategy bottom-up. It means the trajectory is understood by looking at its values at individual time points, and subsequently, more information is gained by looking more points" However, argue that this strategy for trajectory understanding is not optimal in many scenarios. In particular, it is not a natural way for people to understand trajectories, and it is challenging convey time-varying trends and global features by simply looking at individual time points in isolation.
Inconsistent with the natural way people understand trajectories. Standard representation of trajectory [0, + R is a line graph. Research on graph comprehension (Zacks & Tversky) 1999 Xi 2010) suggests that people understand line graphs in terms of trends rather than individual values. For instance, "when x increases, y also increases". They also tend to focus on the minimum and maximum values and trend reversals (Carswell al. 1993). Thus, understanding a (continuous) trajectory by individual values is unnatural for humans. See Appendix E for more details.
Increased cognitive load. As mentioned above, the bottom-up approach requires an increasing number of values to understand the trajectory better. This becomes problematic when we want to understand any change in the trajectory, as it places the cognitive burden on the human interpreter to piece together changes in trends from changes at individual time steps.
Unsuitable for global features. bottom-up approach may be sufficient when we are interested in particular time point (e.g., 5-year survival rate) when there are only a few time points of interest. However, we often want to comprehend the whole trajectory at once, For instance, when administering drug, we are interested in understanding the entire drug concentration curve, includ- ing properties like peak plasma concentration and the time when it is achieved (Han et al. 2018
'Another example of a model issuing static predictions is a multi-output regression where y' c R Note, "hottom-up" refers how the trajectory is comprehended, not how the prediction is generated
3

---

Published as a conference paper at ICLR 2024
2.3 TOP-DOWN: NEW APPROACH TO TRAJECTORY COMPREHENSION
To address the shortcomings of the bottom-up approach, we propose a top-down approach to un- derstanding trajectory. It motivated by the fact that humans tend describe trajectories by referring to the trends and properlies it exhibits rather than just the values it attains (Carswell et al.) 1993j. Consider the natural language descriptions of trajectories presented in Table In all these examples, we have a trend-the general shape of the function (c.g., "increasing", "stays below"), and properties- the details the particular trend (c.g., "for the last 10 years", "below 100mg/dl").
Table 1: When we describe trajectory, often refer the trends and properties it exhibits. We this observation in our definition bi-level transparency, which ultimately informs the design of our method. The table shows three examples of descriptions of trajectories with their corresponding trends and properties.
Description
Trend
Properties
Visualization
"The GDP has been steadily increasing for the last 10 years"
for the last 10 years
increasing
"The blood sugar level in non-diabetic patients should stay below 100mg/dl while fasting"
below
stay below
100mg/dl
"Tumor volume decreases, obtains minimum after 6 months, and then increases"
decreases increases
then
minimum at months
The top-down approach addresses shortcomings of the bottom-up approach, i.e., it is more consis- tent with the natural way people understand trajectories and conveys time-varying trends and global features in an interpretable way. Moreover, it is also compatible with the scientific approach to ana- lyzing various trajectories. For instance, while studying dynamical systems, we are often interested in understanding bifurcations qualitative change in the behavior of system as the parameter changes (Blanchard et al. 2012). This corresponds to understanding the inputs where the trend of the trajectory changes.
understanding how the trends and properties change. By using the top- down approach above, we do not need all the trajectory values to understand it. Instead, we can focus on the trends and properties of the trajectory and only access the exact values when necessary. This is how we can achieve interpretable model: instead of tracking the individual values of the trajectory (as in bottom-up up approaches), we track how the trends and properties of the trajectory change as we vary the input. Thus, we refine the definition of transparency and adapt it specifically for time series forecasting. We call it bi-level transparency.
Bi-level transparency:
A time series forecasting model is (bi-level) transparent if the following holds.
(Level 1) We can understand the impact of the input on the trends of the trajectory. (Level 2) We can understand the impact of the input on the properties of a given trend.
TIME SERIES FORECASTING FROM STATIC FEATURES
Bi-level transparency unweaves the "output" part of transparency into two separate objects: trends and properties. Thus, provides concrete answer the question: what does it mean understand the change of the outpur? However, time series models may take many types of inputs, including static features, information about the future (e.g., upcoming holiday dates), and other exogenous time series (Lim et al. 2021). To provide a clear exposition of our framework, develop formalism, and demonstrate a practical implementation, we focus on settings where inputs are statie features.
Real life settings. Time series forecasting from static features is frequently encountered in medicine and pharmacology, where we are interested in predicting the disease progression or the drug concen- tration based on the patient's covariates. Static features can also include the dosage/strength of the treatment or even the time and type of intervention. If necessary, one or a few initial observations pre-speee pre-specified times can also be considered to be static features. More examples of such scenarios can be found in finance (predicting stock values from the company's s static data), time-to-event prob- lems (predicting the survival or the hazard function), or modeling any 1D dynamical system from
4

---

Published as a conference paper at ICLR 2024
its initial conditions. In some scientific engineering domains, time can be even replaced by other continuous variables. For instance, when modeling stress-strain or current-voltage curves.
Problem formulation. Let T' R be the time horizon. Each sample consists of static features a(()) E IR where M E N is the number of features, and discretely sampled trajectory y R Na at times t(d) Ne, where Na E N is the number of measurements for the dt sample. We assume that (dd)) consists of noisy samples of some true underlying continuous trajectory y!") (0, T R. Given dataset {xx(4) (d)) g(d) ) d== the task is to find model that matches static covariates æ RM to a trajectory ý [0, + I such that ý minimizes the expected value of T J (0(()~ (()()))(())) (t) for all test samples. We denote the class of predicted trajectories V.
MOTIFS AND COMPOSITIONS
In this section, we propose way to formalize the notion of a trend by defining the composition of trajectory. The composition is a sequence motifs where each motif describes the current "shape" of the trajectory at specific interval. For instance, we can choose a set of three motifs: "increasing" (i), "decreasing" " (d), and "constant" (c). Then, we can divide a trajectory into few segments, so each can be classified as being in one of these motifs throughout the interval. Thus, we can assign a sequence of motifs to this trajectory a composition. For instance, a ReLU function on [-1,1] has composition ("constant", "increasing") or just (c, 2), whereas sin on the interval [x, has composition (i, i). The motifs can be chosen based on the application and the required granularity. The points between motifs are called transition points, and their coordinates can be mapped to the properties of trend (see Figure 2)
Concepts Trend
Mathematical objects
Composition
Part of trend
Motif
Properties of trend
Transition points
Figure 2: Correspondence between concepts in Section 2 and mathematical objects in Section 4
Notation. We say I is an interval (of IR) if it is an open interval, closed interval, or half-closed interval. The interval has to contain more than one point. We denote the set of all intervals on IR as I. Let E R, we denote the shifted interval as I c {x + interval, we call any function f - R an interval function and we denote its domain as dom(f). We denote the set of all interval functions as F.
7}. Let I C R be any
Definition (Motif).
motif s binary relation between the set of interval functions F and the F, ZJ). We denote (f,1) fl and read it as "f on has a motif 8", Each motif needs to be:
set of intervals I (i.e.. C F x I ((f.))
well-defined, i.e for any f E F, and any I E I,
flI~s [ = I [ dom(f)
(1)
vanslation-invariann, ie., for any I E I, and any f FF
fl1 > f (I - c) + c) ~ VR
(2)
Now, we would like to assign a minimal sequence of motifs to a given trajectory: a composition. Definition 2 (Composition). Let f
be an interval function and S be set of motifs. A motif sequence of f in S is finite sequence of motifss (si...., such that there exists an interval partition)) ......, La) of such that f14, 8j Vj E [d]. A composition of f in S is the shortest motif se sequence of f in S. The points between the intervals are called the transition points. The set of all compositions for given set of motifs S is denoted by Cs. A set of motifs S is called compatible with subset F C F if for every E F' there exists unique composition, denoted Cs[f]
Compatibility between the set of motifs and the set of trajectories is crucial for an ML model that employs bi-level transparency we want to assign composition to every possible prediction un- ambiguously and, in turn, every feature vector. We call this assignment composition map.
For definition of interval partition, see Appendix
5

---

Published as a conference paper at ICLR 2024
Definition 3 (Composition map). Let set of motifs S be compatible with some subset F F. Let g IR M + F' be an ML model for time series forecasting, where M E N is the number static features. A composition map is denoted Ms RM Cs defined by Ms(a) Cslg(x)).
To understand model g. is crucial to understand its composition map with respect to mean- ingful set of motifs. We discuss examples of motifs and when they can be helpful in Appendix We define a particular of motifs that call dynamical motifs (see Table 2). They encode informa- tion about the trajectory's first and second derivatives. Moreover, the transition points between these motifs correspond local minima, maxima, and inflection points. These are the exact properties used in a standard mathematical exercise of fimction sketching whose goal is precisely to understand the function. These motifs form a backbone of TIMEVIEW introduced in Section 5
Dynamical
motifs are depicted in Table and defined formally in Example A Appendix
Table 2: We introduce a set of dynamical motifs that are often important to understand trajectories.
Symbol
Name
Definition
Visualization
8+0
Straight line with positive slope
f(x) = az + b, 0,bER
A C PA
Straight line with negative slope
f(x) = az + b, 0,
Straight line with zero slope
f(x) = b, D
4
Increasing and strictly convex Increasing and strictly concave Decreasing and strictly convex
f'(x) > 0, (²)
['(c) > 0, 0.)"
8-4
f'(x) < 0,)
8_-
Decreasing and strictly concave
f'(x) < 0,
TIMEVIEW
Based on our formalism in Section we introduce Time series Interpretable Model with Effective VIsualization (TIMEVIEW). This framework consists of two parts: the predictive model based on B-Spline basis functions, and an algorithm for calculating the composition map. This map aims to facilitate model visualization that complements our framework and demonstrated in Section 7
Realizing bi-level transparency through dynamical motifs. To realize bi-level transparency through dynamical motifs, we need to (I) understand the relation between the feature vectors æ and the compositions of the predicted trajectories, and (2) understand the relation between the fea- ture vectors x and the transition points of a given composition. To fulfill these conditions, we need to find a space of trajectories y satisfying the following criteria.
1. The set of dynamical motifs S is compatible with the class of predicted trajectories y 2. For every ÿ E y' we can calculate its composition Cs
Cubic splines are class of functions that satisfies both criteria mentioned above. We demonstrate that dynamical motifs are compatible with cubic splines in Appendix B Moreover, it is easy to calculate the dynamical composition of a cubic spline as it is a piece-wise wiiee-wise function consisting of
cubic polynomials connected at knots. We describe the exact procedure below and in Appendix C
B-Spline basis functions. We describe cubic splines as linear combinations of B-Spline (De Boor 1978) basis functions. Let 6; [0,T] + E. be 5th B-Spline basis function of degree 3. Given set of B basis functions {06 [bB)) we can express cubic spline as a linear combination ÿ()) EB ct)u(t), where c R vb [B]. Thus, each spline is described by latent vector B
Architecture. To match feature vector æ E M to vector E RB describing time series, we an encoder h R + R Ultimately, we define model g RMM - y as
B
((z) = (t) = Σ (())
(3)

---

Published as a conference paper at ICLR 2024
Implementation. We implement the encoder h fully-connected neural network. We choose set of knots for the B-Spline basis functions based on the training dataset using heuristic algorithm described in Appendix (note, the number of knots controls explainability perffrrannce trade-off). The values of ΦG at times to (vb E [B] vd E [D] can be efficiently precomputed before the training using scipy library's BSp1 in class. We want minimize the MSE loss between the predicted values of the trajectory ÿ at points t and the ground truth y². We also add L2 regularization loss LL., that the B-Spline coefficients (and thus the compositions) do not change too abruptly, The final objective is:
poolaiinabliyy-prfffrmmnee
Na
B Σ hox))))((1))
) + aLi2(g)
(4)
D
Nd
We minimize it using gradient descent. The block diagram describing the training procedure be seen in Figure 3 Implementation details, including the pscudocode, can be found in Appendix
Dataset
Before training
Static features
Time series
Knot selection Appendix C.3
f3
fn-2
knots
E [R DXM
y
Define
( RM
(d) jb
Φo (550) [
Evaluate
Encode
h: h:RM RRP
Combine
B-Spline basis functions
(
d=1
Predict
gl()) = ((((0) Calculate loss
Equation 4
Backpropagate
Training
Figure 3: This figure shows block diagram depicting the training procedure of TIMEVIEW
As with many transparent models (e.g.. GAMs, Decision Trees), model visualization is crucial for its interpretability. After TIMEVIEW is trained, we compute the composition map (see Definition and demonstrate how map, we need to perform composition extraction from predicted trajectory, i.e., calculate Cs(ū).
visualize it (or part of it) in Section 7 To compute the composition
Composition extraction. As described earlier, each trajectory is described by latent vector _ and defined linear combination of B-Splines, ÿ(t) b= Em (t))(t). Each di piece-wise chosen by
polynomial defined over the intervals determined by the internal knots (4,, (t,,..,,pp L our heuristic algorithm (Appendix C We can associate cubic in monomial basis (1 1- 4, with each of these intervals for each basis function (this can be precomputed). We call these cubics 4/..k. where k ranges from to B 1-33 3 (the number of intervals). Given vector we can now calculate the cubic in a monomial basis for each interval. The 4thh interval is just Σt= cyth,, As is just a cubic polynomial, we can readily calculate its first and second derivatives and thus assign a composition to the ttth interval. We repeat this process for every other interval, connect all the compositions, and merge some neighboring motifs if they are the same. Ultimately, we get a global composition for the whole ý. See Appendix C for the pseudocode and the block diagram description.
RELATED WORKS
We explain how our work intersects with related areas of ML. Refer to Appendix more details.
Transparent models for static predictions. Standard transparent methods for static predictions include linear/logistic regression, scoring systems (Ustun & Rudin 2016), decision trees/rule lists
For B-Splines of degree 3, B knots produce basis functions.

---

Published as a conference paper at ICLR 2024
Angelino al. 2018 Hu et al. 2019), and generalized additive models (GAMs) (Hastie & Tibshi- rani 1986 Lou et al. 2012). Such methods can often be used for time series forecasting by passing the time f as additional feature. They often satisfy bi-level transparency but have poor perfor- mance. In particular, all trajectories predicted by linear regression and GAMs are parallel; thus, they cannot model different trends (Section 77. Decision Trees capture non-additive interactions, enabling flexible forecasting models. However, they require many splits to approximate the ground truth, leading poor performance or incomprehensibility (Section
Closed-form expressions. Symbolic Regression (Schmidt & Lipson 2009 La Cava et al., 2021) aims fit closed-form expressions to data, i... mathematical formulas composed of a finite number of variables, binary operators (+, x, +), well- functions (e.g., sin, exp, log), and constants. For instance, sin(x²) 22.1y Differential equations represent another category of mathematical expressions that draw significant interest in the scientific community. Numerous algorithms have been proposed for discovering Ordinary Differential Equations (ODEs) (Brunton et al.) 2016) Qian et al. 2022) and Partial Differential Equations (Rudy et al. 2017 Long al. 2019). Mathematical expressions may not always satisfy bi-level transparency. In fact, the reparametrization of equations to reflect key theoretical quantities is an active of research (Preacher & Hancock 2015).
Feature importance for time series. While our research focuses on transparent models, many saliency (or feature importance) methods have been developed to highlight which features the model is sensitive to (Ribeiro et al.] 2016)] Lundberg & Lee 2017). Although these methods have been extended to time series inputs (Crabbé & Schaar [021 Leung al. 2023), limited work has been
done to extend them specifically to time series outputs. Current XAI techniques either assume
the output is a scalar (Siddiqui al., 2019) (e.g., time series classification (Hao & Cao 2020 ).
treat the trajectory as a single object (Gao et al. [023]-thus do not show how a feature c changes
the trajectory-orr show a saliency map at each predicted point separately (Pan et al.) 2020). thus
allowing only for bottom-uy up understanding of the predicted trajectory. The last category also
includes many recently proposed methods with attention mechanisms (Alaa & van der Schaar 2019
Lim et al. 2021). We contrast our framework with feature importance techniques in Appendix
Shapelets and motifs. As our method discusses the shape of the trajectory, may seem related to shapelet-based methods (Ye & Keogh 2009). However, these methods are usually used for data mining and classification tasks. They aim to find subsequences of a time series that represent the most important patterns of each class and thus can be used to distinguish between them (Chen al. 2022]. Similarly, motif discovery identifies short repeating patterns in the time series (Torkamani & Lohweg 2017) usually for insights into the problem classification tasks.
7 TIMEVIEW IN ACTION
Answering questions. Our interactive visualization tool for TIMEVIEW allows for answering ques- tions such as "What If" "How to be that", and "How to still be this" from the XAI Question Bank (Liao al. 2020) discussed Section T As explained earlier, answering such questions with the current bottom-up up approaches may often be futile since the notion of "different prediction" may be non-interpretable or simplistic. In contrast, TIMEVIEW allows the analysis of a trajectory change two levels, i.e., the composition of the trajectory or the coordinate of the transition point.
Visualizing perturbations. We can visualize the effect of perturbing one or two features at time using colorful bands (as in [ and colorful 2D contour plots (Fig- ure 4). In the left panel of Figure D we have movable slider for each feature that changes the predicted trajec- tory the center. The colors on the band below the slider signify the composition of the trajectory if the feature is in the corresponding range. This allows us to understand how the trend of the trajectory changes as we change each feature (level of bi-level transparency). To understand the properties of this trend (level 2), we choose any of the transition points in the central plot, and we can analyze its position with respect the chosen feature the plot on the right. It currently shows how the y-coordinate of the second transition point (local minimum) increases as initial tumor volume increases. (Figure
alpha
2.68
beta
1.93
Figure 4: Our interface can visualize per- turbing two features simultancously.
8

---

Published as a conference paper at ICLR 2024
shows how visualize the effect of changing two features at time. Each color in the contour plot corresponds to a different composition, so it is clear how changing the features influences the composition of the trajectory. Please see Appendix SeerndxxEfoor for more in-depth discussion.
Comparison with other methods. In the absence of time series methods fulfilling bi-level trans- parency, we adapt static transparent methods, such as linear regression, decision trees, and GAMs (Lou et al. 2012 Nori et al.] (019) to time series forecasting by treating time as a feature, denoted as Linear-T, DecisionTree-T, and GAM-T. We also compare with methods discovering closed-form expressions for trajectories, such as PySR for symbolic regression (Cranmer 2020), and SINDy for ODE discovery (Brunton al. 2016). We also include black-box models RNN, Ar-RNN, and state- of-the-art bree-based models adapted to time series forecasting (XGB-T (Chen & Guestrin 2016), (ee et al. 2017). CatBoost-T (Prokhorenkova et al. 2018)).
LGBM-T
Experiments were conducted on four real-world datasets (Airfoil (Brooks et al. 1989). fichain (Dis- penzieri et al. 2012). Stress-Strain (Aakash et al., (019). and Tacrolimus (Woillard al.] (011)) and three synthetic ones (Sine, Beta, and Tumor, the latter based on model from (Wilkerson al., 2.17p. The synthetic datasets are constructed to contain trajectories exhibiting many different trends. Figure [ Figure 4. Figure show TIMEVIEW fitted to Sine, Beta, and Tumor datasets. As shown in Table B TIMEVIEW outperforms the transparent methods and closed-fome expres- sion most datasets and achieves comparable performance to the black boxes. Details about the experiments can be found in Appendix D
Table 3: Comparison between TIMEVIEW, other transparent methods, closed-form expressions, and black boxes. The numbers denote a mean squared error. The lower, the better. Boldface red denotes the best black box results, boldface orange denotes the best closed-form expression results, and boldface green denotes the best transparent results. i denotes failure to converge. Note, RNN only works for regular time series.
Real datasets
Synthetic datasets
Method
Airfoil
flchain
Suress-Strat
Tacrolimus
Tumor
Sine
Beta
Black boxes
RNN
0.26 0.02
0.02 0.01
0.02 : 0.01
0.02 z 0.02
ACRNN
= 0.02
0.27 0.01
0.14 0.01
0.41
0.02 0.01
0.02 0.01
0.02 0.02
XGB-T
0.09 =: 0.00
0.20 0.00
0.02 a 0.00
0.29 a: 0.01
0.01 = 0.00
0.53 a 0.00
0.07 + 0.00
LGBM-T
0.11 = 0.00
0.20 + 0.00
0.08 + 0.00
0.29 + 0.00
0.01 = 0.00
0.00 + 0.00
0.00 = 0.00
CatBoost-T
0.09 = 0.01
0.21 + 0.00
0.05 + 0.00
0.37 + 0.04
0.00 = 0.00
0.00 + 0.00
0.00 = 0.00
Closed-form expressions
PySR
0.26 = 0.03
0.22 0.01
0.48 + 0.11
0.36 + 0.04
0.10 0.02
0.05 = 0.04
0.24 = 0.03 2.74 + 0.00
SINDY
0.61 0.00
0.39 0.00
1
1.11 + 0.00
0.07 =: 0.00
1.30 + 0.00
Transparent models
Linear-T
0.37 0.00
0.34 = 0.00
0.66 + 0.00
0.57 = 0.00
0.68 = 0.00
0.99 = 0.00
1.03 = 0.00
Decision T
0.36 0.00
0.21 0.00
0.15 4 0.00
0.31 + 0.00
0.22 + 0.00
0.10 + 0.00
0.34 + 0.00
GAM-T
0.28=0.00
0.32 + 0.00
0.09 a 0.00
0.38 + 0.00
0.54 0.00
0.54 + 0.00
0.69 0.00
TIMEVIEW
0.13
0.24 0.02
0.04 0.00
0,31 + 0.03
0.00 0.00
0.02 a: 0.00
0.04 : 0.00
8 DISCUSSION AND CONCLUSION
Applications. We believe bi-level transparency and our mathematical framework can inspire future XAI methods. For instance, note, that models adhering our framework (like TIMEVIEW) provide additional output next to the standard forecasted trajectory: the current composition and the coor- dinates of the transition points. Traditional XAI techniques for regression and classification can be applied to these additional outputs, instead of individual trajectory points, to gain more meaningful explanations. Thus, techniques such as feature importance methods (Lundberg & Lee 2017). local surrogates (Ribeiro al.) 2016), and counterfactual explanations (Karimi et al. 2020 can now be extended to time series forecasting settings. These, in turn, can open domains where the applicability of ML has been limited due to transparency concerns, including medicine, finance, and science.
Limitations and open challenges. TIMEVIEW is a particular application of livee transparency for time series forecasting from static features. We hope future works will extend it to settings where the input may contain the previous part of the trajectory or other exogenous time series (further discussion in Appendix F.

---

Published as a conference paper at ICLR 2024
Ethics statement. In this paper, we present novel conceptual framework for enhancing trans- parency in the domain time series forecasting, accompanied by its practical implementation known TIMEVIEW. A better understanding of machine learning models serves critical purposes such as model debugging and identifying and mitigating potential harmful biases. However, XAI techniques also be misused to foster unwarranted trust in models or merely achieve surface- level compliance with regulatory standards, As highlighted in our paper, domains such as medicine and pharmacology involve high-stakes scenarios. Therefore, prior deploying our model in such contexts, a rigorous examination is imperative to ensure it does not endorse decisions that could prove detrimental to individuals' well-being.
Reproducibility statement. All mathematical definitions are provided in Section and Ap- pendix A The proofs of theoretical results are shown in Appendix The implementation, including block diagrams and pseudocode, is discussed in Section S and in Appendix C
The experiment set- tings are discussed Section and Appendix [D] The code to reproduce the results and for the visu- alization tool can be found at https:://gtthu/ http://www......o.oeoaoaaaaaaaat
and at the wider lab repository httpsp/soooopppppe//aaaaoaaooaaa http:///ttttttcccoommeeeeerrarrr
Acknowledgments. This work was supported by Roche and AstraZeneca. We want to thank Katarzyna Kobalczyk, Fergus Imrie, Andrew Rashbass, and anonymous reviewers for their useful comments and feedback on earlier versions of this work.
REFERENCES
B. S. Aakash, JohnPatrick Connors, and Michael D. Shiclds. Stress-strain data for aluminum 6061- T651 from 9 lots 6 temperatures under uniaxial and plane strain tension. Data in Brief, 25: 104085, August 2019. ISSN 2352-3409. doi: 10.1016/j.dib.200 201.......11.14000....
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
Ahmed M Alaa and Mihaela van der Schaar. Attentive State-Space Modeling of Disease Progres- sion. Advances in Neural Information Processing Systems, 32: 1338-13811111 11348, 2019.
Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning Certifiably Optimal Rule Lists for Categorical Data. Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2018
Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado. Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58:82-115, June 2020. ISSN 1566-2535. doi: 10.
L. Biggio, T. Bendinellis, A. Neitz, A. Lucchi, and G. Parascandolo. Neural Symbolic Regression that Scales. In 38th International Conference Machine Learning, July 2021.
Paul Blanchard, Robert L. Devaney, and Glen R. Hall. Differential Equations. Cengage Learning, July 2012. ISBN 178-1-133-38808
Thomas F. Brooks, D. Stuart Pope, and Michael A. Marcolini. Airfoil self-noise and prediction, July 1989.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the Na- tional Academy of Sciences, 1 333115)3392--337, April 2016. ISSN 0027-8424, 1091-6490. doi: 151.:7/
C. Melody Carswell, Cathy Emery, and Andrea M. Lonon Stimulus complexity and information integration the spontaneous interpretations of line graphs. Applied Cognitive Psychology, 7(4): 341-357, 1993. ISSN 1099-0720. doi: 10. 2000//cp.23500700000
10

---

Published as a conference paper at ICLR 2024
Jiahui Chen, Yuan Wan, Xiaoyu Wang, and Yinglv Xuan. Learning-
Learning-based based shapelets discovery by feature selection for time series classification. Applied Intelligence, 52(8):9460 090099990--475,, June 2022. ISSN 1573-7497. doi: 10.1007/s 00007/0000499-000000000---
Tianqi Chen and Carlos Guestrin. XGBoost: Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16 pp. 785-794, New York, NY, USA, August 2016. Association for Computing Machin- ery. ISBN 978-1 978-1-4503-423222.
doi: 10. I 10.1199999997722999755
Erin Chung, Jonathan Sen, Priya Patel, and Winnie Seto. Population Pharmacokinetic Models of Vancomycin in Paediatric Patients: A Systematic Review. Clinical Pharmacokineties, 60(8): 985-1001, August 2021. ISSN 0312-5963, 1179-1926.
doi: 10. 10.10077/44026--22--00000
Jonathan Crabbé and Mihaela Van Der Schaar. Explaining Time Series Predictions with Dynamic Masks. In Proceedings of the 38th International Conference Machine Learning, pp. 2166- 2177. PMLR, July 2021.
Jonathan Crabbe, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar. Explaining Latent Representations with Corpus of Examples. In Advances in Neural Information Processing Systems, volume 34, pp. 12154-12166. Curran Associates, Inc., 2021.
Miles Cranmer. PySR: Fast & parallelized symbolic regression in Python/Julia. Zenodo, September 2020.
Stéphane D' Ascoli, Plerre-Alexandre Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. In Proceedings of the 39th International Confer- ence on Machine Learning, pp. 4520-4536. PMLR, June 2022.
Carl De Boor. A Practical Guide to Splines, volume 27. springer-verlag New York, 1978.
Brian de Silva, Kathleen Champion, Markus Quade, Jean- ChrisShrittephe Loiseau, J. Kutz, and Steven Brunton. PySINDy: A Python package for the sparse identification of nonlinear dynamical sys- tems from data. Journal of Open Source Software, 5(49):2104, 2020. doi: 10.21 105/joss. 02104.
Angela Dispenzieri, Jerry A. Katzmann, Robert A. Kyle, Dirk R. Larson, Terry M. Therneau, Colin L. Colby, Raynell J. Clark, Graham P. Mead, Shaji Kumar, L. Joseph Melton, and S. Vin- cent Rajkumar. Use of Nonclonal Serum Immunoglobulin Free Light Chains Predict Overall Survival in the General Population. Mayo Clinic Proceedings, 1766):5 June 2012. ISSN 0025-6196, 1942-5546. doi: 10. http::/..aaycp.20122...
Adeline Fermanian, Terry Lyons, James Morrill, and Cristopher Salvi. New Directions in the Ap- plications of Rough Path Theory. IEEE BITS the Information Theory Magazine, pp. 1-18. 2023. ISSN 2692-41100 doi: 10. 10.1109/BBIS. 2023.3243885.
Penglei Gao, Xi Yang, Rui Zhang, Kaizhu Huang, and John Y. Goulermas. Explainable Tensorized Neural Ordinary Differential Equations for Arbitrary-Ster Time Series Prediction. IEEE Trans- actions Knowledge and Data Engineering, 35(6))583755800, June 2023. ISSN 1558-2191. doi: 10. 10.1109/TKDEE222..311777
Amirata Ghorbani and James Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. International conference on machine learning, 2019.
Bryce Goodman and Seth Flaxman. European Union Regulations on Algorithmic Secision-Making and "Right to Explanation". Al Magazine, 38(3):50-57 October 2017. ISSN 2371-9621. doi: 10. 1 609//0mag.mma/. wwt.///ma...3......... v3813.2741.
Yi Rang Han, Ping I. Lee, and K. Sandy Pang. Finding Tmax and Cmax in Multicompartmental Models. Drug Metabolism and Disposition, 46(11):1796-188) November 2018. ISSN 0090- 9556, 1521-0099. doi: 10.1 1124/md..1 11111802200 8.8882636.
Yifan Hao and Huiping Cao. A New Attention Mechanism to Classify Multivariate Time Series. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, July 2020. doi: 10. 10.24963/ijcai. 2020/277. 20026027/
11

---

Published as a conference paper at ICLR 2024
Trevor Hastie and Robert Tibshirani. Generalized additive models. Statistical Science, 1(3):297- 318, 1986.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep Generative Symbolic Regression. The Eleventh International Conference on Learning Representations, 2023.
Xiyang Hu, Cynthia Rudin, and Margo Seltzer. Optimal Sparse Decision Trees. In Advances in Neurul Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning. pp. 448 456.. PMLR, June 2015.
Krzysztof Kacprzyk, Zhaozhi Qian, and Mihaela van der Schaar. D-CIPHER: Discovery of Closed- form Partial Differential Equations. In Advances in Neural Information Processing Systems, vol- ume 36, pp. 27609-27644, December 2023.
Kadierdan Kaheman, J. Nathan Kutz, and Steven L. Brunton. SINDy-PI: A robust algorithm for parallel implicit sparse identification of nonlinear dynamies. Proceedings of the Royal Society A: Mathematical. Physical and Engineering Sciences, 47662242):20200079, October 2020. doi: 10. 1i..098//spa.2220007999
Alan A. Kaptanoglu, Brian M. de Silva, Urban Fasel, Kadierdan Kaheman, Andy J. Goldschmidt, Jared Callaham, Charles B. Delahunt, Zachary G. Nicolaou, Kathleen Champion, -hah-Chris Loiseau, J. Nathan Kutz, and Steven L. Brunton. PySINDy: A comprehensive Python package for robust sparse system identification. Journal of Open Source Software, 7(69):3994, 2022. doi: 10.21 00/994
Amir-Hossed Karimi, Gilles Barthe, Borja Balle, and Isabel Valera. Model-Agnostic Counterfactual Explanations for Consequential Decisions. International Conference on Artificial Intelligence and Statistics, 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, pp. 2668-2677. PMLR, July 2018.
William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Vir- golin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary Symbolic Regression Methods and their Relative Performance. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, July 2021.
Kin Kwan Leung, Clayton Rooke, Jonathan Smith, Saba Zuberi, and Maksims Volkovs. Temporal Dependencies in Feature Importance for Time Series Prediction. In The Eleventh International Conference on Learning Representations, 2023.
Q. Vera Liao, Daniel Gruen, and Sarah Miller. Questioning the Al: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, pp. 1-15, New York, NY, USA, April 2020. Association for Computing Machinery. ISBN 978-1-45555 -88---500--60 doi: 10. 03.1145/3313831.33333 33376566990.
Bryan Lim, Arik Sercan, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4): 1748-1764, October 2021. ISSN 0169-2070. doi: 10. 10. iffoo//:iforreasst2200 03..012.
Zichao Long, Yiping Lu, and Bin Dong. PDE-Net 2.0: Learning PDEs from data with numeric- symbolic hybrid deep network. Journal of Computational Physics, 399:39999999 108925, December 2019. ISSN 0021-9991. doi: 10. (0.10069j.epp:2019.. 10892.9
12

---

Published as a conference paper at ICLR 2024
Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '12, pp. 150-158, New York, NY, USA, August 2012. Association for Computing Machinery. ISBN 978- 11-4503-1462244
doi: 10.1 (0.1144/23395302333353
Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Temry Lyons. Rough paths, Signatures and the modelling of functions on streams, May 2014.
Daniel Messenger and David M. Bortz. Weak SINDy: Galerkin-Basee Data-Driven Model Se-
lection. Multiscale Modeling & Simulation, 19(3):17441497, January 2021a. ISSN 1540-3459, 1540-3467. doi: 10. 137720M1
Daniel A. Messenger and David M. Bortz. Weak SINDY for partial differential equations. Journal of Computational Physics, 443:1 10525, October 2021b. ISSN 00219991. doi: 10. 10.1016/1..ep.2221. 110525.
Dr Mould and Rn Upton. Basic Concepts in Population Modeling, Simulation, and Model-Based Drug Development. CPT: Pharmacometrics & Systems Pharmacology. 1(9):6, 2012. ISSN 2163- 8306. doi: 10.1038/psp.2012.4
Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. InterpretML: A unified framework for machine learning interpretability. arXiv preprint arXiv: 2019.
Qingyi Pan, Wenho Hu, and Jun Zhu. Series Saliency: Temporal Interpretation for Multivariate
Time Series Forecasting, December 2020.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825-2888 2830, 2011.
Sebastian Pölsterl. Scikit-survival: A Library for Time-to-Event Analysis Built on Top of scikit- learn. Journal of Machine Learning Research, 21(212): 2020.
Kristopher J. Preacher and Gregory R. Hancock. Meaningful aspects of change as novel random coefficients: general method for reparameterizing longitudinal models. Psychological Methods, 20(1):84-101, 2015. ISSN 1463- 1082-989X. doi: 10. 10.1037/me00000002.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating Training Data Influence by Tracing Gradient Descent. Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930. Curran Associates, Inc., 2020.
Zhaozhi Qian, Krzysztof Kacprzyk, and Mihaela van der Schaar. D-CODE: Discovering Closed- form ODEs from Observed Trajectories. The Tenth International Conference on Learning Repre- sentations, 2022.
L. Rabiner and B. Juang. An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1): 4-16, January 1986. ISSN 3558-1284, doi: 10.1 A..100MMASSSP19990 1165342.
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlin- ear partial differential equations. Journal of Computational Physics, 357:125-141, March 2018. ISSN 0021-999) doi: 10. 11001//.jep.2217.11..0
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Explain- ing the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16, pp. 1135-1144, New York, NY, USA, August 2016. Association for Computing Machinery. ISBN 978- 1-550--4232-22 doi: 10. 00.14457939672.2937777
13

---

Published as a conference paper at ICLR 2024
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206 215, May 2019. ISSN 2522-5839. doi: 00.1088044225--09-00005
Samuel H. Rudy, Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 3(4):e1602614, April 2017. ISSN 2375-2548. doi: 10.1 1..1126sscidv.. 11w.ciiadv..60114....
Michael Schmidt and Hod Lipson. Distilling Free-Form Natural Laws from Experimental Data. Science, 324(5923)81-855, April 2009. ISSN 0036-8075, 1095-9203, 1165893.
doi: 10. 126/science.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning Important Features Through Propagating Activation Differences. In Proceedings of the 34th International Conference on Machine Learning, pp- 3145-3153. PMLR, July 2017.
Shoaib Ahmed Siddiqui, Dominique Mercier, Mohsin Munir, Andreas Dengel, and Sheraz Ahmed. TSViz: Demystification of Deep Learning Models for Time-Series Analysis. IEEE Access, 7: 67027-67040, 2019. ISSN 2169-3536.
doi: 10. 2..1109/ACCESSS2119922911
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The journal of machine learning research, 15(1):1929 1958, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 3319-3328. PMLR, July 2017.
Sahar Torkamani and Volker Lohweg, Survey time series motif discovery. WIREs Data Mining and Knowledge Discovery, 7(2):e1 199, 2017. ISSN 1942-4795. doi: 10.1002/widm.1195
Silviu-Marian Udrescu and Max Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16)eaay2631, April 2020. ISSN 2375-2548. doi: 10.1126/
aciadv.
Berk Ustun and Cynthia Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, 102(3):349-391, March 2016. ISSN 1573-0565. doi: 10.1007/ 510994-015-5522-6
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017.
Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-272. 2020. doi: 10380/4/ (0030///1192-099-000 592-019-0686-2.
Julia Wilkerson, Kald Abdallah, Charles Hugh-Jones, Greg Curt, Mace Rothenberg, Ronit Simantov, Martin Murphy, Joseph Morrell, Joel Beetsch, Daniel Sargent, Howard I Scher, Peter Lebowitz, Richard Simon, Wilfred D Stein, Susan Bates, and Tito Fojo. Estimation of tumour regression and growth rates during treatment in patients with advanced prostate cancer: retrospective analysis. The Lancet Oncology, 18(1):143-154, January 2017. ISSN 1470-2045. doi: 10.1016,
lean-Baptiste Woillard, Brenda C. M. de Winter, Nassim Kamar, Pierre Marquet, Lionel Ros- taing, and Annick Rousseau. Population pharmacokinetic model and Bayesian estimator for two tacrolimus formulations daily Prograf and once daily Advagraf. British Journal of Clinical Pharmacology, 71(3):391-405 March 2011. ISSN 1365-2125. doi: 10.1111/j.136 13.51111,336--2225.220 2610-222..0000 03837.x.
14

---

Published as a conference paper at ICLR 2024
Xiaoming Xi. Aspects of performance on line graph description tasks: Influenced by graph familiar- ity and different task features. Language Testing, 27(1):73-100, January 2010. ISSN 0265-5322. doi: 10.117/0265522299445544
Lexiang Ye and Eamonn Keogh. Time series shapelets: new primitive for data mining. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 947-956, Paris France, June 2009. ACM. ISBN 978-1-60553 778-1-60558-495-9 doi:
10.1145/1557019.1551122
Çagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. forcement Learning, June 2021.
Contimuous-Time Model-Based Rein-
Jeff Zacks and Barbara Tversky. Bars and lines: A study graphic communication. Memory & Cognition, 27(6):1073-1077 November 1999. ISSN 1532-5946. doi: 10.3758/BF03201236
Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, Lecture Notes Computer Science, pp. 818 833, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10590-11 doi: 101110009978--319-111 10590-1553.
Hui Zou and Trevor Hastie. Regularization and Variable Selection Via the Elastic Net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):00--320, April 2005. ISSN 7412. doi: 10.111 11j11.116.. 1467-16666669 2005.00503.x
15

---

(1,2......) Published as a conference paper at ICLR 2024
TABLE OF SUPPLEMENTARY MATERIALS
1. Appendix A
notation and definitions
2. Appendix
theoretical results
3. Appendix
implementation details
4. Appendix
experimental details and additional results
5. Appendix
additional discussion
6. Appendix extended related works
A NOTATION AND DEFINITIONS
A.1 NOTATION
We present the symbols used in this work and their meanings in Table 4
Table 4: Notation used throughout the paper
Symbol
Meaning
[n]
I
Interval (Definition
fli
Restriction onto 1, i.c., to - R. fla((t) = f(t) e to
F
Interval functions (Definition Shifted interval (Definition 4
I+c
M
ff I has motif s (Definition)
S
Set of motifs (Definition)
Cs
Set of compositions for given set of motifs S (Definition) 2
Cslfl
Composition of f in S (Definition 2)
Ms
Composition map for set of motits S (Definition
B-Spline basis function (Section
h
Encoder (Section 5
g
TIMEVIEW model (Section
Predicted trajectory (Section
D
Set predicted trajectories (Section
D
Number of samples
M
Number of static features
Na
Number of measurements for sample d
B
Number of basis functions
A.2 GLOSSARY TERMS
In Table we present glossary of important words and where they be found in the paper.
A.3 DEFINITIONS
Definition (Interval). We say / is an interval (of R) if it open interval, closed interval, half- closed interval. The interval has to contain more than one point. We denote the set of all intervals on IR as Z. Let RR we denote the shifted interval
{ar + c E 1}.
Definition 5 (Interval partition). Interval partition of interval I is sequence of intervals (I,,..., In) such that Uh_ I, I, I4 n 4,+1 = @ Vi E [n 1] and the right boundary of I; should be greater than the left boundary of T for all i [n 1).
Definition 6 (Interval function). Let I C R be any interval, we call any function f I R an interval function and we denote its domain as dom(f). We denote the set all interval functions as F.
16

---

Published as a conference paper at ICLR 2024
Table 5: All important words defined in the paper and their locations
Word
Location in the paper
Bi-level transparency
Section 2
Compatibility
Definition
Composition
Definition
Composition extraction
Section 5
Composition map
Definition
Dynamical motifs
Example
Exclusive motifs
Definition
1
Inclusive motifs
Definition
Interval
Definition
Interval function
Definition
Interval partition
Definition
Motif
Definition
Property (of trend)
Section 2
TIMEVIEW
Section
Transition point Trend
Definition
Section 2
Definition (Motif). motif s is a binary relation between the set of interval functions F and the set of intervals Z (i.e., E F x {(S,T) | E F, E I)). We denote (ʃ, E /l/ and read it as "f on 7 has a motif s" Each motif 8 needs to be:
well-defined, i.e., for any f F, and any I I,
flI fll~s
I I dom(f)
(5)
translation-invariann, i.e., for any 1 E I, and any f F,
flI 1I~~ = f fo c) + c) ~ s VC
(6)
Definition 8 (Composition). Let I R be an interval function and S be a set of motifs. A motif sequence of f in S is finite sequence of motifs (si....,5u), such that there exists an interval partition (I1,-. ........ Ia) of such that fl4, ~ sy Vjj E [d]. A composition of in S is the shortest morif sequence in S. The points between the intervals are called the transition points. The set of all compositions for a given set of motifs S is denoted by Cs. A set of motifs S is called compatible F' there exists a unique composition, denoted Cs[f]
with subset F C F for every
Definition (Compatibility). Let S be a of motifs and F F be subset of interval functionn. S is called compatible with J if for every f E F' there exists a unique composition, Cs [f] and unique sequence of the transition points associated with this composition.
Definition 10 (Composition map). Let set of motifs be compatible with some subset F F. Let RM + F' be a machine learning model for time series forecasting. A composition map is denoted Ms RM Cs defined by Ms(x) Cs (z))).
Definition 11 (Inclusive motifs). We say that motif inclusive if for any function f E and any interval / E Z, we have
J = S Vo C
Definition (Exclusive motifs). We say that a finite set of motifs S exclusive if for any function E F and any interval 1 E I, we have
IT - - ss
(8)
for all s, E S. In other words, if f is in particular motif on the interval I then it cannot be in any other motif on this interval.
A.4 MOTIF EXAMPLES
Example 1 (Range motifs). Let a, E R such that b. We define range motifs 8x, 8=, 8>
17

---

Published as a conference paper at ICLR 2024
fl ~ fz f() <a f ~ s= ff E Ias f( b
f|/~ Vz f(x
These motifs are beneficial when our interest lies not in the trajectory's exact shape, but in whether its value falls within specific range (a, b) and at what point in time. This can be partieularly useful in control problems where different policies apply based on whether variable is within or outside the operating range (a, bj. More motif examples can be found in Appendix A
Example (Threshold motifs). Range motifs (Example T can be modified by choosing a b. Then we end up with only two motifs:
/ ~ 84 Va f(x) <a f Vo E I < f(x)
These motifss are useful for monitoring whether the trajectory is above or below a certain threshold.
Example 3 (Monotonic motifs). Let us define monotonic motifs 848-, So which are defined as:
flI $4 if ff is strictly increasing on I (Vz, y I <
flI ~ s if f is strictly decreasing on I (Vx, xx < = f(a) f(y)
/ So if is constant on (Vx,y E f(x) f(w)
Example (Derivative range motifs). Let a, and b. We define derivative range motifs as:
flI 84 fl, Cl and Ya I f') <a f [ s= if f C and Ya E Ias < 11 8 ʃ/r E C2 and Va l (x)
Example
(Dynamical
seven as follows (also visualized in Table 2
motifs). We define
dynamical
motifs:
80 50 S0o +++ 84,
~ 840 if > 0 E R Vr E f(x = ar + + b
f i- 03 R Vx f())
A ~ 500 i Væ f() b
11~ 8+4 [f]] and Væ int(/) f'(x) > 0, (z)x)
fII~ 8+- fl C² and Vz E int ff()) 0, f,
fl~ s fl/ C2 and Vz int([]] f' 0,
fII S
if f|r
C and Vz E int f()z) (
B THEORETICAL RESULTS
Theorem 1 (Dynamical motifs are compatible with cubic splines). Let S be the set of dynamical motifs and let F be the set of cubic splines on some interval [a, b). Then compatible with F
Before we prove this theorem, we will prove two propositions about dynamical motifs, and two helpful lemmas.
Proposition 1. Dynamical motifs inclusive (as defined in Definition TI
Proof. Observe that $40, S-0, and 800 can be equivalently defined as:
flI s40 flı C² and Vx int(I)ff > 0, (x) -x)==0
ii~ss s-n fl, E C and Vr E int int(I),f((z) 0. 0, (z) -
flI~ Soo if flr E C² and Vz E =
This (with the rest of the motifs defined in Example shows that each motif is uniquely defined by the signs of first and second derivatives.
If sign(f") on an interval is constant then sign(f') sign(f") f') for any interval To T. The same holds for the second derivative. This proves that dynamical motifs are inclusive.
Proposition 2. Dynamical motifs are exclusive (as defined in Definition 72
18

---

Published as a conference paper at ICLR 2024
Proof. As shown in the proof of Proposition I each dynamical motif is uniquely defined by the signs of its first and second derivatives. If sign(f") on an interval / is constant then it cannot be equal to anything else. The same holds for the second derivative. This proves that dynamical motifs IT
are exclusive.
Lemma (Uniqueness of dynamical motif sequences), Let be the set of dynamical motifs and let f be an interval function. there exists motif sequence for J. (S1,-.., 8d), satisfying 8; E SVi E [d]. 8/ A 8441 Vi E (d - 11, then this motif sequence is unique.
Proof. Consider two motif sequences (81 (.1...., 84), (pi,..., ,P)) such that for both of them every two consecutive motifs are different. We show that these sequences need to be the same.
Let compare p1 and 81. Consider the intervals described by 81 and Pt and denote them 7 and J1 respectively. By inclusivity of dynamical motifs (Proposition D. f](1, n
81 and f((4 n J1) ~ Pt. That implies, by exclusivity (Proposition 2) that si = Pi. That also shows that Ji cannot be bigger than 11. If that were the case, it would overlap with Iz (described by 82). But 82 + $1 = P1. So, J1 C I1. If J1 is smaller than I then J2 C (I1 \ Jı). where J2 is deseribed by P2. and P2 81 = Pt. But we assumed that p1 * p2. Thus, J1 = IT. Now, we can apply exactly the same reasoning to the next pair of motifs P2). We continue this procedure and in the end, we get these two motif sequences are the same. Note, it is also possible that Ti \ Ji is just a single point. That is why the intervals corresponding to the motif sequence are unique up to the boundaries of the interval, i... the intervals might differ by single points.
Lemma 2 (Dynamic motifs are compatible with cubics). Let S be the set of dynamical motifs and let Q be the set of polynomials up to a third degree on R. Then S is compatible with Q.
Proof. Consider any polynomial f E Q. Let's write f as f(t) at + +ct We can calculate its first and second derivatives. Respectively, f'(t) = 3at² + 2bt + c. f") = 6at + 2b. f and f" divide R into intervals where each interval f' (or f" respectively) is either positive, negative, or equal to zero. I' divides R into at most 3 intervals, and f" divides into at most two intervals. There might be some isolated points that do not belong to any of these intervals. For instance, if (t) = t, we have two intervals: (-o0, 0) where f" is negative, (0, +o0) where f" is positive, and an isolated point at = O. For f" we have most one isolated point, and for f' we have at most two isolated points. We can now take the intersection of every interval defined by f' with every other interval defined by f". This gives us most intervals where each interval belongs to one of 9 different configurations that we denote by two symbols. The first symbol denotes the sign of the first derivative (+. - 0), and the second sign denotes the sign of the second derivative. For instance, an interval might be described as -+ if it was created by intersecting an interval that has a negative first derivative with an interval that has a positive second derivative. We observe that two configurations are impossible, 0+ and 0- we end up with 7 possible configurations.
We observe that if an interval I is described by -- then by Example 5 f|/ ~ 84-. Similarly for ++. -4. and
If interval is denoted by 00 then it means that 6at + 25 = 0 Vt E I, and 3at + 2bt + c = 0 Vt E I. From the first equation. we get that a = 0, b = 0, and from the second interval / is described by +0 then a = 0,b = 0 and e >> O. That means f is given by cl + d, where e > 0, so f(T ~ 8+0- Analogously with -0. The isolated points mentioned above are exactly the transition points between different motifs.
one we get c = 0. That means that /\T 800- Similarly,
This shows how for every polynomial up third degree, there exists a sequence of dynami- cal motifs and we showed explicitly how to construct such a sequence. Let us call this sequence (s1...., sg). Observe that the shortest motif sequence for ff cannot have two identical consecutive motifs. If that were the case, they could have been combined into shorter motif sequence. As every two consecutive motifs of (81,..., &4) are different then, by Lemma [ this is the only such sequence and thus it is the shortest. So ($1...., s2) is composition. That shows that for every f E @ there exists a unique composition and thus S is compatible with Q.
Now, we can prove Theorem 1
19

---

Published as a conference paper at ICLR 2024
Proof of theorem Theorem 7 Let be cubic spline defined internal knots t1 fi...... tn Let us denote fo the polynomial defined on (t,, t441). To construct composition for f. we construct compositions for each of the polynomials f; (as described in Lemma 2 Then neighboring motifs on both sides of every knot. If the motifs are the same, then we combine them and remove the transition point. This procedure gives motif sequence for f that denote as ($1,...,5s) .d).
look at the
Observe that the shortest motif sequence for f cannot have identical consecutive motifs. If that were the case, they could have been combined into shorter motif sequence (this is true because cubic splines have continuous first and second derivatives at the knots). As every two consecutive motifs of ($1...., sd) are different then, by Lemma [ this is the only such sequence and thus it is the shortest. So (s1,..., 8d) is composition. That shows that for every cubic spline, there exists unique composition, and thus S is compatible with cubic splines.
L
C IMPLEMENTATION
C.I MODEL
Pseudocode. The pseudocode of the model training in TIMEVIEW shown in Algorithm !
Algorithm TIMEVIEW Model training
Input: Static features X E DXX
Input: Time series (1 ( (
D d= (id) E ERN, y RNa
Input: Number of basis functions B
Input: Gradient-based optimization algorithm O
Output: Trained model G RM +
, SELECT (AAISECLNNTS((E Jan (1(1)
Jd
Appendix C.3
(0r) + B-Spline basis functions for knots I,,..., BB2
Initialize matrices ( Id= ((d) NaxB
4 jb
(1 + (t))))
procedure Loss(h)
(d) - 1b(d h (xx(d)
, (10) )/)
L 12
1 No
03
1L12
return L end procedure
h O(Loss)
Training
procedure G( E R.M t E R
Initialize $ NNB
4 + (t,)
2h(x)
return Ú
end procedure return G
Architecture of the encoder. We implement encoder h as fully connected neural network with 3 hidden layers. We also include dropout (Srivastava et al., 2014) and batch normalization (Toffe) & Szegedy 2015). The sizes of the hidden layers, activation functions, and dropout probability are fine-tuned.
Dynamic bias. We add additional constant basis function (equal to everywhere) "bias" that is adjusted for each sample.
Hyperparameters. We perform hyperparameter tuning using Optuna (Akiba et al. 2019) and run it for 100 trials. We describe the hyperparameters we tune and their ranges in Table 6 We fix the number of basis functions to be for all real datasets and Tumor, and 5 for Sine and Beta datasets. We found that 9 usually gives the best or nearly the best performance while maintaining compositions that still short. We chose 5 for Sine and Beta datasets because the compositions
20

---

Published as a conference paper at ICLR 2024
of trajectories in these datasets very short, there need for bigger number of basis functions. In practice, this parameter can be fine-tuned but numbers between and work well by default.
Table 6: TIMEVIEW hyperparameter ranges
Hyperparameter
Considered values
Hidden sizes
Integers from [16,128]
Activation
[ReLU, USigmo.i Siggmoi..TnhhLeaat. Tenk, ReLU,ELU,SeLU)
Dropout probability
Float from [0.0,0.5]
Learning rate
Float (log) from [0.0001,0.1]
Batch size
[64,128]
Weight decay
Float (log) from [0.000001, 0.1]
C.2
COMPOSITION EXTRACTION
Block diagram. The composition extraction procedure in TIMEVIEW is depicted in Algorithm
-(()= = 9 (c)
Knots
E
IR
Φ
Cot0ue
1-1
b-
Pa. Va.2 Ma. Va.
See 14-1
Suglsee
$44/540
C E IR
Figure This figure shows a block diagram depicting TIMEVIEW composition extraction
Pseudocode. The pseudocode of composition extraction implemented in TIMEVIEW shown in Algorithm
C.3 ALGORITHM FOR KNOT SELECTION
Each B-Spline basis function is determined by its knots snots-places where two polynomials meet. The challenge of knot selection in TIMEVIEW comes from the necessity of having fixed knots for all trajectories. As each trajectory may have different optimal knot placement, we aim to find knots that may not be optimal but are nevertheless chosen based the trajectories in the dataset. For instance, there is no need to have many knots in parts of the domain where the trajectories vary little. But it might be beneficial to have more knots in parts of the domain where the trajectories do vary. To achieve this goal we propose the following heuristic algorithm based on the Univariate SnivariateSpline function in scipy library (Virtanen al. 2020
21

---

Published as a conference paper at ICLR 2024
Algorithm 2 TIMEVIEW Composition Extraction
Input: Static features æ E IR
Input: Encoder
Input: Knots f...... fB-
Input: Method for calculating the composition of cubic COMPOSITIONCUBIC
Output: Composition C
( + B-Spline basis functions for knots 11,....
8
{PDR (ee Φb interval [Ex, tx+1 is given by cubic 1OSk h(x)
Cis f empty list
for k to B 3 do
Cist append COMPOSITIONCUBIC fim Co the tx+1)
end for
< concatenate Cisst
for i = 1 to 1en(C)-1 do if C; = C++1 then
remove C, end end for return C
Uni vari ateSpl variatespp ine function takes the trajectory (both and y), smoothing parameter F cal- culates the optimal number of knots, and returns their placement. For each trajectory in training dataset, we test different values of s to find the one that gives the exact number of knots we want (using method similar to binary search). Then we take the positions of these knots and add them to our set of found knots. We repeat this procedure for every trajectory. Then we use the K-means clus- tering algorithm from equal to the desired number of knots. The clusters returned by the algorithm are the selected knots.
library (Pedregosa al. 2011) with the number of clusters
D EXPERIMENTS
D.1 DATASETS
All datasets are split into training, validation, and testing sets with ratios (0.7 : 0.15 0.15).
D.1.1 REAL DATASETS
Airfoil. Airfoil Self-Noise dataset (Brooks et al. 1989) is a UCI dataset, obtained from a series of aerodynamics and acoustic tests of airfoil blade sections conducted in wind tunnel. The static features are the angle of attack, chord length, free-stream velocity, and suction side displacement thickness. The output transform of the raw signal). We treat the sound pressure level with respect to the frequency as time series, i.e., f-frequency, y y-sound pressure level. As the frequencies are uniformly chosen on logarithmic scale, we take the log transform of t. The processed dataset contains 106 samples, each having on average 14 measurements.
sound pressure level at specified frequencies (obtained by a Fourier
fichain. Flchain is dataset of the subjects from study of the relationship between serum-free light chain and mortality (Dispenzieri al. 2012). often used time-to-event problems. train a Random Survival Forest using www.kiitssriiiie... from the datasct, we generate a survival function and sample it at 20 predetermined points. The static features are the same in the original dataset, i.c., age, serum creatinine, the FLC group, kappa portion of serum FLC, lambda portion of FLC, monoclonal gammapothy (MGUS), and sex. The time series is described by y y-the the probability of survival up to this point, t time in days. We subsample the dataset to get 1000 samples, each with 20 measurements.
library (Pölsterl 2020). Then for each sample
Note, do not confuse with denoting motifs in work
22

---

Published as a conference paper at ICLR 2024
Stress-Strain. We take the stress-strain curves of aluminum 6061-T651 obtained by Aakash et al. (2019). The aluminum samples are sourced from 9 different lots at a few temperatures between 20 and 300 C. W treat the temperature and lot as static features and the time series is defined by strain,,, y-experienced stress. After preprocessing, we end up with 100 samples, each with 212 measurements average.
Tacrolimus. We take the drug concentration curves of two tacrolimus formulations obtained by Woillard al. (2011). The static features include sex, weight, hematocrit, hemoglobin, creatinine, dose, CYP3A5 genotype, and formulation. We also supplement with the initial concentration of the drug in the blood. After preprocessing, we end up with 90 samples, each with 10 measurements on average.
D.1.2 SYNTHETIC DATASETS
Tumor. We take the tumor growth model proposed in Wilkerson et al. (2017). This model is described by the following equation:
y(() exxp(-d() (1 x) exp(gl) -
Where c, d, g are free parameters corresponding to the proportion of tumor cells that undergo cell death due to treatment, decay of the tumor, and the growth of the tumor. This equation has been shown to capture many different trends of tumor volume over time and that is why we decided to use it in our experiments. To make the model more realistic we multiply by the initial tumor volume, and add the following static features: age, weight, and drug dosage. We describe the parameters from Equation ( using the following relationships:
9o (age/20.0)) 20..0)
d dg dosage weight
(10)
= 11(1 + exp(-dosage 6o))
These relationships are inspired by covariate models often used in PKPD models (Mould & Upton 2012 Chung et al. 202T).
We choose the following parameters to get trajectories of different and realistic shapes.
9u = 2.0
dg 180
(11)
@u = 10
To generate the data, we create 2000 samples. For each sample, we draw static features from uniform distributions described below.
age ~ Uniform(200 80)
weight ~ Uniform(40, 100)
initial.tumor.rooluee ~ Uniform(0.110.5)
dosage Uniform((.00 1.0)
For each sample, evaluate the tumor volume function y at 20 equally spaced time points on [0, 1].
Sine. This synthetic dataset was created to show how our model can be used when we have one static feature that nevertheless renders trajectories with different trends. We use the following equation:
2tm
(() sin
(12)
In this equation, is the static covariate. As we show in Figure I generate trajectories with different compositions. To create the dataset we generate 200 values for x uniformly on [0, 2.5), and for each generate 20 uniform time points from 0 to and evaluate y on them.
Beta. This synthetic dataset was created to show how our model can be used when we have two static features that render trajectories with different trends. As the probability density functions of the beta
23

---

Published as a conference paper at ICLR 2024
distribution are known for their variety of shapes, decided to them in experiments. In particular, we define our trajectory by the following equation
v(t) B( 3)
10 -1)-1
(13)
where B is the beta function. We treat ar and 3 as static features. To create the datasets, we create grid of (a, B) pairs where and B are created by taking 30 values uniformly spread over [1.0, 4.0).
This gives us 900 samples in total. For each sample, we generate 20 uniform measurements on [0, 1].
D.2 BASELINES
Linear-T. We implement the linear model with elastic net regularization (Zou Hastie 2005 using seikit-leazi package. We adapt them for time series forecasting by passing time as a feature. We perform hyperparameter tuning using Optuna (Akiba al. 2019) and run for 100 trials. We describe the hyperparameters we tune and their ranges in Table
Table 7: Linear-T hyperparameter ranges
Hyperparameter
Considered values
alpha
Float from [0.0, 1.0] Float from [0.0, 1.0]
ratio
RecisionTree-11 We implement decision tree regressor using sci weikkitllee eesiitllarre package. adapt them for time series forecasting by passing time as a feature. To keep the decision trees transparent, we restrict their depth to 5. We perform hyperparameter tuning using Optuna (Akiba et al. 2019) and run it for 100 trials. We describe the hyperparameters tune and their ranges in Table
Table DecisionTree-TT hyperparameter ranges
Hyperparameter
Considered values
max mex_t min samples split
Integers from [2, 32]
min sann_samlee_eees criterion
Integers from [1,32]
(squared error, friedman mse, absolute error)
GAM-T. We implement GAM using the Explainable Boosting Machines (EBMs) (Lou et al. 2012) available in the Int InterpretML package (Nori al. 2019). We adapt them for time series fore- casting by passing time as feature. By doing so, GAM finds a "baseline" trajectory which is the shape function associated with the variable. Then every prediction consists of the same baseline trajectory but shifted vertically depending on the values of other shape functions. Such model is very transparent
is likely to underperform settings where the trajectories have different trends for different features. We perform hyperparameter tuning using Optuna (Akiba al. 2019) and run for 100 trials. We describe the hyperparameters we tune and their ranges in Table
Table 9: GAM-T hyperparameter ranges
Hyperparameter
Considered values
max. bins
Integers from [128, 512]
outer_bage inner Jager_bags
Integers from [4 16]
Integers from [0. 8]
learning_ran rate
Float (log) from (0.001, 0.1]
max_leaves
Integer from [1, 6
min_samples.led
Integers from [1, 4]
24

---

Published as a conference paper at ICLR 2024
SINDy. SINDYy (Brunton et al. 2016) is ODE discovery algorithm that produces closed-fore ODEs. We use the implementation in PySINDy library (de Silva et al.) 2020 Kaptanoglu et al. )022). We adapt SINDy to work with static features by treating static features
constant control input. We perform hyperparameter tuning using Optuna (Akiba et al. 2019) and run for 100 trials. We tune the optimizer _threhhlld by considering values from 0.001 0.1 (on logarithmic scale), and we tune the derivative estimation method. We consider three different kinds of differentiation algorithms: finite difference, spline, and trend filtered. We show their corresponding hyperparameter ranges in Table I0
Table 10: SINDy differentiation algorithms: hyperparameter ranges
Differentiatonn method
Hyperparameter Considered values
Finite difference
k
Integer from [1, 5]
Spline
s
Float (log) from [0.001, 1]
Trend filtered
order
Integer from [0, 2]
Trend filtered
Float (log) from [0.0001, 1]
PySR. PySR (Cranmer, 2020 a symbolic regression library that uses genetic programming. Ge- netic programming algorithms highly depend on the time they are allowed to run for (as they can explore more possibilities). We put a time constraint for PySR that equal to the time it took TIMEVIEW to run, including the hyperparameter tuning. Thus we do not perform hyperparame- ter tuning for PySR (we run with default parameters). Instead, after each run we look at the best equations of each length and try each of them the validation set. We then report the loss of the best-found equation on the test set. We choose the maximum length of the expression to be a maximum of 20 and the number of features times 3. We choose 20 this is reasonable length of expression to comprehend, and we choose 3 times the number of features as that is the length of a GAM that usually considered interpretable. The parameters of PySR are listed in Table []
Table 11: PySR hyperparameters
Hyperparameter
Value
Binary operations
+, - X, log, exp, sin max(20, 3MM)
Unary operators maxsize
RNN. As a Recurrent Neural Network baseline, we use a multi-layer long short-term memory (LSTM) RNN as implemented in pytorch. We implement the encoder as multi-layer neural net- works. We also include dropout (Srivastava al.] 2014) and batch normalization (loffe & Szegedy) 2013). The sizes of the hidden layers, activation functions, and dropout probability are fine-tuned. We perform hyperparameter tuning using Optuna (Akiba et al. (019) and run for 100 trials. We describe the hyperparameters we tune and their ranges in Table 12
Table 12: RNN hyperparameter ranges
Hyperparameter
Considered values
Activation
[Rei.U.Sigmoic Lenk, ReI.U.EL.U,Se.U)) 3 iittegers from [16, 128)
Encoder hidden sizes
Encoder dropout probability
Float from [0.0, 0.5]
Decoder dropout probability
Float from [0.0, 0.5]
Decoder number of layers
Integer from [1,3]
Decoder hidden sizes
Integer from [16, 128
Learning rate
Float (log) from [0.0001,0.01]
Batch size
[64,128) Float (log) from [0.000001, 0.001]
Weight decay
25

---

Published as a conference paper at ICLR 2024
At-RNN. We adapt RNN described above to setting with irregular measurements by passing the difference between time points as an additional feature (Yildız et al. 2021).
XGB-T. We implement XGBoost (Chen & Guestrin, 2016) using py- xgboostt package. We adapt
it for time series forecasting by passing time as a feature. We perform hyperparameter tuning using
Optuna (Akiba et al. 2019) and run it for 100 trials. We describe the hyperparameters tune and
their ranges in Table 13
Table 13: XGB-T hyperpaaatte ranges
Hyperparameter
Considered values
estimators
Integers from [10, 1000] Float (log) from [0.001,0.1]
eta
min_child.am weight max_depth
Integer from [1, 10]
Integer from [3, 9)
gamma
Float (log) from (0.00000001,,.0]
subsample colsample bytree
Float from [0.1, 1.0]
Float from 00.1, 1.0
lambda
Float (log) from [1e 8, 1.0)
CatBoost-T. We implement CatBoost (Prokhorenkova et al. 2018) using catboost package. We adapt it for time series forecasting by passing time as feature. We perform hyperparameter tuning
using Optuna (Akiba et al. 2019) and run it for 100 trials. We describe the hyperparameters we tune and their ranges in Table
Table 14: CatBoost-T hyperparameter ranges
Hyperparameter
Considered values
n_estimators
Integers from [500, 2000]
learning rate depth boosting type bootstrap type 12 leaf reg early stopping rounds
Float (log) from [0.001, 1.0]
Integer from [1, 12]
(Ordered, Plain)
[Bayesian, Bernoulli, MVS]
Float (log) from [0.01, 10]
Integers from [10, 50)
LGBM-T. We implement LightGBM (Ke al. 2017) using 1 gightbb package. We adapt it for time series forecasting by passing time as feature. We perform hyperparameter tuning using Optuna (Akiba et al. 2019) and run for 100 trials. We describe the hyperparameters tune and
their ranges in Table 15
Table 15: LGBM-T hyperparmeee
ranges ranges
Hyperparameter
Considered values
n_estimators
(10, 50, 100, 200, 500, 1000, 2000, 5000)
learning rate num leaves
Float (log) from [0.00001, 10.0)
Integer from [2, 256]
min www..cilldsscc -amples
Integer from [1, 100]
reg alpha
Float (log) from [le 10.0 or 0.0
reg lambda
Float (log) from [1e - 7, 10.0) or 0.0
D.2.1 SENSITIVITY ANALYSIS
In this section, we investigate how sensitive TIMEVIEW to the number of basis functions B. We perform experiments for different numbers of basis functions (from 5 16) two real
26

---

Published as a conference paper at ICLR 2024
datasets Airfoil and Tacrolimus. The results can be in Figure We observe that TIMEVIEW not very sensitive to the number basis functions and achieves good performance even when the number of basis functions is very low. That means it can have good predictive power even if the compositions of the predicted trajectories are very short-and thus easy to understand.
Impact the number of basis functions
0.4
0.3
Enro
0.1
Tacrolimus Airfoil
0.0
Number of basis functions
Figure 6: This figure shows how the error changes with the number of basis functions
D.3 COMPUTATION TIME
important to note that training TIMEVIEW is basically the same as training a standard neural network for multi-output regression. Thanks to the preprocessing of the B-Spline basis functions there no significant overhead due to time series forecasting. In fact, TIMEVIEW uniquely requires single pass through the network predict the whole trajectory matter the time horizon. This may make it faster than methods that forecast trajectory by generating outputs sequentially. The visualization tool is applied only after the model is trained and works in real-time. The computation times (in seconds) can be seen in the Table [6 below. The run times are for all experiments for given dataset. That includes 100 runs of hyperparameter tuning and 10 runs get standard errors.
The experiments were performed on 12th Gen Intel(R) Core 17-12700H with 64 GB of RAM and NVIDIA GeForce RTX 3050 Ti Laptop GPU as well as on the 10th Gen Intel Core 19-10980XE with 60 GB of RAM and NVIDIA RTX A4000. All experiments take around 23.6h to run.
Table 16: Computation times (in seconds)
method
Airfoil
Beta
Sine
Stress-Strain
Tacrolimus
Tumor
flchain
RNN
3098
1121
4299
1034
At-RNN
691
2952
1265
1265
175
3901
1112
XGB-T
34
134
98
124
32
244
79
LGBM-T
25
364
172
9
30
CatBoost-T
70
1426
1098
882
93
1289
715
PySR
214
2018
1040
308
272
2977
1211
SINDy
64
292
77
2757
232
1136
35247
GAM-T
12
28
14
109
56
194
1139
Linear-T
1
2
2
1
5
DecisionTree-T
1
17
50
104
58
TIMEVIEW
191
1908
1022
194
135
2740
987
D.4 LICENSES
The licenses of the software used in this work are presented in Table
27

---

Published as a conference paper at ICLR 2024
Table 17: Software used and their licenses
Software
License
gplearn
BSD 3-Clause "New" "Revised" License
scikit-learn
BSD 3-Clause "New" or "Revised" License
numpy
liberal BSD license
pandas
BSD 3-Clause "New" or "Revised" License
scipy
liberal BSD license
python
Zero-Clause BSD license
PySR
Apache License 2.0
interpret
MIT License
py-xgboost
Apache License 2.0
pytorch
BSD-3
pytorch lightning
Apache License 2.0
tensorboard
Apache License 2.0
matplotlib
Matplotlib license BSD compatible
jupyter notebook
BSD 3-Clause "New" "Revised" License
scikit-survival
GPL-3.0 license
pysindy
MIT License
catboost
Apache License 2.0
lightgbm
MIT License
E ADDITIONAL DISCUSSION
E. ANSWERING QUESTIONS ABOUT TIMEVIEW: TUMOR EXAMPLE
In this section, demonstrate how we
the visualization tool to understand TIMEVIEW predictions. In particular, we show how our tool allows us to answer the questions described in We train TIMEVIEW the synthetic tumor dataset (details in Appendix DD
Section [
Q: Would the predicted tumor volume keep decreasing we adjusted the treatment?
This is question level about the trend of the trajectory. Figure 7 shows that the current trajectory is decreasing. The colorful band under the dosage slider shows the compositions for different drug dosages. We can that the trajectory is increasing for very low dosages (orange). For higher dosage (blue), the trajectory is first decreasing and then increasing. Finally, higher dosage (green and pink) corresponds decreasing trajectories. To answer the question, is enough to check at which points the bands change color. If the drug's 's dose is kept above 0.46, the trajectory will keep decreasing. The secondary plot in the right panel shows how the y-coordinate point changes as the initial tumor volume changes. As expected, the plotted function is close to the identity. a good sanity check confirm that our model learned that one of the static features corresponds to the initial measurement.
the first transition
Q: What feature changes would lower the minimum tumor volume?
This is question on level about the property of particular trend. Let us consider a lower drug dosage where the trajectory decreases and then increases (Figure g). We observe that the minimum of the tumor corresponds to the y-coordinate of the second transition point. We can use the tool on the right to plot how the minimum depends on the dosage (or any other feature). We can see that the minimum volume decreases with increased drug dosage.
Q: How feature changes would impact the time this minimum is achieved?
Similarly, we can consider the time when this minimum is achieved. We can easily do that by just switching from the y-coordinate of the transition point to its t-coordinate (Figure 9). We can see how the time depends the drug, we bring this time closer to 0 where "merges" with the first transition point, and the composition of the trajectory changes. The trajectory becomes increasing (Figure 10).
the dose of the drug. Crucially,
see how by lowering the dose of
28

---

Published as a conference paper at ICLR 2024
Tumor volume
Transiion point
2.00
1.75
100
8922
0.5
1.00
0.75
0.2
dosage
0.50
0.10
0.23
0.37
0.50
0.25
0.00
0.15 0.46
Figure 7: TIMEVIEW applied to the tumor dataset. Different dosages of the drug correspond to trajectorics with different compositions. We infer that if the drug's dose is kept above 0.46, the trajectory will keep de- creasing. The right panel demonstrates that TIMEVIEW has learned that one the static features corresponds the initial measurement.
Tumor volume
52.00
Trasstion
2.00
1.75
Weight
8020
1.50
0.40
0.35
1.25
0:38
1.00
dasge
0.50
0.46
0.56
0.00
0.0
0.2
0.4
0.6
0.8
1.0
dosage
Figure 8: TIMEVIEW applicd to the tumor dataset. The right panel shows how the trajectory's 's minimum depends on the drug's 's dosage.
Tumor volume
52.40
Trasition
2.00
1.75
1.0
Wright
1550
1.25
0.6
0.0
0.2
dosage
0.50
0.0
0.27
0.25
0.00
0.0
0.2
0.4
0.5
0.8
1.0
dosage
Figure 9: TIMEVIEW applied to the tumor dataset. minimum volume depends on the drug's dosage.
The right panel shows how time to achieve the
E.2 CLOSED-FORM EXPRESSIONS
Whether a closed-form expression guarantees bi-level transparency or not highly dependent on the exact for of the expression. For instance, a trajectory described by y sin(2m.rt) is transparent. We can understand the trend of the trajectory (the characteristic sinusoidal shape) and we can understand the impact of the covariate (x:) on the property of this trend (its frequency). However, some even very
29

---

Published as a conference paper at ICLR 2024
Tumor volume
Trarstion paint 2
2,00
175
waile
88.00
1.30
035
1.00
0.75
dosage
0.08
0.50
44.24 $5.96
67.68
79.40
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
Figure 10: TIMEVIEW applied to the tumor dataset. For such low dose of the drug, the trajectory keeps increasing exponentially. We can also see that, for instance, the trajectory has the same trend regardless of the weight of the patient. The right panel shows how the final tumor volume depends on age.
simple expressions do not have this property. A good example is the expression for the probability density function of the beta distribution used in Section 7 i.e., y(t) Bia.pj (1- 1 B is the beta function, Even though the expression is very compact it is very challenging to infer the trend of the trajectory based on the features a and B. It is even harder to understand the impact of the features the properties of these trends. That is why, believe, the Beta dataset is very interesting example for TIMEVIEW.
where
E.3 CONTRAST WITH FEATURE IMPORTANCE METHODS
As we explain throughout the paper, current feature importance techniques are bottom-up, i.e., they calculate the importance scores with respect to individual prediction horizons. Below we show the results of applying SHAP (Lundberg & Lee 2017) ) top of LightGBM (adapted for time series forecasting) to the Sine (Figure ITI. Beta (Figure 12,, and Tacrolimus (Figure [3} datasets.
SHAP values assign an importance score to each feature. Features with positive scores impact the prediction positively, while those with negative SHAP values impact the prediction negatively. As are interested in time series forecasting, the output of the algorithm is not single outcome, but rather a whole trajectory. That is why we apply SHAP separately to a set of predictions at discrete time points.
While SHAP offers insights into the importance of specific features at discrete time points, TIMEVIEW takes this analysis a step further by providing comprehensive understanding of fea- ture impacts on the entire trajectory. It shows exactly how the prediction would change if the value of the feature changed (i.e., how the trend of the trajectory and its properties would change).
Moreover, SHAP indicates feature importance at pre-specified time steps that may lack broader significance. In contrast, TIMEVIEW, captures the essence of the trajectory's behavior by demon- strating the impact on critical properties such as local maxima, minima, and inflection points even as their time coordinates vary.
Even in complex, figh-dimensional scenarios, TIMEVIEW remains adept at addressing counterfac- tual queries. For instance, it can address the question of how the maximum of a trajectory changes one of the variables is adjusted. This is of critical importance in pharmacology, where the trajectory the drug concentration curve and the variable of interest is the drug dose. It also provides clear understanding of how changes to the feature vector influence the trajectory's 's direction, for instance, transitioning it from ascending to descending. It is impossible to answer such queries using SHAP values.
In the table below we summarize examples of questions that can be asked about the model and whether they be answered by SHAP and TIMEVIEW. We also observe that SHAP always be used on top of TIMEVIEW to get additional insights. However, we cannot get TIMEVIEW explanations from just any black box model.
30

---

Published as a conference paper at ICLR 2024
Table 18: Questions about model and whether they be answered using SHAP and TIMEVIEW
Question
SHAP
TIMEVIEW
Which features are important?
Yes
Yes
Which features are important for the prediction at 1.5?
Yes
No
How does the dose of the drug impact the maximum concentration?
No
Yes
How does the feature vector need to be altered to change the trajectory from increasing to decreasing
No
Yes
0.4
SHAP values for each feature over time (Sine dataset)
0.2
Feature
0.0
00.2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Time (hours)
00.6
Figure 11: SHAP values for each feature over time calculated for LightGBM trained on the Sine dataset
0.50
SHAP values for each feature time (Beta dataset)
Feature
alpha
0.00
beta
-0.25
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-0.50
Time (hours)
-0.75
Figure 12: SHAP values for each feature over time calculated for LightGBM trained on the Beta dataset
SHAP values for each feature over time (Tacrolimus dataset)
DOSE
10.0
DV 0
7.5
SEX
WEIGHT
Feature
HT
НB
2.5
CREAT
CYP
FORM
-2.5
-5.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
11.0 12.0
Time (hours)
Figure 13: SHAP values for each feature time caleulated for LightGBM trained the Tacrolimus dataset
31

---

Published as a conference paper at ICLR 2024
E.4 ADDITIONAL SNAPSHOTS OF THE INTERFACE
Tacrolimus. We apply our model to the Tacrolimus dataset (Woillard et al. 2011) that contains measurements of drug concentration blood, and we present a snapshot of the trained model in Figure [4] The band under each slider shows the values of the composition map when change value of the corresponding slider but all other features are fixed. This allows us to answer questions such "What If "How to be that", and "How to still be this" from the XAI Question Bank (Liao et al. 2020) discussed Section [ For instance, in Figure [4 we choose the second transition point that corresponds to the maximum concentration of the drug. On the secondary plot, on the right, we can see how the position of this maximum changes as we vary any of the covariates. Currently shown plot indeed shows that if the dose of the drug is decreased then the maximum concentration decreases as well. We can also see that it is not an entirely linear relationship.
Current feature values
Location of the transition paint
Different
Dos:
compositions
feature value
Predicted trajectory
Tot ilion Juet
Movable sliders
32.0
5-
y ort coordinate
030
00
2.5
3.7
73
20.0
125
100
Transition points
CHEAT FORM
RORM
Figure 14: This figure shows full interface where we are interested in perturbing one feature at time (thus we use colorful bands instead of contour plots)
Sine. We also show a snapshot of our interface when TIMEVIEW is applied to the Sine dataset- Figure 15
Level (trends)
Level (properties)
Movable slider
Current feature value
Predicted trajectory
Transition poirt
1.0
1.00
6.5
3
0.0
0.90
-0.5
1.01
1.22
132
feature
10
Transition points
y or coordinate
Figure 15:
Snapshot of our dynamical visualization of TIMEVIEW. Our model adheres to bi-leve! transparency-4 top-down approach that focuses on the trend of the trajectory (Level 1) and the properties of the particular trend, e.g., transition points (Level 2). The left panel shows how the trajectory trend changes when one of the features perturbed. The right panel investigates the position (y-coordinate) of the second transition point (local minimum) as the initial tumor volume changes.
32

---

Published as a conference paper at ICLR 2024
Beta. Sometimes, may be interested in the effect of perturbing two features at the time. We visualize it using a colorful 2D contour plot. Figure 16 shows TIMEVIEW applied to the Beta dataset.
Predicted trajectory for oipho and beta
1.0
A
B
0.60
0.55
0.50
2.21
1.79
2.00
221
Figure 16: This figure shows how our interface can be used understand perturbations of two features the time. Panel A shows how you predicted trajectory on plot on right. Panel shows different trends corresponding to different regions the feature space. Panel C shows how we can choose any of the transition points and see how their position changes as we change any of the features. In particular, we show how the second transition point, the maximum of the trajectory, moves the left the increases.
the sliders to around the feature space and update the
E.5 INSIGHT INTO RESULTS
Some methods perform better on real datasets than synthetic ones. As we explain in Section 7 we create the two synthetic datasets (Sine and Beta) to be challenging in the following way: trend of the trajectory changes significantly for different features. The same is true for the Tumor datasets. The tumor growth model we use was proposed to fit many different cancer trajectories WWilkerson et al., 2017). That includes exponential increase, exponential decay, decrease and then increase. Although real data can be noisy, the bigger variability of trends makes it harder for some methods.
GAM-T struggles on the Tumor dataset. As we mention in Section 6 GAM-T's predicted trajectories all parallel. They equal to the shape function ft) shifted vertically by for static features æ. That means it can model only one trend. As the focus of the synthetic datasets was on different trends, it is not surprising that GAM-T performs very poorly on them. In particular, the Tumor dataset contains trajectories of different trends (e.g.. exponential increase, exponential decay, decrease and then increase).
fi())) + +fM.......M.....
E.6 LIMITATIONS AND FUTURE WORKS
Static inputs. TIMEVIEW is a particular application bi-level transparency time series fore- casting from static features. We hope future works will extend it to settings where the input may contain the previous part the trajectory or other exogenous time series.
Modelling the trajectory as cubic spline. We chose to model the trajectory cubic spline because cubic splines are flexible, and this representation allows for a straightforward composition extraction (see Section 5 and Appendix C). However, not all trajectories can be effectively approxi- mated by a cubic spline. Future works should investigate other ways to represent the trajectory that would still make it amenable composition extraction.
33

---

Published as a conference paper at ICLR 2024
Regularization. Although use the L2 penalty constrain how quickly the compositions change as we change the input features, future research could explore constraining the learned composition map for improved interpretability or designing better penalties for overly long and abruptly changing compositions.
Likelihood of sample. The visualization interface allows for showing all combinations of input variables, even ones that do not appear in reality. In the future, the interface can be enhanced to display information about the likelihood of particular sample (or a whole set of samples).
Confidence bounds. We could apply a technique like Deep Ensembles to get uncertainty estimates on the coefficients of the B-Spline basis functions which we could then transform into uncertainty estimated for the whole trajectory. Another interesting approach would be to adapt uncertainty estimation to our conceptual framework of trends and properties, i.c., how uncertain we are about particular trend.
Modeling seasonality. Our framework could accommodate seasonality by modeling seasonal changes as a sine function where the neural network determines the frequency, magnitude, and offset (the same way as the B-Spline coefficients). Our model would be defined as:
B
((x)
Σ h(z))ou(t) + A(x) sin(w(x)) + +(x))
(14)
where A RM R. w RM + R. t RM R are either separate neural networks or they share weights with the encoder h.
Then, treat the compositions in the same way the current implementation of TIMEVIEW but with three additional properties to monitor (in addition to transition points). This allows us to answer questions like "How would the variability increase if decrease this particular feature?".
Alternatives to cubic splines. As we describe in Section 5 we chose cubic splines because we needed a space of trajectories that satisfies the following two criteria.
The set of dynamical motifs S is compatible with the class of predicted trajectories D.
For every ò E Y we can calculate its composition Cs(ú]
We could use many other basis functions, such as a sine basis or some types of wavelets. They could satisfy the first condition, but the composition would likely need to be calculated numerically and not analytically as it is done now. In case we are interested in a different set of motifs (please see Appendix A.4 for examples), other choices may also be possible. For instance, for range motifs or monotonic motifs, Haar wavelets may be a good choice, and we suspect the compositions can be calculated efficiently.
Applications to traditional XAI techniques for static predictions. Adhering to our framework provides additional output next to the standard forecasted trajectory: the current composition and the coordinates of the transition points. Traditional XAI techniques for regression and classification can be applied these additional outputs instead of individual trajectory points to gain more meaningful explanations. Thus, techniques such as dimensionality reduction, feature importance methods, local surrogates, and counterfactual explanations can now be extended to time series forecasting. This opens up numerous potential extensions and applications of our approach.
E.7 INHERENTLY DISCRETE TRAJECTORIES
Modeling phenomenon continuous system is an established practice in sciences and engineer- ing. That includes settings mentioned in our paper, such as disease modeling or drug concentration in blood. Of course, in practice, we only observe discrete measurements, which may be irregular and noisy. However, sometimes the phenomenon cannot be modeled as a continuous system. This happens where the set of time points is inherently discrete, and there is no notion of time "in between the time points". For instance, the number sunshine hours for each day would be an inherently discrete trajectory there is meaningful "number of sunshine hours" between two consecutive days. We note that our conceptual work in Section 2 still accommodates such scenarios. However, the formalism in Section must be adapted.
34

---

Published as a conference paper at ICLR 2024
E.8 TRAJECTORY CHANGES
In Section 2 we claim that although a prediction of a regression algorithm can change in only ways (increase, decrease, remain constant), a trajectory can change in numerous ways. To see that, let us consider a set of time points T = {1, 2,,,.... 9, 10). The function can increase, decrease, or remain constant for each time step. Thus, the function described by these ten values can change in 3td = 59049 ways (which is much bigger than 3-the number of ways an output of regression model can change). We describe this in the paragraph on "increased cognitive load" in Section ? It is infeasible for human to reason about all these different possibilities, which motivates our proposal of top-down approach. The situation becomes even more complex where the trajectory is not defined over ten time points but over a whole real interval-whit is the focus of our work.
F EXTENDED RELATED WORKS
Transparent models for static predictions. Standard transparent methods for static predictions include linear/logistic regression, scoring systems (Ustun & Rudin 2016), decision trees/rule lists (Angelino et al. 2018: ; Hu et al. 2019), and generalized additive models (GAMs) (Hastie & Tibshi- rani 1986 Lou et 2012). Such methods can often be used for time series forecasting by passing the tmm f as an additional feature. They often satisfy bi-level transparency but have poor perfor- mance. For instance, all trajectories predicted by linear regression and GAMs are parallel; thus, they cannot model different trends (Section 77). Decision Trees capture non-additive interactions, enabling flexible forecasting models. However, they require many splits to approximate the ground truth, leading to poor performance or incomprehensibility (Section 7
Closed-form expressions. Closed-form expressions are mathematical formulas composed of a fi- nite number of variables, binary operators (+, -, x, +), well-known functions (c.g., sin, exp, log), and constants. For instance, sin(x²? 22.yy, A machine learning area that aims to find such ex- pressions fitting the data is called Symbolic Regression (La Cava et al. (02I). This area originated from attempts to unearth equations describing physical systems through machine learning (Schmidt & Lipson| 2009 Udreseu & Tegmark] 2020; HHll et al., [023] Biggio et al.) 2021 D D'Ascoli et al., 2022). Differential equations represent another category of mathematical expressions that draw significant interest in the scientific community. Numerous algorithms have been proposed for dis-
covering Ordinary Differential Equations (ODEs) (Brunton et al. 2016]] Qian et al. 2022 Kaheman
al. 2020 Messenger & Bortz (021a) and Partial Differential Equations (Rudy et al. | Long
et al. 2019: Raissi & Karniadakis 2018 Messenger & Bortz 202Tb) Kacprzyk et al. (023). Al-
though some mathematical expressions may satisfy bi-level transparency, this not guaranteed as it depends on the actual form of the found equation. In fact, reparametrization of equations so that their parameters reflect quantities of key thcoretical interest is an active area of research (Preacher & Hancock, 2015). We delve into this subject in greater detail and provide examples in Appendix B
Overview of XAI techniques. While our research focuses on transparent models, the landscape of post-hoc explainability methods has experienced significant growth. Such methods are used to explain the predictions
black box model. These include feature importance methods (Ribeiro)
let al., 2016: Lundberg & Lee 2017) (also called saliency methods) that highlight which features
the model is sensitive to, example importance methods (Ghorbani & Zou 2019] Pruthi et al.) 2020
Crabbe et al.
202I) that identify important training samples, and concept-based explanations (Kim)
ct al. 2018). Relatively little attention has been devoted to time series (Barredo Arrieta et al.)
2020
but a few recent methods aim to extend feature importance to this setting (Crabbé & Schaar 202
Leung et al.
2023).
Feature importance for time series. While our research focuses on transparent models, many saliency (or feature importance) methods have been developed to highlight which features the model sensitive to (Ríbeiro et al. 2016)] Lundberg & Lee 2017). Although these methods have been ex- tended to time series inputs (Crabbé & Schaar
et al. (023), limited work has been done to extend them specifically to time series outputs. Current XAI techniques either assume the output is scalar (Siddiqui et al.] 2019) (e.g., time series classification (Hao & Cao. 2020)), treat the trajectory as a single object (Gao et al.) 2023) -thus do not show how a feature changes the trajectory- show saliency map at each predicted point separately (Pan et al., 2020). thus allow- ing only for a bottom-up understanding of the predicted trajectory. Saliency methods can be broadly divided into Gradient-based (Sundararajan et al. 2017 Shrikumar et al. 2017). Perturbation-based
35

---

Published as a conference paper at ICLR 2024
Zeiler & Fergus 2014), and iirention-basee methods (Vaswani al. 2017 Alaa & van der Schaar 2019 Lim et al. 2021). Other important examples include SHAP (Lundberg Lee 2017) based on Shapley values and LIME (Ribeiro et al. 2016) that fits local linear model.
Shapelets and motifs. As our method discusses the shape of the trajectory, it may seem related to chapelet-based methods (Ye & Keogh 2009). However, these methods usually used for data mining and classification tasks. They aim to find subsequences of a time series that represent the most important patterns of each class and thus be used to distinguish between them (Chen al. 2022). Similarly, motif discovery identifies short repeating patterns in the time series (Torkamani & Lohweg 2017) usually for insights into the problem or classification tasks.
Statistical methods. Although methods like ARIMA or Hidden Markov Models (Rabiner & Juang 1986 have potential for compact state transition equations, these models generally fail meet bi- level transparency standards due to the complexity of understanding how the input feature changes influence the entire trajectory- they do not allow for top-down understanding as the trajectory constructed sequentially.
36


Reviews:
[{"review_id": "Q1ZSXaSByY", "reviewer": "Reviewer_nunX", "summary": "In this paper, Authors propose a transparent model for Time Series Forecasting, which should provide direct explanation for the results generated by the model. They investigate a top-down approach that decompose a time series in trend and properties. \n\nThey demonstrate the efficiency of their platform TIMEVIEW with time series based on static characteristics. The proposal enables to explore the impact of varying the static characteristics onto the output, while demonstrating reasonable performances.", "questions": "I had difficulties to fully understand the paper as I had to go back and forth during the reading. And I have a Time Series Forecasting (TSF) background and the I think the terminology used in the paper got me confused. However, the most important question I have is:\n\nWhere is the explainability? If I understand correctly, the proposal can help determine the impact of varying given static parameters (figure 4) on the output of the model. However, it does not infer on why it has this specific impact (the hidden relation in the model). Current, AI models are considered as black boxes because what is happening inside the model to link input and output is complex and too difficult for human to understand. But in my opinion the proposal is not different from such models. It does not give clear relation between input characteristics that generate a given output.  I am not an expert on XAI, but in my opinion, it looks like more to an exploratory platform than a XAI model.\nFurthermore, why is it more comprehensible than other baselines? Someone could set up a platform to visualize the impact of changing the input of a linear model on the produce output, but will it be a good XAI model?\n\nAnd here are some more comments that in my opinion would need to be treated to increase the current score.\n\n# Bottom-up versus Top-down\n\nThe bottom-up approach mentioned by the Authors is rather a “bottom” approach in my understanding. Meaning that models used the raw data (usually regularly spaced) to do TSF. Authors need to clarify this point what is the “up” especially for reader to understand the difference with the proposed approach.\nIs it the following:\n\n*Bottom-up*: from time points predict time points and define a trajectory, but \n\n> understanding trajectory from time points is not natural for human\n\nThen, why not from time point determine the trend (for instance with moving average), min, max and other statistics (with sliding window) that will make understanding \"simpler\"? Why the top-down approach is better?\n\n*Top-down*: Is this approach doing from time point define a trajectory then predict a trajectory for which we first look at trend and properties and then time points? Or from time point predict trajectory for which we first look at trend and properties and then time points?  \n\nIn addition, Authors never demonstrated the issue with the bottom-up approach and why do bottom-up approaches are not bi-level transparent. In related work, they say, \n\n# Terminology\n## Feature\nIn the example box in the introduction, what is the definition of feature? And instance? These terminologies are not clear and may confuse readers.\n\nIn usual TSF, feature (or variate) are other time series that support the prediction of the target (what Authors also called exogenous features). But it looks like a feature in the introduction example is a characteristic or a specific time step of the time series.  Authors would need to clarify these points to avoid confusion when reading.\n\nIn addition, does instance here mean an input sample of the model?\n\n## Regression versus Time series\nIn my opinion, __time series setting__ is not an appropriate terminology. \nit should be multistep forecasting or something similar.\n\n## Knots\nDefinition of knots should not be in appendix.\n\n\n# Continuous time series\n\n> interval [0, T] ⊂ R, where T ∈ R is a time horizon, and the underlying trajectory is continuous\n\nEvery time series are discrete as values are points/snapshot taken at a given time. Authors might review this sentence or at least clarify their point.\n\n> Thus, understanding a (continuous) trajectory by individual values is unnatural for humans\n\nBut again, time series are never continuous. Even with a very low frequency sampling, it is still a set of points that, in the best scenario, can be matched or approximated by a function/trajectory.\n\n# Function versus trajectory\n\n> there are numerous ways a function can change\n\nI disagree for each time steps the function will either increase, decrease or remain constant, same as regression. Probably, Authors want to say that the function(trajectory?) on the prediction window, may have different shapes depending on different outputs.\n\n# Claims\n\n> often satisfy bi-level transparency but have poor performance\n\nBut where is the proof for that claim? Which model from the experiments are bottom-up approaches?\n\n> However, these methods are not directly applicable to time series forecasting\n\nIt should be backed up by some reference. Linear Regression can be applicable to time series forecasting. Indeed, Linear models such as proposed in [1] have proven to be competitive with latest Transformer-based models, which indeed are less explainable.\n\n# Comparison with decomposition proposal or multi-resolution\n\n[1] and [2] are proposal to decompose time series in trend and seasonality (with either transformer or linear based), why such method would be less explainable than the proposal?\n\n[1] https://arxiv.org/pdf/2205.13504.pdf\n\n[2] https://arxiv.org/pdf/2106.13008.pdf\n\nWhat are the differences of the proposal with multi-resolution solution such as [3] (learning from different temporal granularity)?\n\n[3] https://link.springer.com/chapter/10.1007/978-3-031-05933-9_6\n\n\n\n# Paper needs a proof-reading\n * ”different prediction” wrong opening brackets\n * Smoothing parameter is s and motif is also s change to avoid confusion", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Lots of effort are made to describe the context, the goal and the importance of transparency. \n\nThe baselines models are well-presented, and all parameters are clearly defined, which will greatly help researchers reproduce these results.", "weaknesses": "In my opinion, the writing style is not adapted and will confuse or have readers get lost. \nThere are too many references to future sections/definitions/details, which complexify the reading. For instance, Authors introduce some concept, but do not give further explanation (even simplified one), but rather referred to specific sections. Which makes readers go back and forth through the paper.\n\nIn addition, there is no conclusion to this paper, which should give readers the most important points to remember from this study.\n\nIt is surprising that in a paper oriented in TSF there is only one TSF baseline (RNN).\n\nFinally, I think that there are too many claims that are not referenced or back-up in this paper, which diminishes its credibility.\n\n(Note the introduction and discussion style are really unconventional for this type of paper)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N28mgul8rp", "reviewer": "Reviewer_eJfZ", "summary": "Interpretability in time series forecasting is typically regarded a post-hoc task, and takes a \"bottom-up\" approach that relies heavily on the observations at individual timestamps. In this paper, the authors argue that this view is overly granular and non-intuitive, whereas focusing on the trends and properties of the time series trajectory would be more informative. In the spirit of explainable ML, the authors then propose the notion of bi-level transparency, along with a framework (TIMEVIEW) that encodes time series are a composition of B-splines. The interpretability aspect is further complemented by the development of a UI tool that allows for further explainability, counterfactual reasoning and sensitivity analysis on the time series. The paper concludes with a comparison to other forecasting methods, where the authors note that predictive performance remains comparable with the aded benefit of enhanced interpretability.", "questions": "See sections above.\n\n** Post-rebuttal Update **\n\nIncreased score from 6 to 8.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, and a pleasure read. It is well structured, and the included visualisations successfully convey the paper’s propositions. I commend the authors for their attention and care to detail when preparing this submission.\n- I appreciated how the authors considered different flavours of interpretability here, as illustrated in the Figure on Pg 2. The complete example presented in Appendix E also conveys the paper’s contributions very effectively.\n- The inclusion of a notebook for executing the aforementioned analysis also serves as a useful guide for practitioners looking to experiment with this framework, and lays the foundations for future extensions of this work.", "weaknesses": "- Although the authors already include a few examples of real-world problems where the tooling can be applied, I think that a more fleshed out experiment that runs throughout the whole paper can be especially helpful for conveying the contributions in a clearer manner (Appendix E accomplishes this very well).\n- The comparison to other methods towards the end of the paper addresses a question I had regarding the trade-offs between interpretability and performance, but this constraint could also be featured more prominently throughout the paper, such that the proposed methodology is consistently presented in light of this compromise.\n- I would be interested in learning more about instances where the proposed simulation is *not* suitable, and what properties of time series one would need to inspect before deciding to apply this technique (beyond the requirement for static features).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6tdcFiGfm", "reviewer": "Reviewer_PiQk", "summary": "This article discusses the importance of transparent machine learning models, particularly in high-stakes domains like healthcare, finance, and criminal justice. While transparent models have been proposed for classification and regression, time series forecasting poses unique challenges for transparency.\n\nThe article introduces a top-down framework for bi-level transparency in time series forecasting, aiming to understand both higher-level trends and lower-level properties of predicted time series. It emphasizes the need to comprehend changes in trajectory, which is more complex than single-label outputs.\n\nThe article contrasts this with the challenges of using a more traditional bottom-up approach that focuses on individual time points and values, which may not provide a holistic understanding of the entire time series.\n\nTo implement the proposed framework the article introduces TIMEVIEW, a transparent machine learning model for time series forecasting based on static features. The model uses B-Spline basis functions and an encoder to match feature vectors to trajectory descriptions.", "questions": "- How would your framework behave when one of the features influences the variability of the time series?\n- Are there alternatives to the cubic spline setup when using your framework?\n- Are there ways to incorporate domain knowledge/priors into this framework?\n- Does your framework seamlessly apply to the common time series setting of a single long sample, i.e. when predicting on overlapping chunks of the time series with a corresponding panel of features? Would there be a concern with the selection of nodes being non-congruent for the overlapping samples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem addressed in the article is important and under-developed in the existing literature.\n- The presentation is clear and intuitive.\n- The idea of a top-down framework in this context appears novel.", "weaknesses": "- The main contribution in the method appears to be in suggesting a pre-determined dimension reduction to the time series and operating in that new space.\n- The framework appears limiting in applications where time series variability is an important aspect of the interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sx1anjxc9g", "reviewer": "Reviewer_pJsr", "summary": "The authors introduce TIMEVIEW, a top-down transparent framework designed for trajectory forecasting with static variables. They aim for two levels of transparency, explaining the forecasting through both its trends and properties. To formalize this approach, the authors introduce the concepts of motifs and compositions, avoiding the tracking of individual trajectory values. TIMEVIEW comprises two components: a predictive model based on B-Spline basis functions, and an algorithm for calculating the composition map to facilitate easy visualization. The efficacy of TIMEVIEW is assessed using four real-world datasets and three synthetic ones, and it is compared against several competitors. Transparency is demonstrated through a user-selected example on a synthetic dataset, with additional examples provided in the appendices.", "questions": "- The focus on static variables seems to limit the framework's utility in true time-series forecasting. Given that you claim to forecast time-dependent variables based solely on fixed attributes, isn't this more akin to regression analysis? This approach seems niche. \n- I order to avoid the two variables limit, have you considered visualizing the contour plot using techniques like t-SNE, PCA, UMAP, or similar?\n- Since transparency is the primary goal of your paper, more space should be devoted to proving this claim. For instance, I had difficulty understanding Figure 4.\n- Considering Table 2, which classifies types of motifs, have you evaluated how performance would change if motifs were characterized as sets of classes rather than as continuous values? If the end user ultimately requires such a table, perhaps classification would suffice.\n- In the related works, you distinguish your method from shapelet-based methods, stating that these are primarily used for data mining and classification tasks. However, if these shapelet methods are unsupervised (e.g., Karlsson, Isak, Panagiotis Papapetrou, and Henrik Boström. \"Generalized random shapelet forests.\" _Data mining and knowledge discovery_ 30 (2016): 1053-1085.), they can also be applied to regression—and consequently, forecasting through reduction.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The focus on transparency in forecasting is extremely important.\n- The paper is generally well-written and interesting.\n- The proposed framework appears sound and potentially useful in specific domains.", "weaknesses": "- The limitation to static variables is a significant drawback. Additionally, the model can visualize the effects of only two variables, which is another strong limitation.\n- The paper should provide a more robust assessment of the model's transparency, rather than relegating this to the appendix.\n- No specific code references were found.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)