A Functional Extension of Semi-Structured Networks
Authors: David Rügamer
Bernard X.W. Liew
Zainab Altai
Almond Stöcker
Venue: NeurIPS 2024 poster (2024)
Keywords: Functional Data
Neural Networks
Biomechanics
Primary area: interpretability_and_explainability

Abstract: Semi-structured networks (SSNs) merge the structures familiar from additive models with deep neural networks, allowing the modeling of interpretable partial feature effects while capturing higher-order non-linearities at the same time. A significant challenge in this integration is maintaining the interpretability of the additive model component. Inspired by large-scale biomechanics datasets, this paper explores extending SSNs to functional data. Existing methods in functional data analysis are promising but often not expressive enough to account for all interactions and non-linearities and do not scale well to large datasets. Although the SSN approach presents a compelling potential solution, its adaptation to functional data remains complex. In this work, we propose a functional SSN method that retains the advantageous properties of classical functional regression approaches while also improving scalability. Our numerical experiments demonstrate that this approach accurately recovers underlying signals, enhances predictive performance, and performs favorably compared to competing methods.

A Functional Extension of Semi-Structured Networks
David Rügamer
Department of Statistics, LMU Munich
Munich Center for Machine Learning (MCML)
Munich, Germany
david@stat. uni wwt.:uniimmmmeeeeeee -mwnnchen.d.e de
Bernard X.W. Liew, Zainab Altai
School of Sport, Rehabilitation and Exercise Sciences University of Essex
Colchester, UK [b] 2...622,z..ltal altaaa@sssaee]ass....... @essex. ac. uk
Almond Stöcker
Institute of Mathematics
École Polytechnic Fédéral de Lausanne (EPFL)
Lausanne, Switzerland
almond. seaeendetttcckee@el...... ch
Abstract
Semi-structured networks (SSNs) merge the structures familiar from additive models with deep neural networks, allowing the modeling of interpretable partial feature effects while capturing higher-order non-linearities at the same time. significant challenge in this integration is maintaining the interpretability of the additive model component. Inspired by large-scale biomechanics datasets, this paper explores extending SSNs to functional data. Existing methods in functional data analysis are promising but often not expressive enough to account for all interactions and non-linearities and do not scale well to large datasets. Although the SSN approach presents a compelling potential solution, its adaptation to functional data remains complex. In this work, propose functional SSN method that retains the advantageous properties of classical functional regression approaches while also improving scalability. Our numerical experiments demonstrate that this approach accurately recovers underlying signals, enhances predictive performance, and performs favorably compared competing methods.
Introduction
Incorporating additive structures in neural networks enhance interpretability has been major theme recent advances in deep learning. For example, neural additive models (NAMs; (T) allow users to learn basis functions in automatic, data-driven fashion using feature subnetworks and thereby provide an alternative modeling option to basis function approaches from statistics such generalized additive models (GAMs; (17). Various extensions and related proposals have been published in recent years. Notable examples include the joint basis function learning by [41 or the combination of GAMs and neural oblivious decision ensembles 3 When combining structural assumptions with arbitrary deep architectures, the resulting network is often referred to as wide- and-deep [10] or semi-structured network (SSN; 49 (2)). Various extensions of SSNs have been
proposed in recent years, including SSNs for survival analysis 24 25), ordinal data fitting (2), for
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Bayesian SSNs [ or SSNs incorporated into normalizing flows 23]. One of the key challenges in SSNs is ensure the identifiability of the additive model part in the presence of deep network predictor in order to maintain interpretability.
uncertainty quantification 12
In this work, motivated by large-scale biomechanical application, study how efficiently transfer these two concepts the embedding of basis functions into neural networks and the extension to SSNs for functional data analysis.
Functional data analysis Functional data analysis (FDA; 42 is a research field of increasing )) that extends existing modeling approaches to address the functional nature of some data types (e.g.. sensor signals, waves, or growth curves). One well-knownn extension is the class of functional additive models [[5] 53], which represent the functional analog of GAMs. These models adopt the classical regression model point of view but allow for both in- and outputs to be (discretized) functions. Functional additive models, however, often not expressive enough to account for all interactions and non-lineasssies and do not scale well to large datasets.
significance in statistics and machine learning (e.g.. B 37 38 57
Biomechanics prominent example of a research field dealing with functional data is biomechanics. As in many other applications concerned with physical or biological signals, the integration of machine learning in biomechanics research offers many advantages (see, e.g., [26] 29, 30). The research on joint moments in biomechanics, for example, provides insight into muscular activities, motor control strategies, chronic pain, and injury risks (see, e.g., [33 3I kinematic features with machine and deep learning predict joint moments, researchers can bypass the need to gather high-quality biomechanics data and instead use "cheap" sensor data obtained through, e.g., everyday mobile devices which in turn allow predicting the "expensive" signals that can only be obtained in laboratory setups (20) 32 33). While a promising research direction, existing methods face various challenges including lack of generalization and missing clear understanding of the relationship learned.
36 47 ). By combining
1.1 Related work in functional data analysis
In FDA, there has been a successful translation of many different methods for scalar data to function- valued data, including functional regression models and functional machine learning techniques.
Functional regression models Regression models with scalar outputs and functional inputs are called caalar-on-fuccioon regression models. An overview of such methods can be found in [45] Models with functional output and scalar inputs are called function-on-scalaa models [5] They can be applied when the outcome error function is assumed to be repeated realizations of stochastic process. Combining these two approaches yields the function-on-funeto regression model with functions input and output [see, e.g., 35 53). We provide more technical description
of function-or on-function regression is given in Section 2.1
Functional machine learning In recent years, several machine learning approaches for functional data have been proposed. One Gaussian process functional regression [GPFR: 21 which combines mean estimation and a multivariate Gaussian process for covariance estimation. As for classical Gaussian process regression, the GPFR suffers from scalability issues. Gradient boosting approaches for functional regression models have been proposed by 4 6). These approaches result in sparse and interpretable models, rendering them especially meaningful in high dimensions, but lack expressivity provided by neural network approaches. Pioneered by the functional multi-layer perceptron (MLP) of (48), various researchers have suggested different functional neural networks (FNNs) [6 44 555. including MLPs with functional outcome (cf. Section 222 for technical details). A
recent overview is given in [59]. Existing approaches are similar in that they learn a type of embedding to represent functions in space where classical computations can be applied (cf. Fig, 2(a).
1.2 Current limitation and our contribution
From the related literature, we can identify the most relevant methods suitable for our modeling challenge. Neural network approaches such as the functional MLP provide the most flexible class for functional data, whereas additive models and additive boosting approaches such as 66] 53 yield interpretable models. An additive combination of these efforts to obtain both an interpretable model part and the possibility of making the model more flexible is hence attractive option.

---

Open challenges While semi-structured models are suited for combination of interpretable additive structures and deep neural networks, existing approaches cannot simply be transferred for application with functional data. Due to the structure and implementation of SSNs, a naïve implementation would neither be scalable to large-scat scale datasets nor is it clear how to incorporate identifiability constraints to ensure interpretability of the structured functional regression part in the presence of a deep neural network.
Our contributions In this work, we address these limitations and propose an SSN approach for functional data. Our method is not only the first semi cemi-structured modeling approach in the domain of functional data, but also presents a more scalable and flexible alternative to existing functional regression approaches. In order preserve the original properties of functional regression models, we further suggest an orthogonalization routine to solve the accompanied identifiability issue.
2 Notation and background
We assume functional inputs given by J > second-order, zero-mean stochastic processes X, with square integrable realizations 2; Sj - IR, i.e. with x, E j 1, ......J J on real intervals Sj. Similarly, a functional outcome is given by suitable stochastic process Y over a compact set T C R with realizations y T + R. y (T) =: G. For the functional imputs, we simplify notation by combining all (random) input functions into vector X(s) = (Xi(s),..... X)(ss)) ( with realization a(8) TR where 8 = (81,..... 85) E S c R is a J-uple from domain : S x ... x S,, and H. == (,)())) x ... x (2)())).
Next, introduce cmntioon-on-function regression (FFR) and establish link FNNs motivate our own approach.
2.1
Function-on-function regression
An FFR for the expected outcome ya()= = ((Y(()))) of (f), as follows:
using inputs X can be defined
B(Y(I)XX= = = b(t) + > 5=1 S.
wy(s, (s)))))
(I)
where b(t) is functional intercept/bias and the (s, t) are weight surfaces describing the influence of the jth functional predictor time point s E ES on the functional outcome time point E T. point-wise transformation function, mapping the affine transformation of functional predictors to a suitable domain (e.g., T(-) exp(-) to obtain positive values case Y is a count process). Various extensions of the FFR model in ① exist. Examples include time-varying integration limits or changing the linear influence (t) to a non-linear mapping.
Example Figure D shows example of a function-on-functia regression by visualiz- ing the corresponding learned weight surface, with three highlighted areas: a) An isolated positive weight multiplied with a positive fea- ture signal at (90)) induces a small spike at v10).. b) A more extensive negative weight multiplied with mostly positive feature values for [1, 20] and integrated over all time points results in a bell-shaped negative out- come signal at around
89-
w(s.t)
Response function yto
78-
0.10
C)
0.05
56
45
34-
-0.00
a)
30. c) A large but faded positive weight region multiplied with positive feature values results in slight in- crease in the response function in the range I E [50, 100). In most applications of FFR, the goal is to find these salient areas in the weight surface to better understand the relationship between input and output signal.
23
34
45
56
67
70
69
100
Feature function x(s)
Figure I: Exemplary weight surface (center), feature signal (bottom), and the resulting response signal (left) when integrating ʃ xxs)w(ss t)ds.
3

---

2.2 Functional neural networks
To establish the connection between FFR and FNNs, is instructive to consider a function-on-fuuction MLP (FFMLP). In its basic form, an MLP for scalar values consists of neurons h R + RR + r(b+ w x) arranged multiple layers, each layer stacked one on top of the other. The extension to fully functional L-layer MLP can be defined recursively by the kth output neuron h of a functional layer / 1,..., as
Mi-1
hx (t) = 7!!)
(t)+ +
Σ m==
(2)
where Mj denotes the number of neurons in layer 1, b// E (2(() and w m.k (2 u
for some functional domains 4 input layer hm (+) Im (+) for all predictors 1,..., J, and last layer with Mr. neuron (1) (()
Semi-structured functional networks
Our approach generalizes the previous models by considering general neural network A H. - g that models the input-output mapping from the set of functional features X to the expected outcome (t) of the functional response for E T
ttt) = ((X)()) = (X+(X)()+
comprising an FFR part (X) and deeper FNN architecture A -x()((t), which are added and transformed using an activation function This combination can also be thought of as functional version of residual connection. In the following, we drop the functional input arguments of X and X for better readability. The model in (3) combines structured interpretable models as described in Section 2.1] with an arbitrary deep network such as the one Section 2.2 In particular, this allows for improving the performance of simple FFR while retaining as much interpretability possible (cf. Section 3.3).
Interpretable model part As in QT. we model the interpretable part x+ sum of linear functional terms > (t) = Ss, t)(s,t) ()) (s)ds. To make our model more concrete, we expand each weight surface w, in finite dimen-iimeniiona tensor- cusor-produuct basis
wy ws(s, (()) (s)
(4)
over fixed function bases o, = 1 Ojk E (')),) and (t) = (Wa. v.. E (2(T))
with weight matrix Qj E TR x R, Analogously, we can represent the intercept as xt(
b(t) (t) O We may understand the model's interpretable part for the jth feature as functional encoder (X,) (X,) I O (s))) encoding X, into a latent variable 2j RR,,,, and a linear decoder &b with weights θ mapping zj to the function X (t):
Here, Φf presents the dual basis to Øy. Visually, the weight surface wj in (4) can be interpreted as exemplarily shown in Figure]] [
Deep model part The part A (t) in (3) is placeholder for a more complex network. For FNNs, conventional neural network might be applied after encoding X with &" and before decoding its results with 2b, such that en- and decoding are shared with the interpretable part A+ as depicted in Fig. p2. Alternatively, general FFMLP described in Section 2.2 can embedded into the approach in Fig. P... As researchers have often already found well-working architectures for their data, a practically more realistic architecture for semi-structured FNNs is to allow for a more generic deep model as depicted in Fig. p.. This is also the case in our application on biomechanical sensor data in Section 4.2 Here, we choose X X- (1) to be specific InceptionTime network architecture [19) that is well-established in the field and train its weights along with the weights e of X (t).

---

3.1 Implementation for discretized features
As functional predictors are usually only observed on a discretized grid on the domains Sy. we now derive a way to implement X (t) in practice. Depending on the architecture used for the deep part, we might proceed analogously with A (t)). Let x/ (///((ss) (s,) E R be the evaluation of the ith realization (8) T of X, at time points S,, r = 1. I..... . R, stacked into vector z; E RR For better readability, we assume that these time points are equal across all J features. With only discrete evaluations available, we approximate integrals over Sj numerically with integration weights A (sy), e.g. using trapezoidal Riemann weights. Given the tensor-product basis representation of the weight surface wo,(s, L) in effectively evaluate (s,,,) for all R time points, yielding Φ (0) (sRR) RR x and obtain the row-vector Dj of approximate functionals
n
(11))))) = / O5 ~
Σ (,(o)))os(()))))))
: = (A, o x Φ E RIXKs
where o denotes the Hadamard product and A, = together, we can represent the jth interpretable functional model term > (1) as
A As(n)] Putting everything
1) 1 s, (s) @
This model part is still a function over the outcome domain T, but discretized over the predictor domains. We can now proceed with the optimization when the functional outcome is represented with finitely many observations.
3.2 Optimization for discretized outcomes
For the ith observation, we can measure the goodness-of-fit of our model (ut) by taking the point-wise loss function (t) (1), (()) of the ith realized function y(t) and the predicted outcome ( given (+) and integrate over the functional domain to obtain the ith functional loss contribution E, i.e.
(00)))))) = f, (t), (()))t(())d
(6)
The overall objective, our empirical risk, is then given by the sum of (6) over all n observations. In practice, integrals in (6) are not available in closed form in general and the functional outcome is only observed finite grid. Therefore, similar to the discretization of the weight surface, let y (t,)) be the observations of the outcome for time points b4 E T,4 = 1, , Q. summarized for all time points as (6) = (tr),,, y (to)). Given dataset D with n observations of these functions, i... D = (x)(), y )= our overall objective then becomes
Q
(t).
min
where E(.) are integration weights.
As it less efficient represent f as actual function of t if Y is only given for fixed observed time points Egp we can adapt (5) to predict a Q-dimensional vector using
x
Rc T L
Panational
Punctiorol
Punctiorol Decoding
Fanztioraa Ourgat
Encoding
Panccional laquu
Functoraa Encoding
Pranctinaa Decodinu
Puncticnal Ouapuz
M M
Interpetible
Interperabee
D M
Dop Puxsiieal Networt
Newwak
(a) semi-str. FNN with shared en- and decoding.
(b) semi-str. FNN with shared and output layer.
Figure 2: Different semi-structured architectures.
5

---

where & = = (t))) The deep network À naturally yields the same sume-dimensional output when sharing the decoder. flexible deep model is used for X as in Fig. 2p. A can be defined with Q output units to match the dimension of > For one observation, the objective 7 can now be
written as a vector- vector-based loss function of y E IR lx and pz()) = x
E R I or,
when stacking these vectors, as loss function between two n x Q matrices.
3.3 Post-hoc orthogonaliizatin
Ankle
Hipp Adduction
Hipp Flextion
Knee
Subtalar
100
WDb Conrectionn
75
50
25
0-
AM
75
50
0-
100
00
200
100
w(s.b)
Figure 3: Estimated weight surfaces of the one functional shank gyroscope predictor X for the different joints (columns), before and after correcting with the ortogoonalization (rows). The color of each surface corresponds the weight predictor sensor signal time points (x-axis) is estimated to have on the tth time point (y-axis) of the outcome (ef. Fig. M Without correction (upper row), the interpretable model part is only incorrectly estimated no effect all attributed to After correction, distinct patterns for some of the time point combinations and joints are visible, e.g.. suggesting that an increased gyroscope value for early time points has negative influence (dark purple color) on the first half of the hip adduction moment (bottom row, second plot from the left).
Since - is potentially very expressive deep neural network that can also capture all functional linear dependencies modeled by X ,, the interpretable model part of the semi-structured FNN is not identifiable without further constraints. Non-identiffabiltyy. in turn, means that we cannot simply interpret the weights of the interpretable model part as is not clear how much these are diluted by the deep network part. This can, e.g., be seen in Figure which shows selected weight surfaces from our biomechanical application in Section 4 In this case, the deep network (an InceptionNet) captures almost all linearity of the structured part (upper row) and only after correction do we obtain the actual weight surfaces. We achieve this by extending the post-hoc orthogonalization [PHO; 49 for functional and outputs. First, we rewrite A
R, the sum of all J terms in 5 as
L T = > (P)(() 8 4 ve)c = Σ 20
x+ vec (1))))
5 vec
j-1
j=1
with (1) (1) 69 3))) E RIIK,-U where & denotes the Kronecker product, and 0, vec( (,)) E RKU This reformulation allows us to represent the functional interpretable model part as linear combination (2()) = [2, ( ..., ,,0] coefficients, and thereby apply PHO for semi-structured FNNs. matrices for the N
0 of basis products and their Let Let Se(N) and X(N) be the stacked Q data points in the training set D after stacking the l2(t) and X (t) for all Q time points of all n observed curves. Then we can obtain an identifiable and thus interpretable functional model part from the trained semi-structured FNN with basis matrix S2Ny parameters O, and deep network part ( by computing the following:
Ø 22 (NN
>TN) and ((x) PaIN) Tin
where 0 is the corrected basis coefficient vector, Stt ((N) the Moore-Penrose pseudoinverse of [[(N).
P sy is projection matrix projecting into the orthogonal complement spanned by columns of

---

[N} are the corrected deep network predictions orthogonal to the interpretable part. Using 03 from surfaces from 4 of the interpretable functional model part, which can be interpreted irrespective of the presence of deep network.
((N) and X
......., Oy] instead of the original 0,s will yield updated weights úij for the
3.4 Scalable implementation
When implementing semi-structured FNNs, x be chosen arbitrarily to the needs of the data modeler. Hence, we do not explicitly elaborate on a particular option here and assume an efficient implementation of this part. Instead, focus careful implementation of the structured model part to avoid unfavorable scaling as in other existing implementations (cf. Figure 4(b)). To mitigate this problem (in particular reducing the space complexity), we propose two implementation tricks. While these mainly reduce the complexity scaling w.r.t. . and R, implementing an FFR in neural network already yields an improvement by allowing cap the memory costs to fixed amount through the use of mini-batch training. In contrast, full-batch optimization routine of 7 as implemented in classic approaches (e.g.. (3 scales both with and Q as data transformed in formaf,,,, which can grow particularly fast for sensor data. For example, only n 1000 observations of functional response evaluated at Q = 1000 time points already results in data matrix of million rows. Classic implementations also scale unfavorably in terms of the number of features J and observed times points R for these functions. predictors are measured on the same scale, fitting an FFR model requires inverting a matrix of size (n- Q) (Eizz Kj) U), which becomes infeasible for larger amounts of predictors. Even setting up and storing this matrix be bottleneck.
the case where all functional
Array computation In contrast, when using the representation in ()) the basis matrices in s- and I-dimension (i.e. for in- and output) are never explicitly combined into larger matrix, but set up individually, and a network forward-pass only requires their multiplication with the weight matrix O. As the matrix & is the same for all J model terms, we can further recycle it for these computations.
Basis recycling In addition, if some or all predictors in X share the same time domain and evaluation points 81, we can save additional memory by not having to set up J individual bases Dy.
4 Numerical experiments
In the following, we show that our model is able to recover the true signal in simulation experiments and compares favorably to state-of-the-a methods both in simulated and real-world applications. As comparison methods, we use additive model formulation of the FFR model as proposed in [53] an FFR model based on boosting [5] and a deep neural network without interpretable model part. For the latter and the deep part of the semi-structured model, we use tuned Inception' T'mme architecture from the biomechanics literature (32). As FFR and boosting provide an automatic mechanism for smoothing, no additional tuning is required. In all our experiments, we use thin plate regression splines [60] but also obtained similar results with B-splines of order three and first-order difference penalties.
1
Hypotheses In our numerical experiments, we specifically investigate the following hypotheses:
Comparable performance with better scalability: For additive models without the deep part, our model can recover complex simulated function-on-functian relationships (Hla) with similar estimation and prediction performance as other interpretable models (H1b) while scaling better than existing approaches (H1e).
Favorable real-world properties: In real-world applications, our model is on par or better in prediction performance with current approaches (H2a), with a similar or better resemblance when generating output functions (H2b), and provides a much more meaningful interpretation than the deep MI.P (H2c). Further, we conjecture that our approach is better in prediction performance than its two (structured, deep) individual components alone (H2d).
7

---

Truth
Additiv Nodel
40
Nr. of Observations
Nr. of Predictors
Nr. of Time Points
30-
Memory Consumption (in MB)
4000
3000
R
Dcosting
Neural Newwork
40
40-
2000
30
20-
62
Value
44 -55
14
[ Additve Model
Boosting
E
Network
Network
(a) True weight surface w(s, I) used in the simulation study for large SNR along with estimation results of different methods.
(b) Memory consumption of different methods (colors) for different amounts of functional observations n (left), functional predictors (center), and time points (right).
Figure 4: Simulation study results.
4.1 Simulation study
Prediction and estimation performance (Hla,b) For the model's estimation and prediction performance, we simulate function cncciionoonffuctiion relationship based on a complex weight relationship t(s,t) between one functional predictor and functional outcome. Figure 4(a) depicts this true weight along with an exemplary estimation by the different methods. The estimation performance of the weight surface and all methods' prediction performance over 30 different simulation runs each ratio SNR {0.1,1] 1} and n E {320, 640. 1280} functional observations is
with a summarized in Figure 6 the Appendix. While the models perform on par most cases, find that the network works better for more noisy settings (SNR 0.1) and is not good as other methods for higher signal-to-nois ratio (as also shown Figure {.)). but with negligible differences. Overall, there are only minor differences in both estimation and prediction performance. This result suggests that our method works just as well as the other methods for estimating FFR.
Scalability (HIc) To investigate the scalability of all methods, we subsample the data from the Carmago study presented in the following subsection by reducing observations features J, and/or the number of observed time points R. More specifically, we investigate the memory consumption of the three implementations previously discussed (additive model, boosting, neural network) for n E (25, 50, 100} functional observations, Q = R E {25, 50, 100} observed data points, and E 21, 4} functional features. We restrict this study to rather small functional datasets as the memory consumption other methods is superlinear. This can also be seen in Figure 4(b) presented earlier, which summarizes the result of this simulation study and clearly shows the advantage of our implementation compared to existing ones form [ 53].
4.2 Real-world datasets
Our method being originally motivated by applications in the field of biomechanics, we now analyze two real-world datasets with biomechanical sensor data. In addition these applications in biome- chanics, we provide further applications to EMG-EEG synchronization, air quality prediction and hot water consumption profiles in Appendix C demonstrate the versatility of approach.
4.2.1 Fukuchi and Liew datasets (H2a)
The data analyzed in the first experiment is collection of three publicly available running datasets 14 28). A recent study of [32) used this data set to show the potential of deep learning approaches for predicting joint moments during gait (in total 12 different outcome features). The given data it provides only
set collection is challenging
490 functional samples but J = 27 partially highly correlated functional features. We follow the preprocessing steps of [32] and split the data into 80% training and 20% test data for measuring performance according to the authors' split indices.

---

As predictors we use the gait cycle of all available joint measurements, that is the 3D joint angle, velocity, and acceleration of the bilateral ankle, knee, and hip joints. Predictors in both training and test datasets are separately demeaned and scaled using only the information from the training set. As in the simulation studies, we run the additive model, Boosting, and our approach and compare the performance across all 6 different outcome variables. We use the relative and absolute root mean squared error (RMSE) as suggested in [46) and the Pearson correlation coefficient [20] to measure performance. In contrast to our simulation study, the additive model implementation of FFR cannot deal with this large amount functional features. We, therefore, split our analysis into one comparison of the additive model, an FNN, and semi-structured FNN (SSFNN) features, and a comparison of SSFNN against boosting (which has an inbuilt feature selection).
selected set of
Results in Table [ summarize test performances across all 12 different outcome types. We see that the (SS)FNN approaches perform equally well or better compared to the additive model on the selected set of features, and SSFNN also matches the performance of Boosting when using all features.
Table Median (and mean absolute deviation in brackets) of relative RMSE across all outcome responses in the Fukuchi and Liew datasets for different methods (columns) using either selected (sel.) or all features (all). The best method is highlighted in bold.
Add. Model (sel.)
FNN (sel.)
SSENN (sel.)
BBoostig (all)
SSFNN (all)
rel. RMSE (L)
0.36 (0.08)
0.31 (0.08)
0.30 (0.03)
0.30 (0.10)
0.25 (0.04)
4.2.2
Carmago data set (H2a-c)
The second data set is a publicly available dataset on lower limb biomechanics in walk- ing across different surfaces level-ground walking, treadmill walking, stair ascent, stair descent, ramp ascent, and ramp descent [7].
Improvementin Relativv MSE
0.15
0.10
Due to the size of the data set consisting of 21787 functional observations, Q = R = 101 observed time points, and J 24 predic- tors, it is not feasible to apply either addi- tive model-based FFR or boosting. We, there- fore, compare SSFNN to a (structured) neural- based FFR model and deep-only neural net- work. Based on the pre-defined 70/30% split for training and testing, we run the models for each of the 5 outcomes (joints) and com- pute the relative MSE difference, a functional R² analogon with values in (-o0. 1] (see Ap- pendix I. for definition). Table 2 shows the results averaged for all outcome types while Figure provides performance changes individually for the 5 different joints.
0.05
0.00-
Structured only
Deep only
Semi-structured
Ankle
Hip Flexlon
8 Subtalar
Hip Adduction
Knee
Figure 5: Comparison of performance improvements
(larger is better) in mean squared error (MSE) for different joints (outcomes).
We observe that the semi-structured model performs better than the deep model, which in turn is better than the structured model. Figure [ in the Appendix further depicts the predicted functions for all 5 outcomes using the models. While for some joints the prediction of the semi-structured and deep-only model is very similar, the performance for the subtalar joint is notably better when combining a structured model and the deep neural network form a semi-structured model.
Table 2: Mean performance (standard deviation in brackets) of the relative MSE difference (a value of 1 corresponds to the optimal model with error) across all five outcomes.
Structured
Deep
Semi-Structured
0.872 (0.094) 0.923 (0.065)
0.955 (0.030)

---

Discussion
Our proposed method an extension of semi-structured networks to cater to functional data and deals with the issue of identifiability by proposing functional extension of post-hoc orthogonalization. Using array computations and basis recycling the method also solves scalability issues present in existing methods. Our experimental results reveal that functional semiistrrutured networks yield similar or superior performance compared to other methods across varied biomechanical sensor datasets.
Novelty The approach proposed in Section 3 shows similarities to an autoeconder architecture. While functional autoencoder has been proposed in the past [18] and most recently by [6]], these studies focus on the representation learning (and reconstruction) of the same signal. Other papers focusing on function-on-funee the option to jointly train a structured model and deep network.
regression (e.g.. [34 43 588 suggest similar approaches to ours but without
Limitations and Future Research While our model shows good performance various datasets and in simulations, we believe that our approach can be further improved for applications on high- dimensional sensor data by incorporating appropriate sparsity penalties. Due to the functional nature of the data, feature selection would result great reduction of complexity and thereby potentially yield better generalization. Our general model formulation would further allow the extension to non- linear FFR model X (t) Ei Ss, (a)((s), t)ds using smooth functions fy in three dimensions (feature-, 8- and t-direction). Although this extension is still an additive model in the functional features, it is challenging to interpret the resulting additive effects.
Acknowledgments and Disclosure of Funding
We thank the four anonymous reviewers and the area chair for providing valuable feedback.
DR's research is funded by the Deutsche Forschungsgemeinschatt (DFG, German Research Foun- dation) 548823575. BL is funded by the Medical Research Council, UK (MR//013557/I) and Innovate UK (10093679). AS acknowledges support from SNSF Grant 200020 207367.
References
[1] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699-4711 2021.
[2] Philipp F. M. Baumann, Torsten Hothorn, and David Rügamer. Deep conditional transformation models. In Machine Learning and Knowledge Discovery in Databases (ECML-PKDD), pages 3-18. Springer International Publishing, 2021.
[3] Tobia Boschi, Matthew Reimherr, and Francesca Chiaromonte. A highly-efficient group elastic net algorithm with an application to function-on-se
regression. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9264-9277. Curran Associates, Inc., 2021.
[4] Sarah Brockhaus, Michael Melcher, Friedrich Leisch, and Sonja Greven. Boosting flexible functional regression models with a high number of functional historical effects. Statistics and Computing, 27(4):113-95 926, 2017.
[5] Sarah Brockhaus, David Rügamer, and Sonja Greven. Boosting functional regression models with fdboost. Journal of Statistical Softwrre Articles, 94(10):1500, 2020.
[6] Sarah Brockhaus, Fabian Scheipl, Torsten Hothorn, and Sonja Greven. The functional linear array model. Statistical Modelling, 15(3):29-330,, 2015.
[7] Jonathan Camargo, Aditya Ramanathan, Will Flanagan, and Aaron Young. A comprehensive, open-source dataset of lower limb biomechanies in multiple conditions of stairs, ramps, and level-ground ambulation and transitions. Journal of Biomechanics, 1 119:110320, 2021.
10

---

[8] Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. NODE-GAM: Neural generalized additive model for interpretable deep learning. In International Conference on Learning Representations, 2022.
[9] Yakuan Chen, Jeff Goldsmith, and R. Todd Ogden. Variable selection in function-on -n-ssallrr regression. Stat, 501)
[10] Heng-Tze Cheng. Levent Koc. Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
[11] Daniel Dold, David Rügamer, Beate Sick, and Oliver Dürr. Semi-structured subspace inference. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, PMLR, 2024.
[12] Emilio Dorigatti, Benjamin Schubert, Bernd Bischl, and David Rügamer. Frequentist uncertainty quantification in semi-structured neural networks. In Proceedings The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 1924-1941. PMLR, 2023.
[13] Deborah Falla, Valter Devecchi, David Jiménezz Grande, David Rügamer, and Bernard XW Liew. Machine learning approaches applied in spinal pain research. Journal of Electromyography and Kinesiology, 61: 102599, 2021.
[14] Reginaldo K Fukuchi, Claudiane A Fukuchi, and Marcos Duarte. A public dataset of running biomechanies and the effects of running speed on lower extremity kinematics and kinetics. PeerJ, 5:e3298, 2017.
[15] Sonja Greven and Fabian Scheipl. A general framework for functional regression modelling. Statistical Modelling, 17(1-2):1-35, 2017.
[16] William H Guss. Deep function machines: Generalized neural networks for topological layer expression. arXiv preprint urXiv:1612.0479) 2016.
[17] Trevor Hastie and Robert Tibshirani. Generalized additive models. Statist. Sci., 1(4):297-310, 1986.
[18] Tsung- Yu Hsieh, Yiwei Sun, Suhang Wang, and Vasant G Honavar. Functional autoencoders for functional data representation learning. In Proceedings of the SIAM Conference on Data Mining, 2021.
[19] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery,
2020.
[20] William Robert Johnson, Jacqueline Alderson, David Lloyd, and Ajmal Mian. Predicting athlete ground reaction forces and moments from spatio-temporal driven cnn models. IEEE Trunsuctions on Biomedical Engineering, 66(3):689-694, 2018
[21] Evandro Konzen, Yafeng Cheng, and Jian Qing Shi. Gaussian process for functional data analysis: The gpfda package for r, 2021.
[22] Lucas Kook, Lisa Herzog, Torsten Hothorn, Oliver Dür, and Beate Sick. Deep and interpretable regression models for ordinal outcomes. Pattern Recognition, 122:108263, 2022.
[23] Lucas Kook, Philipp Schiele, Chris Kolb, Daniel Dold, Marcel Arpogaus, Cornelius Fritz, Philipp Baumann, Philipp Kopper, Tobias Pielok, Emilio Dorigatti, and David Rügamer. How Inverse Conditional Flows Can Serve as Substitute for Distributional Regression. In Pro- ceedings of the Fourteenth Conference on Uncertainty Artificial Intelligence, Proceedings of Machine Learning Research. PMLR, 2024.

---

[24] Philipp Kopper, Sebastian Pölsterl. Christian Wachinger, Bernd Bischl, Andreas Bender, and David Rügamer. Semi-structured deep piecewise exponential models. In Proceedings of AAAI Spring Symposium on Survival Prediction - Algorithms, Challenges, and Applications, PMIR, pages 40-53, 2021.
[25] Philipp Kopper, Simon Wiegrebe, Bernd Bischl, Andreas Bender, and David Rügamer. Deep- pamm: Deep piecewise exponential additive mixed models for complex hazard structures in survival analysis. In Advances in Knowledge Discovery and Data Mining (PAKDD), pages Springer International Publishing, 2022.
249-261.
[26] Bernard Liew, Ho Yin Lee, David Rügamer, Alessandro Marco De Nunzio, Nicola R Heneghan, Deborah Falla, and David W Evans. A novel metric of reliability pressure pain threshold measurement. Scientific Reports, 11(1):6944, 2021.
[27] Bernard XW Liew, Susan Morris, Justin WL Keogh, Brendyn Appleby, and Kevin Netto. Effects of two neuromuscular training programs on running biomechanics with load carriage: a study protocol for a randomised controlled trial. BMC Musculoskeletal Disorders, 17:1-10, 2016.
[28] Bernard XW Liew, Susan Morris, and Kevin Netto. The effects of load carriage on joint work different running velocities. Journal of Biomechanics, 49 (9(14):3255-320 2016.
[29] Bernard X.W. Liew, David Rügamer, Deepa Abichandani, and Alessandro Marco De Nunzio. Classifying individuals with and without patellofemoral pain syndrome using ground force profiles Mhoflles-evelleee development of a method using functional data boosting. Gait & Posture, 80:90-95, 2020.
[30] Bernard XW Liew, David Rügamer, Alessandro Marco De Nunzio, and Deborah Falla. In- terpretable machine learning models for classifying low back pain status using functional physiological variables. European Spine Journal, 1945)118 2020.
[31] Bernard X.W. Liew, David Rügamer, Almond Stocker, and Alessandro Marco De Nunzio. Classifying neck pain status using scalar and functional biomechanical variables development of a method using functional data boosting. Gait & Posture, 76:146-150 150,,,0022200 2020.
[32] Bernard X.W. Liew, David Rügamer, Xiaojun Zhai, Yucheng Wang, Susan Morris, and Kevin Netto. Comparing shallow, deep, and transfer learning in predicting joint moments in running. Journal of Biomechanics, 129: 10820, 2021.
[33] Yu Liu, Shi-Min Shih, Shi-Liu Tian, Yun-Jian Zhong, and Li Li. Lower extremity joint torque predicted by using artificial neural network during vertical jump. Journal of biomechanics, 42(7):906-9)) 911, 2009.
[34] Ruiyan Luo and Xin Qi. General nonlinear cunction-on-function regression via functional universal approximation. Journal of Computational and Graphical Statistics, 33(2):578-588 2024.
[35] Mark J Meyer, Brent A Coull, Francesco Versace, Paul Cinciripini, and Jeffrey S Morris. Bayesian -unetion-on-funcctoon regression for multilevel functional data. Biometrics, 71(3):563- 574, 2015.
[36] Gregory D Myer, Kevin R Ford, Stephanie Di Stasi, Kim D Barber Foss, Lyle J Micheli, and Timothy E Hewett. High knee abduction moments are common risk factors for patellofemoral pain (pfp) and anterior cruciate ligament (acl) injury in girls: is pfp itself a predictor for subsequent acl injury? British journal of sports medicine, 1--12, 2015.
[37] Felix Ott, David Rügamer, Lucas Heublein, Bernd Bisch and Christopher Mutschler. Joint classification and trajectory regression of online handwriting using a multi-task learning ap- proach. In Proceedings of the IEFF/CVF winter conference applications of computer vision, pages 266-276, 2022.
[38] Carlos Misael Madrid Padilla, Daren Wang, Zifeng Zhao and Yi Yu. Change-point detection for sparse and dense functional data in general dimensions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
12

---

[39] Jan Priesmann, Lars Nolting, Christina Kockel, and Aaron Praktiknjo. Time series of useful energy consumption patterns for energy system modeling. Scientific Data, 8(1):148, 2021.
[40] Xin Qi and Ruiyan Luo. Nonlinear function-on-fuuction additive model with multiple predictor curves. Stafistica Sinica, 29(2):199-739, 2019.
[41] Filip Radenovic, Abhimanyu Dubey, and Dhruv Mahajan. Neural basis models for interpretabil- ity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[42] J. O. Ramsay and B. W. Silverman. Functional Data Analysis. Springer Series in Statistics. Springer, New York, NY, 2005.
[43] Aniruddha Rajendra Rao and Matthew Reimherr. Modern non-linear function-on-fuuction regression. Statistics and Computing, 33(6);130, 2023.
[44] Aniruddha Rajendra Rao and Matthew Reimherr. Nonlinear functional modeling using neural
networks. Journal of Computational and Graphical Statistics, 0(0):1-10, 2023.
[45] Philip T. Reiss, Jeff Goldsmith, Han Lin Shang, and R. Todd Ogden. Methods for Scalar-on- Function Regression. International Statistical Review, 85(2):228-249, 2017.
[46] Lei Ren, Richard Jones, and David Howard. Whole body inverse dynamics over a complete gait cycle based only on measured kinematics. Journal of biomechanics, 41(2))2275 41(12):7500-275, 2008
[47] Rosie E Richards, Michael Skipper Andersen, Jaap Harlaar, and JC Van Den Noort. Relationship between knee joint contact forces and external knee joint moments in patients with medial knee osteoarthritis: effects of gait modifications. Osteoarthritis and Cartilage, 2639):120--1214, 2018.
[48] Fabrice Rossi, Bricuc Conan-Guez, and François Fleuret. Functional data analysis with multi layer perceptrons. In Proceedings of the 2002 International Joint Conference Neural Networks. IJCNN'02 (Cat. No. 02CH37290), volume 3, pages 2843-2848. IEEE, 2002.
[49] David Rügamer. A New PHO-rmula for Improved Performance of Semi-Structured Networks. In International Conference on Machine Learning, 2023.
[50] David Rügamer. Sarah Brockhaus, Kornelia Gentsch, Klaus Scherer, and Sonja Greven. Boost- ing factor-specific functional historical models for the detection of synchronization in bioclectri- cal signals. Journal of the Royal Statistical Society: Series C (Applied Statistics), 67()):621-62,, 2018.
[51] David Rügamer, Chris Kolb, Cornelius Fritz, Florian Pfisterer, Philipp Kopper, Bernd Bischl, Ruolin Shen, Christina Bukas, Lisa Barros de Andrade Sousa, Dominik Thalmeier, Philipp Baumann, Lucas Kook, Nadja Klein, and Christian L. Müller. deepregression: flexible neural network framework for semi-structured deep distributional regression, 2023.
[52] David Rügamer, Chris Kolb, and Nadja Klein. Semi-structured distributional regression. The American Staristician, pages 12 2023.
[53] Fabian Scheipl, Ana-Maria Staicu, and Sonja Greven. Functional additive mixed models. Journal of Computational and Graphical Statistics, 24(2):477500) 2015.
[54] Jian Qing Shi and Taeryon Choi. Gaussian process regression analysis for functional data. CRC Press, 2011.
[55] Barinder Thind, Kevin Multani, and Jiguo Cao. Deep learning with functional inputs. arXiv preprint arXiv:2006.09599, 2020.
[56] Megan E Toney and Young- Chang. The motor and the brake of the trailing leg in human walking: leg force control through ankle modulation and knee covariance. Experimental brain research, 234(10):30113322 2016.
13

---

[57] José L Torrecilla and Alberto Suárez. Feature selection in functional data classification with recursive maxima hunting, Advances in Neural Information Processing Systems, 29, 2016.
[58] Qiyao Wang, Haiyan Wang, Chetan Gupta, Aniruddha Rajendra Rao, and Hamed Khorasgani.
A non-linear function-on-functin model for regression with time series data. In 2020 IEEE International Conference on Big Data (Big Data), pages 232-239. IEEE, 2020.
[59] Shuoyang Wang, Wanyu Zhang. Guanqun Cao, and Yuan Huang. Functional data analysis using deep neural networks. WIREs Computational Statistics, 16(4))e70001, 2024.
[60] Simon N Wood. Thin plate regression splines. Journal of the Royal Statistical Society Series B: Statistical Methodology, 65(1):95-114, 2003.
[61] Sidi Wu, Cédric Beaulac, and Jiguo Cao. Functional autoencoder for smoothing and representa- tion learning. arXiv preprint arXiv:2401.09499 2024.
14

---

A Further details
A.1 Functional R definition
Let n be the number of observed curves indexed with i. Given the ith observation of function output signal (t) for & E T with functional domain T and corresponding predicted value f (t), we define the functional R
a
- (1)) (t)(
xfx . .1 /==1.... m)
(10)
Since - 2 0, the nominator is the smallest for y(t) ptt). in which case x = In contrast, if the prediction μ is μ 0, x = 0. For models that perform worse than the constant zero predictor (t))((t) -r (t))² dt can be larger than ()() yielding negative
nominator and hence negative x values.
A.2 Penalization scheme
In order to enforce smoothness of estimated functional relationships in X+ in both s- and t-direction, we can employ quadratic smoothing penalty. For regression splines with evaluated bases Φ and V, smoothness is usually defined by some form of difference penalties, which we define to encoded in the matrices P. and P., respectively. For the array computation, we want to penalize all differences in s-direction, which can be done by using
pen, = vec(O) T(u & P.) = ve (P.0)) 1.
Analogously, the t-direction can be penalized as
pen, vec ( (e)) vec(( 1
Putting both penalties together, we obtain the total penalty for the array product as pen, + pen,
B Additional results
B.1
Simulation study
F shows the estimation error of the weight surface using different established methods and our proposed implementation (Neural Network).
All methods perform equally well with minor differences in the weight surface estimation. As expected, neural networks perform better small signal-to-noise (SNR) settings due to their implicit regularization when trained with stochastic gradient descent and using mini-batches, whereas for large SNR, the full batch routines implemented by boosting and additive models perform slightly better. As becomes apparent in Figure 4(a) these differences negligible in practice.
B.2 Application
Figure 7 gives an excerpt of the estimated weight surfaces that provide insights into how the relation- ship between input features (rows; here acceleration measures from the x-dimension) and outcome joints is estimated.
For example, the trunk acceleration (last row in Figure having short negative and almost instantaneous effect (dark circle) on the ankle and hip adduction moment (first and second column) is reasonable result from a biomechanical perspective.
We can further plot the predicted curves by employed methods (Figure 3 and compare the results with the ground truth. While for some joints the prediction of the semi-structured and deep-only model very similar, the performance for the subtalar joint notably better when combining structured model and the deep neural network to form semi-structured model.
15

---

SNR =.11
SNR =1
2.0
Confficientt Surfice
1.0
relative MSE
0.5
Prediction per performmnnc TraTnc
0.44
0.42
0.40
320
640
1280
320
640
1280
Number observations
Boasting
Additive Madel
E Neural Network
Figure 6: Surface estimation and prediction performance (top and bottom row, respectively) using different optimization methods (colors) for different (X-axis) and signal-to-noise ratios (SNR; columns).
Adbuctin
Hip Flexion
Knee
Suptalar
75
50
15
100S
0
100
WALL
25
100
50
25
25 75 751000
25
75
1000
25
75
1000
25
50
75
3000
50
100
W1s,0
Figure 7: Estimated weight surfaces for different accelcrometer predictors (rows) for the different joints (columns). The color of each surface corresponds the weight predictor sensor signal time points (x-axis) estimated to have the (th time point (y-axis) of the outcome.
As requested by an anonymous reviewer, we have also compared our method against the approach suggested 332. Compared to 32 we use different pre-pre pre-procsssing of functional predictors which is more in line with main results. We then run their best model once using a deep-only variant once using a semi-structured model. The ReIRMSE results are given in Table 3 showing that the semi-structured extension works similarly well or in most cases better.
Comparison with 32
C Additional experiments
In order demonstrate our method's applicability beyond biomechanical applications, we run several additional experiments. Datasets come from different fields of application and have different sizes
16

---

Strudured
Deep
Semi-ditussueed
Truth
Ho Plaston
-1
0.5
0.0
0.5
-1.0
(Predictad) Value
0.25
0.00
0.50
75
1000
75
100
1000
Figure 8: Comparison of predictions of the structured, deep, and semi-structured model as well as the true sensor data (columns) for different joints (rows).
Table 3: Comparison of a plain deep and semi-structurred approach, where the former is based 32 and the latter extension SSN.
Mcasurement
Deep
Semi-str.
ankle (dim 1)
0.261
0.212
ankle (dim
0.247
0.208
ankle (dim
0.423
0.359
com (dim
0.054
0.048
com (dim 2)
0.275
0.275
com (dim 3)
0,077
0,078
hip (dim 1)
0.342
0.314
hip (dim 2)
0.301
0.300
hip (dim 3)
0.376
0.303
knee (dim 1)
0.281
0,225
knee (dim 2)
0.318
0.270
knee (dim 3)
0.405
0.383
(number of observations, number of functional predictors, number of observed time points). We also
use these examples to 1) compare the two different architectures suggested in Fig. ((a) and Fig. 2(b)
2) compare against non-linear FFR implementation from 40
For all the additional experiments, the deep network version from Fig. 2(a) uses the same encoding and decoding basis for both X and A (using K, 20 basis functions for the s-direction and U = 20 basis function for the t-direction). The trunk of the deep network part X is defined as fully connected neural network with 100 neurons each, followed by dropout layer with dropout rate of 0.2 and batch normalization layer. The deep network version from Fig. 2(b) uses the same architecture, but instead of using a shared encoding and decoding part, it simply concatenates the inputs and processes the three-dimensions have the correct dimensions. dropout rate, the purpose of these additional experiments is solely to demonstrate that the (SS)FNN approach yields competitive results with only the structured part while the deep part can further help to improve predictive performance.
object until the last layer, which is then reshaped to have not tuned this architecture, the number layers, neurons, or
17

---

C.1 Predicting EMG from EEG signals in cognitive affective neuroscience
The dataset analyzed in [50] contains 192 observations of three different EMG signals from facial muscles used different outcomes (similar to the different joint moments) and up to 64 EEG signals measured at different parts of the brain. Both EMG and EEG signals are measured at 384 equidistant time points. Following [50], choose the EXG4-EXGS signal outpu signal and the Fz-, FCz-, POz-, and Pz-electrode as input signals. Based on 10 train/test-splits, we obtain the following table:
Table 4: Model performance comparison
Model
Rel. RMSE (4)
Correlation (1)
Boosting
0,195 (0.137)
0.184 (0. (3.130)
FNN
0.190 (0.132)
0.171 (0.111)
SSFNN (a)
0,210 (0.152)
0.201 (0.162)
SSFNN (b)
0.192 (0.177)
0.184 (0.287)
Results confirm again that our FNN implementation leads similar values Boosting, while including a deep neural network part can improve the performance.
C.2 Predicting air quality from temperature curves
The dataset analyzed in 40 contains 355 observations of daily NO2 measurement curves (i.e., the functional domain is the hours to 24) together with four other functional pollutants as well as the temperature and relative humidity. The goal is to predict the NO2 concentration using other pollutants, the temperature, and the humidity. The proposed approach in [40] uses a non-linear cnaction-on-funciin regression (FFR) which we compare against. The results are given in the following table:
Table 5: Comparison of Model Metrics
Model
Rel. RMSE (l)
Correlation (f)
Non-lincar FFR
1.000 (0.000)
0.878 (0.024)
FNN
1.000 (0.000)
0.908 (0.009)
SSFNN (a)
0.988 (0.024)
0.880 (0.052)
SSFNN (b)
0.991 (0.042)
0.897 (0.047)
Results suggest that the given task is rather easy and linear FFR model is already sufficient obtain good prediction performance. Hence FNN outperforms both the non-linear FFR model and the two semi-structured approaches
C.3 Predicting hot water consumption profiles in urban regions
The dataset taken from [39 contains hourly hot water consumption profiles (i.e., the functional domain is again 1 to 24) which are available for different administrative districts. The goal of the analysis predict the hot water consumption of specific region given multiple (here 6) far-distant administrative districts. The results are given in the following table:
Table 6: Model Performance Evaluation
Model
Rel. RMSE (L)
Correlation (1)
Additive Model
0.814 (0.006)
0.761 (0.006)
FNN
0.810 (0.008)
0.754 (0.010)
SSFNN (a)
0.887 (0.052)
0.799 (0.042)
SSENN (b)
0.893 (0.077)
0.803 (0.037)
Similar to the first dataset, the FNN achieves similar performance as the reference model while including deep neural network improves the performance.
18

---

D Experimental details and computational environment
D.1 Experimental details
In all experiments, we use the Adam optimizer with default hyperparameters. No additional learning rate schedule was used. The batch size, maximum number of epochs and early stop- ping patience was adjusted depending on the size of the dataset. A prototypical implemen- com/
tation is available as an add-on package of deepregression [51 at adweeeddaaddiiieee wdwwiive-eerrrinnngg///o/oeea//e
D.2 Computational environment
All computations were performed on a user PC with Intel(R) Core(TM) 17-8665U CPU @ 1.90GHz, cores, 16 GB RAM using Python 3.8, R 4.2.1, and TensorFlow 2.10.0. Run times of each experiment do not exceed 48 hours.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims reflect the contribution and scope. Experimental results support the theoretical claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the Discussion Section. Computational efficiency is part of the numerical experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes].
Justification: While mathematical details are not formulated as theorems or similar, the nota- tion and background section defines the necessary basis for further theoretical elaboration Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The algorithm's implementation is given in detail, allowing readers recon- struct our implementation. The details provided in the numerical experiments should further allow readers to reproduce simulation studies and
experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code is attached as supplementary material. Open access data is referenced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
cublic/guidess httpii/ggiiiee//C/euubbiisee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips cc/ wttppppllllllppie http://ooooooooosssooooooooeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The information is given in the respective section and in Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Simulation studies are repeated multiple times and report error bars; for real- world data, replications are done for different data scenarios, also allowing report standard deviations.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is given in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpps:////euriii nwwrips....../. wwweerps..cppublllllee hccppullic/tthhccccudee........e
Answer: [Yes]
Justification:
We have read the code of ethics. There are no ethical concerns or similar.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While biomechanical applications often involve data from humans, our work does not deal with this type of data in a way that would imply any particular socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not provide data. Models themselves do not pose a risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets used are referenced and code is CC-BY licensed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
No crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "fqrTPdT7X3", "reviewer": "Reviewer_DyRG", "summary": "The paper proposes a hybrid approach that combines the benefits of neural networks with those of more structured models (generalised additive models). They show benefits on real and simulated data.", "questions": "See weaknesses. I am willing to increase my score if these are addressed.", "limitations": "Adequately addressed", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed idea makes sense and is novel AFAIK\n- The empirical results show good performance compared to either purely deep models or purely structured models\n- The description of the method in sections 3.1 to 3.4 is clear.", "weaknesses": "- How much tuning has been done for the pure deep-only network baseline from Section 4.2.2? I would be a bit worried that the findings are very dependent on the choice of architecture of this model? E.g. does it have any increased capacity over the deep part of the semi-structured network? And does this architecture have suitable inductive biases for fitting linear functions (maybe residual connections which add a linear transformation of the input to the output)?\n- The presentation could be made clearer in a couple of areas. In particular, (1) Equation 3 is the first time that the “double function call” notation of e.g. $\\lambda^-(X)(t)$ is used. Having a double function call makes sense given that there is a function which takes X as input and outputs another function. However it is not clear to me why this is then not used in Equation 1, where you could presumably replace $\\mu(t)$ with $\\mu(X)(t)$? And e.g. in the last part of Section 2.2, why write $h^{(L)}(t)$ instead of $h^{(L)}(X)(t)$? Would it be possible to make this clearer? And also (2), when functional data analysis is introduced in Section 1 and 2, it would be helpful if an example application was described", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer raises two points that we answer below. We thank the reviewer for these thoughtful comments, but in both cases would like to point out that these \"weaknesses\" have already been addressed in our original submission. \n\n-----\n\n> How much tuning has been done for the pure deep-network baseline? I would be a bit worried that the findings are very dependent on the choice of architecture of this model?\n\nWe agree that this is an important aspect that needs to be taken into account. Fortunately, some already-tuned architectures are widely adopted in biomechanics. As we write in the beginning of Section 4 “we use a tuned InceptionTime [16] architecture from the biomechanics literature [ 24]. As FFR and boosting provide an automatic mechanism for smoothing, no additional tuning is required.”. In other words, we don’t tune any of the models. While tuning might certainly benefit one or the other model, we think that it is difficult to make a perfectly fair comparison between all methods. In this light, we adopt a strategy that is closest to what practitioners in the field would likely do — use the tuned Inceptionnet with the hyperparameters found to be optimal in previous studies and take this architecture without further modifications. \n \nWe will make this more clear in a revised version of the manuscript.\n\n> The presentation could be made clearer. E.g. when $\\lambda^+$  and $\\lambda^-$  are introduced, they should be more clearly defined.\n\nWe thank the reviewer for this comment. We have defined $\\lambda^+$ in the paragraph right after it was introduced ($\\lambda^+(t) = \\sum_{j=0}^J \\lambda_j^+(t)$ with $\\lambda_j^+(t) = \\int_{\\mathcal{S}_j} w_j(s,t) x_j(s)ds)$ and in the following paragraph (“Deep model part”) discuss choices for $\\lambda^-(t)$. \nWe are happy to include further details if there is anything else the reviewer is missing.\n\n-----\n\nPlease let us know whether we have addressed all weaknesses and whether these answers are sufficient to re-evaluate your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gK0zSXtezf", "reviewer": "Reviewer_SfMu", "summary": "An extension to semi structured networks is introduced that combines a traditional linear and interpretable model component $\\lambda^+$ with a flexible $\\lambda^-$ neural network component to better approximate relations in biomechanical application. The model performs equally to traditional methods but requires less memory and remains interpretable thanks to a post-hoc orthoganalization.", "questions": "1. What are the additive structures you are referring to in l15, can you provide some examples?\n2. What is the proportional contribution of $\\lambda^+$ and $\\lambda^-$ to the overall solution? How is guaranteed that $\\lambda^-$ is not solving the entire problem solely, thus loosing all interpretability? The orthogonal formulation seems to take a step into that direction, but I am not sure what the orthogonal formulation actually does, and if the patterns revealed after its application are helpful and realistic.\n3. What are the integration weights $\\Xi$ concretely in line 182 and what are they required for (can they be dropped for simplicity yielding a conventional loss function)?\n4. The orthogonalization approach in Section 3.3 appears very beneficial, yet, frankly, I could not follow it as the math is quite abstract. Can you share an intuition what the orthogonalization implements and how it effects $\\lambda^-$ to be less dominant, if so? Is it some kind of regularization to minimize its contribution (or how would, e.g., L2 regularization compare)?\n5. Relating to Figure 4, how does runtime compare for the three implementations for $\\lambda^-$? Given the memory is capped for the batched neural network, it should take multiple evaluations to process the same amount of data. How does the memory compare if the other methods are implemented in a batched fashion too?\n6. Can you share an intuition of why the neural network performs worse in high SNR regimes? I'd expect a neural network be superior in all conditions, given it is sufficiently large and regularized appropriately.\n7. Could you include the results from reference [24] into your results in Section 4.2.1 to allow a comparison between your and an established method? It is crucial to assess how your approach performs in comparison to existing methods.", "limitations": "Limitations are addressed by the authors.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "_Originality:_ The introduced method nicely combines established approaches and compares it to some, but not to all relevant related articles (see details under weaknesses).\n\n_Quality:_ The submission is technically sound in that most claims are well supported by experimental results; some exceptions are listed under weaknesses. The paper is nicely structured but, for my taste, lacks an intuitive examples and explanations to nudge and inform readers that are unfamiliar with the topic. An exhaustive number of experiments is conducted on several problems, both on synthetic and real-world data, allowing a quantitative assessment of the introduced method.\n\n_Clarity:_ Although authors could add more intuition about the effect and function of certain (mathematical) concepts, the manuscript is well organized and clearly written. In particular, the authors very nicely outline their hypothesis and refer to them in the results section. Apart from some aspects detailed in the weaknesses, the manuscript informs the reader adequately. The submission is complemented with code, yet the data to reproduce experiments is lacking (due to space constraints).\n\n_Further comments_:\n- Figure 2 nicely illustrates the two proposed implementation options for hybrid models.\n- Figure 3 is helpful, but I find it hard to understand entirely. What exactly does the x-axis represent and are the values normalized to [0, 100] or what is the data range and what does it represent? The last sentence in the caption is not clear to me, i.e., when stating \"... early time points $s$\". This is probably formulated suboptimally, as $s$ is reported to relate to a sensor signal and $t$ to time.", "weaknesses": "_Significance_: Even though the authors emphasize their method's superiority in lines 144-146, the performance improvement over existing approaches is rather mediocre and does not seem significant, in particular when considering the comparably large standard deviations. The error distributions of the different methods seems to overlap to large extend. In this vein, it would be appreciated if error bars were reported in Figure 5. In that figure, you may consider to flip the y-axis to make it more intuitive and report the relative error reduction over the baseline.\n\n_Further comments_:\n- It took me very long to get a grasp of the concept of functional data/analysis/regression. In fact, until the very end of the manuscript, I did not fully understand the actual task at hand and permanently felt somewhat lost. Authors may give a quick practical example of functional analysis early to catch readers sooner. Even the example in lines 109--123 did not resolve my uncertainty about where such processes find application. I thus suggest to add one clear and illustrative practical example and spend more effort to introduce the problem at the beginning of the introduction rather than starting with related work.\n- While novelty for functional data analysis seems reasonable, I cannot find a substantial contribution to the machine and deep learning community (given the paper is submitted to a top tier ML conference). Even though I like the paper's argumentation and substantiation and perceive it as a generally sound work, the manuscript might better be suited to a journal that deals with functional analysis or with biomechanics.\n- Claim in l265 not supported with experimental results. Please provide error and dispersion metrics (such as in Table 1 and 2) to allow a quantitative comparison of the methods.\n- Typo in line 331: Either \"reveal that\" or \"yields\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thorough review of our manuscript. \n\n-----\n\n### Weaknesses\n\n> improvement [...] rather mediocre [...] would be appreciated if error bars were reported \n\nWe only report results for a single train/test split as the application fixes this. To still provide a measure of uncertainty, we show individual joint performances in Fig. 5, and Tab. 2 summarizes findings with standard dev., showing a significant improvement for semi-structured models. However, it's not only about performance improvement but also being able to quantify the explained proportions by the structured and deep part.\n\n> I cannot find a substantial contribution to the [ML/DL] community [...] better be suited to a [biomechanics] journal\n\nWe thank the reviewer for the comment but politely disagree. 1) The paper is too technical for a biomechanics journal (a manuscript focusing on interpretation will be submitted to such a journal). 2) We believe that projects with a motivating application should be valued at least as much as those proposing methods without real applications. 3) We also think that we provide a substantial contribution to the ML/DL community as functional data is well represented at NeurIPS (e.g., Boschi et al., 2021; Gonschorek et al., 2021; Madrid Padilla et al., 2022), ICML (e.g. Yao et al., 2021; Heinrichs et al., 2023), etc., and our paper advances both orthogonalization and semi-structured models, also discussed in these communities (e.g., Lu et al., 2021; Vento et al., 2022; Rügamer et al., 2023, 2024).\n\n> Claim in l265 not supported with experimental results. Please provide error and dispersion metrics \n\nWe provide these metrics as boxplots in Fig. 6 (Appendix). \n\n### Questions \n\n> What are the additive structures [...] provide some examples\n\nWe wrote: “For example, neural additive models [...] such as generalized additive models”. We will revise the text to make these examples more clear.\n\n> proportional contribution of  $\\lambda^+$  and $\\lambda^-$ [...] How [..] $\\lambda^-$ is not solving the entire problem\n\nThe proportional contribution will depend on the application. $\\lambda^-$ is not solving the entire problem due to the orthogonalization (see answer below).\n\n> What are the integration weights $\\Xi$ concretely [...] can they be dropped\n\nThe integration weights are used to approximate the integral (trapezoidal Riemann weights in our paper). For time points on an equidistant grid, one could potentially drop them.\n\n> intuition what the orthogonalization implements [...] Is it some kind of regularization \n\nThe orthogonalization process subtracts everything that could have been explained through $\\lambda^+$ from $\\lambda^-$ and adds it to $\\lambda^+$, ensuring that the two parts are orthogonal. This concept can be understood by analogy to applying multiple regression models. First, generate predictions by regressing the actual outcome on the features via $\\lambda^+$ and $\\lambda^-$. Next, regress these predictions on $\\lambda^+$ to determine the portion that can be explained by $\\lambda^+$. The residual term, representing what cannot be explained by the structured part, will be orthogonal to this part. The orthogonalization employs this approach using appropriate projection matrices and can be applied whenever the structured part is linear in the coefficients. While it may not be straightforward to see how this works for our model, Section 3.3 demonstrates how to do this using vector operations. It is not a regularization in the classical sense, as it exactly enforces orthogonality between the two parts.\n\n> Relating to Figure 4, how does runtime compare for the three implementations for $\\lambda^-$? \n\nFigure 4 investigates whether “without the **deep part**, [...] can [we] recover complex simulated function-on-function relationships [...] while scaling better than existing approaches”. Hence, no deep part was included in this experiment. Also, neither the boosting nor the additive model approach would allow the incorporation of the $\\lambda^-$ part. \n\n> How does the memory compare if the other methods are implemented in a batched fashion\n\nThis would result in improved memory consumption w.r.t. number of observations, however, not w.r.t. the number of functional predictors.\n\n> intuition of why the neural network performs worse in high SNR regimes? \n\nFull-batch optimization (additive model, boosting) is beneficial when there is less noise in the data (and vice versa, the neural network's stochastic optimization induces regularization and might thereby better deal with noise).\n\n> Could you include the results from reference [24]\n\nAs [24] used a different processing of functional predictors, we first had to adapt their code. We then ran their best model once using a deep-only variant and once using a semi-structured model. RelRMSE results are as follows:\n\n| | Deep | Semi-str. |\n|-|-|-|\n| ankle (dim 1) | 0.261 | 0.212 |\n| ankle (dim 2) | 0.247 | 0.208 |\n| ankle (dim 3) | 0.423 | 0.359 |\n| com (dim 1)| 0.054 | 0.048 |\n| com (dim 2) | 0.275 | 0.275 |       \n| com  (dim 3) | 0.077 | 0.078 |\n| hip (dim 1) | 0.342 | 0.314 |\n| hip (dim 2) | 0.301 | 0.300 |\n| hip (dim 3) | 0.376 | 0.303 |\n| knee (dim 1) | 0.281 | 0.225 |\n| knee (dim 2) | 0.318 | 0.270 |\n| knee (dim 3) | 0.405 | 0.383 |\n\n-----\n\n### References\n\n- Boschi et al., 2021. A highly-efficient group [...] NeurIPS.\n- Gonschorek et al., 2021. Removing inter-experimental [...] NeurIPS.\n- Heinrichs et al., 2023, Functional Neural Networks [...] ICML.\n- Lu et al., 2021. Metadata normalization. CVPR.\n- Madrid Padilla et al., 2022. Change-point detection [...] NeurIPS.\n- Rügamer, 2023. A new PHO-rmula [...] ICML.\n- Rügamer, et al., 2024. Generalizing Orthogonalization [...] ICML.\n- Vento et al., 2022. A penalty approach [...] MICCAI.\n- Yao et al., 2021. Deep learning for functional [...] ICML.\n\n-----\n\nPlease let us know whether we have addressed all points and whether these answers are sufficient to re-evaluate your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jVtYe8AKr", "reviewer": "Reviewer_9i2N", "summary": "In this paper, the authors develop a semi-structured model for functional data, summing an interpretable linear model with a more general nonlinear functional neural network. The authors validate the improved performance of the combination relative to the individual components on a variety of biomechanics datasets. The authors also discuss how to preserve the interpretability of the semi-structured model with a linear weight surface by projecting the neural network predictions into the subspace of the linear model.", "questions": "One aspect I did not see mentioned in the paper was the actual choice of basis functions for encoder and decoder. These should certainly be specified along with any justification or tuning of hyperparameters involved (e.g. bandwidth parameters?).\n\nFig2a and 2b present two different potential options for constructing this semi-structured model, might it be worth evaluating the performance of structure a)? Would we expect the performance to be similar, what are the drawbacks?\n\nWhen fitting the parameters of the linear model, it is mentioned the large computational cost. I did not find the explanation of how the scalable implementation circumvents this to be particularly clear. Also, even for solving the linear system could one not use more sophisticated matrix free methods such conjugate gradients with the kronecker product matrix, or for example SVRG which would converge quickly even when the number of data points and features is large?", "limitations": "I think the limitations are sufficiently discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The approach provides a natural adaptation of semi-structured models to functional data and is worthy of detailed investigation. Experiments performed on synthetic and real data are reasonably thorough and help validate the claims on efficacy and interpretability. The ability to retain the linear model interpretability using the projection is interesting and specific to the setting.", "weaknesses": "The paper accomplishes what it sets out to do fairly well, though the contributions of the paper through providing evidence for new phenomena, novel insights, or new algorithmic ideas seem much more limited.\n\nIn terms of the structured part of the model, the linear regression with the functional encoder and decoder bases for function -> function regression, I assume that this approach has been used before but I did not see it in any of the explicit references. Can the authors comment on whether this component is original to their work or whether this component is building on what already exists in the literature?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and pointing out ways to improve our manuscript. Below we address the mentioned weaknesses and questions.\n\n-----\n\n### Weaknesses\n\n> Can the authors comment on whether this component is original to their work [...]\n\nWe thank the reviewer for bringing this up. While a functional autoencoder has been proposed in the past (e.g., Hsieh et al., 2021) and most recently by Wu et al. (2024), these studies focus on the representation learning (and reconstruction) of the same signal. Other papers focusing on function-on-function regression (e.g., Luo and Qi, 2024; Rao and Reimherr, 2023; Wang et al., 2020) suggest similar approaches to ours but without the option to jointly train a structured model and a deep network.\n\nWe also believe that our idea to encode and decode signals using a general basis representation is to some extent novel. However, any mapping from function to scalar values and back (such as the functional PCA in Wang et al., 2020) can potentially be interpreted as such an encoding strategy. We will clarify this distinction and our contribution, and we thank the reviewer for highlighting this point.\n\n-----\n\n### Questions\n\n> [...] choice of basis functions for encoder and decoder. These should certainly be specified [...]\n\nWe thank the reviewer for pointing this out. In our experiments, we used thin plate regression splines but also obtained similar results with B-splines of order three and first-order difference penalties. We will provide this information in an updated version and will add a short discussion of possible other options with their pros and cons.\n\n> Fig2a and 2b present two different potential options for constructing this semi-structured model, might it be worth evaluating the performance of structure a)? Would we expect the performance to be similar, what are the drawbacks?\n\nThis is indeed an important question. We have included such a comparison in Appendix C in the original version of the paper, where we “compare the two different architectures suggested in Fig. 2(a) and Fig. 2(b)”. Using different real-world datasets, we found that both architectures perform quite similarly. In practice, our biomechanics project partners might be in favor of option b) as it allows them to use an established network for the deep model part and gives more flexibility, whereas from a practical point of view, option a) will be potentially more parameter-sparse and allows to jointly learn the function embedding, which in turn increases interpretability. \n\n> When fitting the parameters of the linear model, it is mentioned the large computational cost. I did not find the explanation of how the scalable implementation circumvents this to be particularly clear. \n\nWe thank the reviewer for this question. In Section 3.4, we have provided an explanation of how our implementation solves this problem with a focus on the space complexity (memory costs). In particular,\n\n- memory costs are reduced by using mini-batch training and not having to cast our model into long-format as done by classical approaches. Our approach could indeed then be combined with the SVRG approach by Johnson and Zhang as the reviewer suggested.\n- In contrast to classical implementations, we rely on array computations (as e.g. given in Equation 8 in the paper). Using the array model formulation, we never have to explicitly construct the Kronecker product of the basis matrices, which saves additional memory.\n- In addition to the two previous solutions, we recycle the basis functions in the s-direction, which is non-trivial for other software implementations. In the neural network, however, this “simply” boils down to having multiple connections to the same matrix object in the computational graph.\n\n\n-----\n\n### References\n\nHsieh, T. Y., Sun, Y., Wang, S., & Honavar, V. (2021). Functional autoencoders for functional data representation learning. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM) (pp. 666-674). Society for Industrial and Applied Mathematics.\n\nLuo, R., & Qi, X. (2024). General Nonlinear Function-on-Function Regression via Functional Universal Approximation. Journal of Computational and Graphical Statistics, 33(2), 578-587.\n\nRao, A.R. and Reimherr, M., 2023. Modern non-linear function-on-function regression. Statistics and Computing, 33(6), p.130.\n\nWang, Q., Wang, H., Gupta, C., Rao, A.R. and Khorasgani, H., 2020, December. A non-linear function-on-function model for regression with time series data. In 2020 IEEE International Conference on Big Data (Big Data) (pp. 232-239). IEEE.\n\nWu, S., Beaulac, C. and Cao, J., 2024. Functional Autoencoder for Smoothing and Representation Learning. arXiv preprint arXiv:2401.09499.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ms9hpIQo2h", "reviewer": "Reviewer_Ta2r", "summary": "This paper extends the application of semi-structured networks to functional data. The orthogonalization technique makes this method more scalable than existing methods.", "questions": "Q1: The authors claim that FFR can serve as \"a functional version of a residual connection \"(lines 141-142) and show that the FFR component can empower FNN for better model performance, as shown in Figure 5. Why do we not add the residual design to the FNN directly? Comparing the model performance between FNN, FNN with residual network design, and FNN+FFR in Figure 5 could answer this question.\n\nQ2: Under the discretization setting, what are the differences between time series modeling and functional networks?", "limitations": "The limitations are well discussed, but the authors have not stated the GPU specification in Appendix D.2", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is well-structured and well-written. The idea is easy to follow despite the experiment being conducted in an abstract functional space. The authors have conducted extensive experiments to show the new method's applicability and generalizability.", "weaknesses": "The authors emphasize the interpretability of the proposed method, but they have not demonstrated this property in their experiments. Besides, the FFR $\\lambda^+$ can only explain the linear part of the target function $y$, but most of the time, we hope to explain the nonlinear interaction in the neural network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and detailed comments. Below we address the mentioned weaknesses and questions.\n\n-----\n\n### Weaknesses\n\n> [...] interpretability of the proposed [...] not demonstrated in their experiments \n\nWe agree that this is an important part of our work. We would like to point out that we have included result interpretations in Figures 1 and 3, and presented the estimated weight surfaces in Figure 7 in the Appendix, accompanied by a brief explanation. While placing Figure 7 with interpretability results in the Appendix might seem less ideal, it is essential to note that interpreting the estimated weight surfaces of function-on-function regression models is inherently complex. This complexity is further compounded by the need for domain knowledge in biomechanics to fully grasp the results. Therefore, we chose to provide Figure 1 with a toy example and place Figure 7 in the Appendix. Nonetheless, we can confirm that the results are highly plausible from a biomechanical perspective and offer valuable insights into the relationships between accelerations and moments in the analyzed joints.\n\n> Besides, the FFR $\\lambda^+$  can only explain the linear part of the target function $y$, but most of the time, we hope to explain the nonlinear interaction in the neural network.\n\nWe agree with the reviewer that nonlinearities are not of lesser importance. However, we would like to point out that, although the $\\lambda^+$ part of the model is linear in the coefficients of the basis functions, this implies non-linearity in the estimated function-on-function effects. Figures 1, 3, and 7 clearly illustrate this non-linearity, where certain combinations of input and output signals change nonlinearly over the two different time domains. In addition, while it is possible to include interactions between input signals in $\\lambda^+$, this increases the complexity of interpretation. We believe it is up to the modeler to decide the appropriate order of interactions to assess in the structured part and what effects to move into the black-box mechanics of the deep network.\n\n-----\n\n### Questions\n\n> Q1: [...] Why do we not add the residual design to the FNN directly?\n\nWe thank the reviewer for this interesting question. While we agree that adding a residual connection directly into the FNN would also improve performance compared to the deep-only network, stacking multiple residual connections would complicate the interpretation. In such a case, we would need to understand the total effect of the structured (linear) part on the outcome in this residual network and, even more challenging, find a way to orthogonalize such a model. Our current proposal addresses this issue clearly, as there is a direct connection from the input to the output that is linear in the coefficients.\n\n> Q2: Under the discretization setting, what are the differences between time series modeling and functional networks?\n\nThis is an excellent question that often arises in functional data analysis (FDA). The primary difference between time series modeling (TSM) and FDA is that, in FDA, the curves or “time series” are observed repeatedly and are viewed as replications of the same underlying process (e.g., acceleration curves across the time of one stride). In contrast, TSM typically deals with time series that do not have replications (we cannot observe today’s stock prices multiple times with different noise processes; we only observe each stock price time series once).\nIn FDA, we further predict entire curves on the same domain as the training data, while TSM often focuses on forecasting specific future values. Additionally, FDA usually assumes the smoothness of the process (acceleration profiles, at least theoretically, do not have discontinuities) and is concerned with the shape of the curve (e.g., the curvature and course of the signal over time). In TSM, the shapes of the time series are often not the primary focus.\n\n > [...] the authors have not stated the GPU specification in Appendix D.2\n\nAll models were trained on a CPU; we did not use any GPUs.\n\n----- \n\nPlease let us know whether we have addressed all weaknesses as well as questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank reviewers DyRG, SfMu, 9i2N, and Ta2r for their detailed and thoughtful comments. We appreciate your efforts and believe your reviews have allowed us to further improve our paper. We think that have addressed all points raised and eliminated all uncertainties. In detail:

1. **[Reviewer DyRG](https://openreview.net/forum?id=WJAiaslhin&noteId=22uGAaOUUW)**: 
    - We clarified uncertainties regarding model tuning and 
    - highlighted existing text passages that answer questions about network definitions.
2. **[Reviewer SfMu](https://openreview.net/forum?id=WJAiaslhin&noteId=1jH7ELJVbj)**: 
    - We argued that our work is suitable for the ML/DL community, 
    - pointed to existing results in the appendix that might have been overlooked, 
    - clarified various points regarding additive structures, proportional contributions, integration weights, orthogonalization, and runtime comparisons, and 
    - added a comparison study with another approach from the literature as requested.
3. **[Reviewer 9i2N](https://openreview.net/forum?id=WJAiaslhin&noteId=ScrPcIafBW)**: 
    - We clarified our contributions, 
    - discussed the choice of basis functions, and 
    - pointed to additional results in the appendix that might have been overlooked.
4. **[Reviewer Ta2r](https://openreview.net/forum?id=WJAiaslhin&noteId=es1ujpx7su)**: 
    - We explained where interpretability is demonstrated in our results and 
    - clarified questions about residual connections, non-linearity in the model, and the difference from time series modeling.

-----

Please let us know if there are any further questions or if additional clarifications are needed. We are happy to discuss any open points in the next discussion phase.

Meta-review: The paper proposes a functional extension of semi-structured networks, consisting of an additive combination of an interpretable function-on-function regression and an expressive functional neural network. The paper also addresses questions regarding scalability and identifiability, and demonstrates the effectiveness of the proposed architecture in synthetic scenarios and on real-world biomechanics data.

This paper received reviews on the positive side, with all reviewers recommending acceptance after the rebuttal phase. The reviewers commended the paper’s clear presentation, novel approach, and competitive performance. The orthogonalization approach, in particular, was highlighted as an interesting solution to the identifiability problem in architectures with an interpretable and an expressive component. 

Initial concerns expressed by the reviewers included accessibility/relevance and experiment design/results. Multiple reviewers felt that a more thorough introduction to functional data analysis and biomechanics, supported by figures and examples, would improve accessibility/relevance of the manuscript. The authors promised and are expected to update the relevant sections. Initial feedback on the experiment design/results was mixed, with reviewers mentioning marginal improvements over baselines, missing baselines, and limited analysis of interpretable results. The authors responded well to these concerns, providing scalability arguments, additional results, and pointers to interpretable weight surfaces.

The majority of reviewers felt that their concerns were adequately addressed during the rebuttal phase and multiple reviewers increased their rating as a result. Based on the paper’s contributions to the field of semi-structured networks, and with the expectation that the feedback received and clarifications provided during the discussion period will be integrated, the paper is recommended for acceptance as a poster.
Decision: Accept (poster)