A Functional Extension of Semi-Structured Networks

Authors: david  rügamer
bernard x w liew
zainab  altai
almond  stöcker
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 21:56:58

Abstract: Semi-structured networks (SSNs) merge the structures familiar from additive models with deep neural networks, allowing the modeling of interpretable partial feature effects while capturing higher-order non-linearities at the same time. A significant challenge in this integration is maintaining the interpretability of the additive model component. Inspired by large-scale biomechanics datasets, this paper explores extending SSNs to functional data. Existing methods in functional data analysis are promising but often not expressive enough to account for all interactions and non-linearities and do not scale well to large datasets. Although the SSN approach presents a compelling potential solution, its adaptation to functional data remains complex. In this work, we propose a functional SSN method that retains the advantageous properties of classical functional regression approaches while also improving scalability. Our numerical experiments demonstrate that this approach accurately recovers underlying signals, enhances predictive performance, and performs favorably compared to competing methods.

Main paper: 
 introduction: 
 Incorporating additive structures in neural networks to enhance interpretability has been a major theme of recent advances in deep learning. For example, neural additive models (NAMs; [1] ) allow users to learn basis functions in an automatic, data-driven fashion using feature subnetworks and thereby provide an alternative modeling option to basis function approaches from statistics such as generalized additive models (GAMs; [17] ). Various extensions and related proposals have been published in recent years. Notable examples include the joint basis function learning by [41] or the combination of GAMs and neural oblivious decision ensembles [8] . When combining structural assumptions with arbitrary deep architectures, the resulting network is often referred to as wideand-deep [10] or semi-structured network (SSN; [49, 52] ). Various extensions of SSNs have been proposed in recent years, including SSNs for survival analysis [24, 25] , ordinal data fitting [22] , for uncertainty quantification [2, 12] , Bayesian SSNs [11] , or SSNs incorporated into normalizing flows [23] . One of the key challenges in SSNs is to ensure the identifiability of the additive model part in the presence of a deep network predictor in order to maintain interpretability.In this work, motivated by a large-scale biomechanical application, we study how to efficiently transfer these two concepts -the embedding of basis functions into neural networks and the extension to SSNs -for functional data analysis.Functional data analysis Functional data analysis (FDA; [42] ) is a research field of increasing significance in statistics and machine learning (e.g., [3, 37, 38, 57] ) that extends existing modeling approaches to address the functional nature of some data types (e.g., sensor signals, waves, or growth curves). One well-known extension is the class of functional additive models [15, 53] , which represent the functional analog of GAMs. These models adopt the classical regression model point of view but allow for both in-and outputs to be (discretized) functions. Functional additive models, however, are often not expressive enough to account for all interactions and non-linearities and do not scale well to large datasets.Biomechanics A prominent example of a research field dealing with functional data is biomechanics. As in many other applications concerned with physical or biological signals, the integration of machine learning in biomechanics research offers many advantages (see, e.g., [26, 29, 30] ). The research on joint moments in biomechanics, for example, provides insight into muscular activities, motor control strategies, chronic pain, and injury risks (see, e.g., [13, 31, 36, 47, 56] ). By combining kinematic features with machine and deep learning to predict joint moments, researchers can bypass the need to gather high-quality biomechanics data and instead use "cheap" sensor data obtained through, e.g., everyday mobile devices which in turn allow predicting the "expensive" signals that can only be obtained in laboratory setups [20, 32, 33] . While a promising research direction, existing methods face various challenges including a lack of generalization and a missing clear understanding of the relationship learned.In FDA, there has been a successful translation of many different methods for scalar data to functionvalued data, including functional regression models and functional machine learning techniques.Functional regression models Regression models with scalar outputs and functional inputs are called scalar-on-function regression models. An overview of such methods can be found in [45] . Models with functional output and scalar inputs are called function-on-scalar models [9, 15] . They can be applied when the outcome or error function is assumed to be repeated realizations of a stochastic process. Combining these two approaches yields the function-on-function regression model with functions as input and output [see, e.g., 35, 53] . We provide a more technical description of function-on-function regression is given in Section 2.1.In recent years, several machine learning approaches for functional data have been proposed. One is Gaussian process functional regression [GPFR; 21, 54] which combines mean estimation and a multivariate Gaussian process for covariance estimation. As for classical Gaussian process regression, the GPFR suffers from scalability issues. Gradient boosting approaches for functional regression models have been proposed by [4, 6] . These approaches result in sparse and interpretable models, rendering them especially meaningful in high dimensions, but lack expressivity provided by neural network approaches. Pioneered by the functional multi-layer perceptron (MLP) of [48] , various researchers have suggested different functional neural networks (FNNs) [16, 44, 55] , including MLPs with functional outcome (cf. Section 2.2 for technical details). A recent overview is given in [59] . Existing approaches are similar in that they learn a type of embedding to represent functions in a space where classical computations can be applied (cf. Fig. 2(a) ).From the related literature, we can identify the most relevant methods suitable for our modeling challenge. Neural network approaches such as the functional MLP provide the most flexible class for functional data, whereas additive models and additive boosting approaches such as [6, 53] yield interpretable models. An additive combination of these efforts to obtain both an interpretable model part and the possibility of making the model more flexible is hence an attractive option.Open challenges While semi-structured models are suited for a combination of interpretable additive structures and deep neural networks, existing approaches cannot simply be transferred for application with functional data. Due to the structure and implementation of SSNs, a naïve implementation would neither be scalable to large-scale datasets nor is it clear how to incorporate identifiability constraints to ensure interpretability of the structured functional regression part in the presence of a deep neural network.In this work, we address these limitations and propose an SSN approach for functional data. Our method is not only the first semi-structured modeling approach in the domain of functional data, but also presents a more scalable and flexible alternative to existing functional regression approaches. In order to preserve the original properties of functional regression models, we further suggest an orthogonalization routine to solve the accompanied identifiability issue. 
notation and background: 
 We assume functional inputs are given by J ≥ 1 second-order, zero-mean stochastic processes X j with square integrable realizations x j : S j → R, i.e. with x j ∈ L 2 (S j ), j = 1, . . . , J on real intervals S j . Similarly, a functional outcome is given by a suitable stochastic process Y over a compact set T ⊂ R with realizations y : T → R, y ∈ L 2 (T ) =: G. For the functional inputs, we simplify notation by combining all (random) input functions into a vector X(s) = (X 1 (s 1 ), . . . , X J (s J )) with realization x(s) ∈ R J , where s := (s 1 , . . . , s J ) ∈ S ⊂ R J is a J-tuple from domainS := S 1 × • • • × S J , and H := L 2 (S 1 ) × • • • × L 2 (S J ).Next, we introduce function-on-function regression (FFR) and establish a link to FNNs to motivate our own approach.An FFR for the expected outcome µ(t) := E(Y (t)|X) of Y (t), t ∈ T using inputs X can be defined as follows:EQUATIONwhere b(t) is a functional intercept/bias and the w j (s, t) are weight surfaces describing the influence of the jth functional predictor at time point s ∈ S j on the functional outcome at time point t ∈ T . τ is a point-wise transformation function, mapping the affine transformation of functional predictors to a suitable domain (e.g., τ (•) = exp(•) to obtain positive values in case Y is a count process).Various extensions of the FFR model in (1) exist. Examples include time-varying integration limits or changing the linear influence of x(t) to a non-linear mapping. Example Figure 1 shows an example of a function-on-function regression by visualizing the corresponding learned weight surface, with three highlighted areas: a) An isolated positive weight multiplied with a positive feature signal at x(90) induces a small spike at y (10) . b) A more extensive negative weight multiplied with mostly positive feature values for s ∈ [1, 20] and integrated over all time points results in a bell-shaped negative outcome signal at around t = 30. c) A large but faded positive weight region multiplied with positive feature values results in a slight increase in the response function in the range t ∈ [50, 100] . In most applications of FFR, the goal is to find these salient areas in the weight surface to better understand the relationship between input and output signal.To establish the connection between FFR and FNNs, it is instructive to consider a function-on-function MLP (FFMLP). In its basic form, an MLP for scalar values consists of neurons h : R → R, x → τ (b + w ⊤ x) arranged in multiple layers, each layer stacked one on top of the other. The extension to a fully functional L-layer MLP can be defined recursively by the kth output neuron h(l)k of a functional layer l = 1, . . . , L asEQUATIONwhere M l denotes the number of neurons in layer l, b(l) k ∈ L 2 (U (l) m ) and w (l) m,k ∈ L 2 (U (l-1) m × U (l) m ) for some functional domains U (l) m , input layer h (0) m (•) = x m (•)for all predictors m = 1, . . . , J, and last layer withM L = 1 neuron h (L) (t) = E(Y (t)|X).3 Semi-structured functional networks Our approach generalizes the previous models by considering a general neural network Λ : H → G that models the input-output mapping from the set of functional features X to the expected outcome µ(t) of the functional response for t ∈ T asEQUATIONcomprising an FFR part λ + (X)(t) and a deeper FNN architecture λ -(X)(t), which are added and transformed using an activation function τ . This combination can also be thought of as a functional version of a residual connection. In the following, we drop the functional input arguments of λ + and λ -for better readability. The model in (3) combines structured interpretable models as described in Section 2.1 with an arbitrary deep network such as the one in Section 2.2. In particular, this allows for improving the performance of a simple FFR while retaining as much interpretability as possible (cf. Section 3.3).Interpretable model part As in (1), we model the interpretable part λ + (t) = J j=0 λ + j (t) as a sum of linear functional terms λ + j (t) = Sj w j (s, t)x j (s)ds. To make our model more concrete, we can expand each weight surface w j in a finite-dimensional tensor-product basis asEQUATIONover fixed function bases ϕ j = {ϕ jk } Kj k=1 , ϕ jk ∈ L 2 (S j ) and ψ(t) = {ψ u } U u=1 , ψ u ∈ L 2 (T ) with weight matrix Θ j ∈ R U × R Kj . Analogously, we can represent the intercept as λ + 0 (t) = b(t) = ψ(t) ⊤ Θ 0 . We may understand the model's interpretable part for the jth feature as a functional encoder ϕ * j (X j ) = {ϕ * jk (X j ) = ϕ jk (s)X j (s)ds} Kj k=1 , encoding X j into a latent variable z j ∈ R Kj , and a linear decoder ψ with weights Θ j mapping z j to the function λ + j (t):X j ϕ * j -→ z j Θj ψ -→ λ + j .Here, ϕ * j presents the dual basis to ϕ j . Visually, the weight surface w j in (4) can be interpreted as exemplarily shown in Figure 1 .The part λ -(t) in (3) is a placeholder for a more complex network. For FNNs, a conventional neural network might be applied after encoding X with ϕ * and before decoding its results with ψ, such that en-and decoding are shared with the interpretable part λ + as depicted in Fig. 2a . Alternatively, a general FFMLP as described in Section 2.2 can be embedded into the approach in Fig. 2a . As researchers have often already found well-working architectures for their data, a practically more realistic architecture for semi-structured FNNs is to allow for a more generic deep model as depicted in Fig. 2b . This is also the case in our application on biomechanical sensor data in Section 4.2. Here, we choose λ -(t) to be a specific InceptionTime network architecture [19] that is well-established in the field and train its weights along with the weights Θ of λ + (t). 
implementation for discretized features: 
 As functional predictors are usually only observed on a discretized grid on the domains S j , we now derive a way to implement λ + (t) in practice. Depending on the architecture used for the deep part, we might proceed analogously with λ -(t). Let x (i) j (s r ) ∈ R be the evaluation of the ith realization x (i) j of X j at time points s r , r = 1, . . . , R, stacked into a vector x j ∈ R R . For better readability, we assume that these time points are equal across all J features. With only discrete evaluations available, we approximate integrals over S j numerically with integration weights ∆(s r ), e.g., using trapezoidal Riemann weights. Given the tensor-product basis representation of the weight surface w j (s, t) in (4), we effectively evaluate ϕ j (s r ) for all R time points, yielding Φ j = [ϕ j (s 1 ), . . . , ϕ j (s R )] ∈ R Kj ×R , and obtain the row-vector Φ * j of approximate functionalsϕ * jk (x j ) = ϕ jk (s)x j (s)ds ≈ R r=1 ∆ j (s r )ϕ jk (s r )x j (s r ) as Φ * j = (∆ j • x j )Φ ⊤ j ∈ R 1×Kj, where • denotes the Hadamard product and ∆ j = [∆ j (s 1 ), . . . , ∆ j (s R )] ⊤ . Putting everything together, we can represent the jth interpretable functional model term λ + j (t) asEQUATIONThis model part is still a function over the outcome domain T , but discretized over the predictor domains. We can now proceed with the optimization when the functional outcome is represented with finitely many observations.For the ith observation, we can measure the goodness-of-fit of our model µ(t) by taking the point-wise loss function l(y (i) (t), μ(i) (t)) of the ith realized function y (i) and the predicted outcome μ(i) given x (i) and integrate over the functional domain to obtain the ith functional loss contribution ℓ, i.e.,EQUATIONThe overall objective, our empirical risk, is then given by the sum of ( 6) over all n observations. In practice, integrals in (6) are not available in closed form in general and the functional outcome is only observed on a finite grid. Therefore, similar to the discretization of the weight surface, let y (i) (t q ) be the observations of the outcome for time points t q ∈ T , q = 1, . . . , Q, summarized for all time points as y (i) = (y (i) (t 1 ), . . . , y (i) (t Q )). Given a dataset D with n observations of these functions, i.e., D = {(x (i) , y (i) )} i=1,...,n , our overall objective then becomesEQUATIONwhere Ξ(•) are integration weights.As it less efficient to represent µ as an actual function of t if Y is only given for fixed observed time points t q , we can adapt (5) to predict a Q-dimensional vector using where Ψ = [ψ(t 1 ), . . . , ψ(t Q )]. The deep network λ -naturally yields the same-dimensional output when sharing the decoder. If a flexible deep model is used for λ -as in Fig. 2b , λ -can be defined with Q output units to match the dimension of λ + . For one observation, the objective ( 7) can now be written as a vector-based loss function ofEQUATIONy (i) ∈ R 1×Q and µ (i) = j λ + j (i) + λ -(i) ∈ R 1×Q , or,when stacking these vectors, as loss function between two n × Q matrices. w(s,t)Figure 3 : Estimated weight surfaces of the one functional shank gyroscope predictor in λ + for the different joints (columns), before and after correcting with the orthogonalization (rows). The color of each surface corresponds to the weight a predictor sensor signal at time points s (x-axis) is estimated to have on the tth time point (y-axis) of the outcome (cf. Fig. 1 ). Without correction (upper row), the interpretable model part is not only incorrectly estimated but no effect at all is attributed to it. After correction, distinct patterns for some of the time point combinations and joints are visible, e.g., suggesting that an increased gyroscope value for early time points s has a negative influence (dark purple color) on the first half of the hip adduction moment (bottom row, second plot from the left).Since λ -is a potentially very expressive deep neural network that can also capture all functional linear dependencies modeled by λ + , the interpretable model part of the semi-structured FNN is not identifiable without further constraints. Non-identifiability, in turn, means that we cannot simply interpret the weights of the interpretable model part as it is not clear how much these are diluted by the deep network part. This can, e.g., be seen in Figure 3 which shows selected weight surfaces from our biomechanical application in Section 4. In this case, the deep network (an InceptionNet) captures almost all linearity of the structured part (upper row) and only after correction do we obtain the actual weight surfaces. We achieve this by extending the post-hoc orthogonalization [PHO; 49] for functional in-and outputs. First, we rewrite λ + (t) ∈ R, the sum of all J terms in (5), asλ + (t) = vec(λ + (t)) (5) = vec   J j=1 Φ * j Θ j ψ(t)   = J j=1 (ψ(t) ⊤ ⊗ Φ * j )vec(Θ j ) =: J j=1 Ω j (t)θ j with Ω j (t) = (ψ(t) ⊤ ⊗ Φ * j ) ∈ R 1×Kj •U, where ⊗ denotes the Kronecker product, and θ j = vec(Θ j ) ∈ R Kj •U . This reformulation allows us to represent the functional interpretable model part as a linear combinationΩ(t)θ = [Ω 1 (t), . . . , Ω J (t)][θ ⊤ 1 , . . . , θ ⊤ J ]⊤ of basis products and their coefficients, and thereby to apply PHO for semi-structured FNNs. Let Ω {N } and λ {N } be the stacked matrices for the N = n • Q data points in the training set D after stacking the Ω(t) and λ -(t) for all Q time points of all n observed curves. Then we can obtain an identifiable and thus interpretable functional model part from the trained semi-structured FNN with basis matrix Ω {N } , parameters θ, and deep network part λ - {N } by computing the following:EQUATIONwhere θ is the corrected basis coefficient vector, Ω † {N } is the Moore-Penrose pseudoinverse of Ω {N } , P ⊥ Ω {N } is a projection matrix projecting into the orthogonal complement spanned by columns of Ω {N } , and λ ⊥ {N } are the corrected deep network predictions orthogonal to the interpretable part. Using θ j from θ = [ θ 1 , . . . , θ J ] instead of the original θ j s will yield updated weights w j for the surfaces from (4) of the interpretable functional model part, which can be interpreted irrespective of the presence of a deep network.When implementing semi-structured FNNs, λ -can be chosen arbitrarily to the needs of the data modeler. Hence, we do not explicitly elaborate on a particular option here and assume an efficient implementation of this part. Instead, we focus on a careful implementation of the structured model part to avoid unfavorable scaling as in other existing implementations (cf. Figure 4(b) ). To mitigate this problem (in particular reducing the space complexity), we propose two implementation tricks. While these mainly reduce the complexity scaling w.r.t. J and R, implementing an FFR in a neural network already yields an improvement by allowing to cap the memory costs to a fixed amount through the use of mini-batch training. In contrast, a full-batch optimization routine of (7) as implemented in classic approaches (e.g., [5, 53] ) scales both with n and Q as data is transformed in a long-format, which can grow particularly fast for sensor data. For example, only n = 1000 observations of a functional response evaluated at Q = 1000 time points already results in a data matrix of 1 million rows. Classic implementations also scale unfavorably in terms of the number of features J and observed times points R for these functions. In the case where all functional predictors are measured on the same scale, fitting an FFR model requires inverting a matrix of size(n • Q) × (( J j=1 K j ) • U ),which becomes infeasible for larger amounts of predictors. Even setting up and storing this matrix can be a bottleneck.In contrast, when using the representation in ( 5), the basis matrices in sand t-dimension (i.e. for in-and output) are never explicitly combined into a larger matrix, but set up individually, and a network forward-pass only requires their multiplication with the weight matrix Θ. As the matrix Ψ is the same for all J model terms, we can further recycle it for these computations.Basis recycling In addition, if some or all predictors in X share the same time domain and evaluation points s 1 , . . . , s R , we can save additional memory by not having to set up J individual bases Φ j . 
numerical experiments: 
 In the following, we show that our model is able to recover the true signal in simulation experiments and compares favorably to state-of-the-art methods both in simulated and real-world applications. As comparison methods, we use an additive model formulation of the FFR model as proposed in [53] , an FFR model based on boosting [5] and a deep neural network without interpretable model part. For the latter and the deep part of the semi-structured model, we use a tuned InceptionTime [19] architecture from the biomechanics literature [32] . As FFR and boosting provide an automatic mechanism for smoothing, no additional tuning is required. In all our experiments, we use thin plate regression splines [60] but also obtained similar results with B-splines of order three and first-order difference penalties.Hypotheses In our numerical experiments, we specifically investigate the following hypotheses:• Comparable performance with better scalability: For additive models without the deep part, our model can recover complex simulated function-on-function relationships (H1a) with similar estimation and prediction performance as other interpretable models (H1b) while scaling better than existing approaches (H1c).• Favorable real-world properties: In real-world applications, our model is on par or better in prediction performance with current approaches (H2a), with a similar or better resemblance when generating output functions (H2b), and provides a much more meaningful interpretation than the deep MLP (H2c). Further, we conjecture that our approach is better in prediction performance than its two (structured, deep) individual components alone (H2d). Prediction and estimation performance (H1a,b) For the model's estimation and prediction performance, we simulate a function-on-function relationship based on a complex weight relationship w(s, t) between one functional predictor and a functional outcome. Figure 4 (a) depicts this true weight along with an exemplary estimation by the different methods. The estimation performance of the weight surface and all methods' prediction performance over 30 different simulation runs each with a signal-to-noise ratio SNR ∈ {0.1, 1} and n ∈ {320, 640, 1280} functional observations is summarized in Figure 6 in the Appendix. While the models perform on par in most cases, we find that the network works better for more noisy settings (SNR = 0.1) and is not as good as other methods for higher signal-to-noise ratio (as also shown in Figure 4 (a)), but with negligible differences. Overall, there are only minor differences in both estimation and prediction performance. This result suggests that our method works just as well as the other methods for estimating FFR.To investigate the scalability of all methods, we subsample the data from the Carmago study presented in the following subsection by reducing observations n, features J, and/or the number of observed time points R. More specifically, we investigate the memory consumption of the three implementations previously discussed (additive model, boosting, neural network) for n ∈ {25, 50, 100} functional observations, Q = R ∈ {25, 50, 100} observed data points, and J ∈ {1, 2, 4} functional features. We restrict this study to rather small functional datasets as the memory consumption of other methods is superlinear. This can also be seen in Figure 4 (b) presented earlier, which summarizes the result of this simulation study and clearly shows the advantage of our implementation compared to existing ones form [5, 53] .Our method being originally motivated by applications in the field of biomechanics, we now analyze two real-world datasets with biomechanical sensor data. In addition to these applications in biomechanics, we provide further applications to EMG-EEG synchronization, air quality prediction and hot water consumption profiles in Appendix C to demonstrate the versatility of our approach.The data analyzed in the first experiment is a collection of three publicly available running datasets [14, 27, 28] . A recent study of [32] used this data set to show the potential of deep learning approaches for predicting joint moments during gait (in total 12 different outcome features). The given data set collection is challenging as it provides only n = 490 functional samples but J = 27 partially highly correlated functional features. We follow the preprocessing steps of [32] and split the data into 80% training and 20% test data for measuring performance according to the authors' split indices.As predictors we use the gait cycle of all available joint measurements, that is the 3D joint angle, velocity, and acceleration of the bilateral ankle, knee, and hip joints. Predictors in both training and test datasets are separately demeaned and scaled using only the information from the training set.As in the simulation studies, we run the additive model, Boosting, and our approach and compare the performance across all 6 different outcome variables. We use the relative and absolute root mean squared error (RMSE) as suggested in [46] and the Pearson correlation coefficient [20] to measure performance. In contrast to our simulation study, the additive model implementation of FFR cannot deal with this large amount of functional features. We, therefore, split our analysis into one comparison of the additive model, an FNN, and a semi-structured FNN (SSFNN) on a selected set of features, and a comparison of SSFNN against boosting (which has an inbuilt feature selection).Results in Table 1 summarize test performances across all 12 different outcome types. We see that the (SS)FNN approaches perform equally well or better compared to the additive model on the selected set of features, and SSFNN also matches the performance of Boosting when using all features. The second data set is a publicly available dataset on lower limb biomechanics in walking across different surfaces level-ground walking, treadmill walking, stair ascent, stair descent, ramp ascent, and ramp descent [7] .Due to the size of the data set consisting of 21787 functional observations, Q = R = 101 observed time points, and J = 24 predictors, it is not feasible to apply either additive model-based FFR or boosting. We, therefore, compare SSFNN to a (structured) neuralbased FFR model and a deep-only neural network. Based on the pre-defined 70/30% split for training and testing, we run the models for each of the 5 outcomes (joints) and compute the relative MSE difference, a functional R 2 analogon with values in (-∞, 1] (see Appendix A.1 for definition). Table 2 shows the results averaged for all outcome types while Figure 5 provides performance changes individually for the 5 different joints.We observe that the semi-structured model performs better than the deep model, which in turn is better than the structured model. Figure 8 in the Appendix further depicts the predicted functions for all 5 outcomes using the 3 models. While for some joints the prediction of the semi-structured and deep-only model is very similar, the performance for the subtalar joint is notably better when combining a structured model and the deep neural network to form a semi-structured model.  
discussion: 
 Our proposed method is an extension of semi-structured networks to cater to functional data and deals with the issue of identifiability by proposing a functional extension of post-hoc orthogonalization.Using array computations and basis recycling the method also solves scalability issues present in existing methods. Our experimental results reveal that functional semi-structured networks yield similar or superior performance compared to other methods across varied biomechanical sensor datasets.Novelty The approach proposed in Section 3 shows similarities to an autoeconder architecture. While a functional autoencoder has been proposed in the past [18] and most recently by [61] , these studies focus on the representation learning (and reconstruction) of the same signal. Other papers focusing on function-on-function regression (e.g., [34, 43, 58] ) suggest similar approaches to ours but without the option to jointly train a structured model and a deep network.Limitations and Future Research While our model shows good performance on various datasets and in simulations, we believe that our approach can be further improved for applications on highdimensional sensor data by incorporating appropriate sparsity penalties. Due to the functional nature of the data, feature selection would result in a great reduction of complexity and thereby potentially yield better generalization. Our general model formulation would further allow the extension to a nonlinear FFR model λ -(t) = J j=1 Sj f j (x j (s), t)ds using smooth functions f j in three dimensions (feature-, sand t-direction). Although this extension is still an additive model in the functional features, it is challenging to interpret the resulting additive effects.Let n be the number of observed curves indexed with i. Given the ith observation of a function output signal y (i) (t) for t ∈ T with functional domain T and corresponding predicted value µ (i) (t), we define the functional R 2 as κ({yEQUATION)Since T (y (i) (t) -µ (i) (t)) 2 dt ≥ 0, the nominator is the smallest for y(t) ≡ µ(t), in which case κ = 1. In contrast, if the prediction µ is µ ≡ 0, κ = 0. For models that perform worse than the constant zero predictor T (y (i) (t) -µ (i) (t)) 2 dt can be larger than T (y (i) (t)) 2 dt, yielding a negative nominator and hence negative κ values.In order to enforce smoothness of estimated functional relationships in λ + in both sand t-direction, we can employ a quadratic smoothing penalty. For regression splines with evaluated bases Φ j and Ψ, smoothness is usually defined by some form of difference penalties, which we define to be encoded in the matrices P s and P t , respectively. For the array computation, we want to penalize all differences in s-direction, which can be done by usingpen s = vec(Θ) ⊤ (I U ⊗ P s )vec(Θ) = vec(Θ • (P s Θ)) ⊤ 1.Analogously, the t-direction can be penalized aspen t = vec(Θ) ⊤ (P t ⊗ K j )vec(Θ) = vec((ΘP t ) • Θ) ⊤ 1.Putting both penalties together, we obtain the total penalty for the array product as pen s + pen t .Figure 6 shows the estimation error of the weight surface using different established methods and our proposed implementation (Neural Network).All methods perform equally well with minor differences in the weight surface estimation. As expected, neural networks perform better in small signal-to-noise (SNR) settings due to their implicit regularization when trained with stochastic gradient descent and using mini-batches, whereas for large SNR, the full batch routines implemented by boosting and additive models perform slightly better. As becomes apparent in Figure 4 (a), these differences are negligible in practice.Figure 7 gives an excerpt of the estimated weight surfaces that provide insights into how the relationship between input features (rows; here acceleration measures from the x-dimension) and outcome joints is estimated.For example, the trunk acceleration (last row in Figure 7 ) having a short negative and almost instantaneous effect (dark circle) on the ankle and hip adduction moment (first and second column) is a reasonable result from a biomechanical perspective.We can further plot the predicted curves by all employed methods (Figure 8 ) and compare the results with the ground truth. While for some joints the prediction of the semi-structured and deep-only model is very similar, the performance for the subtalar joint is notably better when combining a structured model and the deep neural network to form a semi-structured model. Comparison with [32] As requested by an anonymous reviewer, we have also compared our method against the approach suggested in [32] . Compared to [32] , we use a different pre-processing of functional predictors which is more in line with our main results. We then run their best model once using a deep-only variant and once using a semi-structured model. The RelRMSE results are given in Table 3 , showing that the semi-structured extension works similarly well or in most cases better.In order to demonstrate our method's applicability beyond biomechanical applications, we run several additional experiments. Datasets come from different fields of application and have different sizes Table 3 : Comparison of a plain deep and a semi-structured approach, where the former is based on [32] and the latter its extension to a SSN. (number of observations, number of functional predictors, number of observed time points). We also use these examples to 1) compare the two different architectures suggested in Fig. 2 (a) and Fig. 2(b),  2) compare against a non-linear FFR implementation from [40] .For all the additional experiments, the deep network version from Fig. 2 (a) uses the same encoding and decoding basis for both λ + and λ -(using K j = 20 basis functions for the s-direction and U = 20 basis function for the t-direction). The trunk of the deep network part λ -is defined as a fully connected neural network with 100 neurons each, followed by a dropout layer with dropout rate of 0.2 and a batch normalization layer. The deep network version from Fig. 2 (b) uses the same architecture, but instead of using a shared encoding and decoding part, it simply concatenates the inputs and processes the three-dimensional object until the last layer, which is then reshaped to have the correct dimensions. We have not tuned this architecture, the number of layers, neurons, or dropout rate, as the purpose of these additional experiments is solely to demonstrate that the (SS)FNN approach yields competitive results with only the structured part while the deep part can further help to improve predictive performance.The dataset analyzed in [50] Results confirm again that our FNN implementation leads to similar values as Boosting, while including a deep neural network part can improve the performance.The dataset analyzed in [40] contains 355 observations of daily NO2 measurement curves (i.e., the functional domain is the hours 1 to 24) together with four other functional pollutants as well as the temperature and relative humidity. The goal is to predict the NO2 concentration using other pollutants, the temperature, and the humidity. The proposed approach in [40] uses a non-linear function-on-function regression (FFR) which we compare against. The results are given in the following table: Results suggest that the given task is rather easy and a linear FFR model is already sufficient to obtain good prediction performance. Hence FNN outperforms both the non-linear FFR model and the two semi-structured approachesThe dataset taken from [39] contains hourly hot water consumption profiles (i.e., the functional domain is again 1 to 24) which are available for different administrative districts. The goal of the analysis is to predict the hot water consumption of a specific region given multiple (here 6) far-distant administrative districts. The results are given in the following table: • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The information is given in the respective section and in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Simulation studies are repeated multiple times and report error bars; for realworld data, replications are done for different data scenarios, also allowing to report standard deviations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The information is given in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We have read the code of ethics. There are no ethical concerns or similar.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: While biomechanical applications often involve data from humans, our work does not deal with this type of data in a way that would imply any particular societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not provide data. Models themselves do not pose a risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The datasets used are referenced and code is CC-BY licensed.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The code is documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The paper proposes a functional extension of semi-structured networks, consisting of an additive combination of an interpretable function-on-function regression and an expressive functional neural network. The paper also addresses questions regarding scalability and identifiability, and demonstrates the effectiveness of the proposed architecture in synthetic scenarios and on real-world biomechanics data.

This paper received reviews on the positive side, with all reviewers recommending acceptance after the rebuttal phase. The reviewers commended the paper’s clear presentation, novel approach, and competitive performance. The orthogonalization approach, in particular, was highlighted as an interesting solution to the identifiability problem in architectures with an interpretable and an expressive component. 

Initial concerns expressed by the reviewers included accessibility/relevance and experiment design/results. Multiple reviewers felt that a more thorough introduction to functional data analysis and biomechanics, supported by figures and examples, would improve accessibility/relevance of the manuscript. The authors promised and are expected to update the relevant sections. Initial feedback on the experiment design/results was mixed, with reviewers mentioning marginal improvements over baselines, missing baselines, and limited analysis of interpretable results. The authors responded well to these concerns, providing scalability arguments, additional results, and pointers to interpretable weight surfaces.

The majority of reviewers felt that their concerns were adequately addressed during the rebuttal phase and multiple reviewers increased their rating as a result. Based on the paper’s contributions to the field of semi-structured networks, and with the expectation that the feedback received and clarifications provided during the discussion period will be integrated, the paper is recommended for acceptance as a poster.

Reviews: review: paper_summary: paper_summary: The paper proposes a hybrid approach that combines the benefits of neural networks with those of more structured models (generalised additive models). They show benefits on real and simulated data.
strength_weakness: strength_weakness: - How much tuning has been done for the pure deep-only network baseline from Section 4.2.2? I would be a bit worried that the findings are very dependent on the choice of architecture of this model? E.g. does it have any increased capacity over the deep part of the semi-structured network? And does this architecture have suitable inductive biases for fitting linear functions (maybe residual connections which add a linear transformation of the input to the output)?
- The presentation could be made clearer in a couple of areas. In particular, (1) Equation 3 is the first time that the “double function call” notation of e.g. $\lambda^-(X)(t)$ is used. Having a double function call makes sense given that there is a function which takes X as input and outputs another function. However it is not clear to me why this is then not used in Equation 1, where you could presumably replace $\mu(t)$ with $\mu(X)(t)$? And e.g. in the last part of Section 2.2, why write $h^{(L)}(t)$ instead of $h^{(L)}(X)(t)$? Would it be possible to make this clearer? And also (2), when functional data analysis is introduced in Section 1 and 2, it would be helpful if an example application was described
questions: questions: See weaknesses. I am willing to increase my score if these are addressed.
limitations: limitations: Adequately addressed
paper_summary: paper_summary: An extension to semi structured networks is introduced that combines a traditional linear and interpretable model component $\lambda^+$ with a flexible $\lambda^-$ neural network component to better approximate relations in biomechanical application. The model performs equally to traditional methods but requires less memory and remains interpretable thanks to a post-hoc orthoganalization.
strength_weakness: strength_weakness: _Significance_: Even though the authors emphasize their method's superiority in lines 144-146, the performance improvement over existing approaches is rather mediocre and does not seem significant, in particular when considering the comparably large standard deviations. The error distributions of the different methods seems to overlap to large extend. In this vein, it would be appreciated if error bars were reported in Figure 5. In that figure, you may consider to flip the y-axis to make it more intuitive and report the relative error reduction over the baseline.

_Further comments_:
- It took me very long to get a grasp of the concept of functional data/analysis/regression. In fact, until the very end of the manuscript, I did not fully understand the actual task at hand and permanently felt somewhat lost. Authors may give a quick practical example of functional analysis early to catch readers sooner. Even the example in lines 109--123 did not resolve my uncertainty about where such processes find application. I thus suggest to add one clear and illustrative practical example and spend more effort to introduce the problem at the beginning of the introduction rather than starting with related work.
- While novelty for functional data analysis seems reasonable, I cannot find a substantial contribution to the machine and deep learning community (given the paper is submitted to a top tier ML conference). Even though I like the paper's argumentation and substantiation and perceive it as a generally sound work, the manuscript might better be suited to a journal that deals with functional analysis or with biomechanics.
- Claim in l265 not supported with experimental results. Please provide error and dispersion metrics (such as in Table 1 and 2) to allow a quantitative comparison of the methods.
- Typo in line 331: Either "reveal that" or "yields"?
questions: questions: 1. What are the additive structures you are referring to in l15, can you provide some examples?
2. What is the proportional contribution of $\lambda^+$ and $\lambda^-$ to the overall solution? How is guaranteed that $\lambda^-$ is not solving the entire problem solely, thus loosing all interpretability? The orthogonal formulation seems to take a step into that direction, but I am not sure what the orthogonal formulation actually does, and if the patterns revealed after its application are helpful and realistic.
3. What are the integration weights $\Xi$ concretely in line 182 and what are they required for (can they be dropped for simplicity yielding a conventional loss function)?
4. The orthogonalization approach in Section 3.3 appears very beneficial, yet, frankly, I could not follow it as the math is quite abstract. Can you share an intuition what the orthogonalization implements and how it effects $\lambda^-$ to be less dominant, if so? Is it some kind of regularization to minimize its contribution (or how would, e.g., L2 regularization compare)?
5. Relating to Figure 4, how does runtime compare for the three implementations for $\lambda^-$? Given the memory is capped for the batched neural network, it should take multiple evaluations to process the same amount of data. How does the memory compare if the other methods are implemented in a batched fashion too?
6. Can you share an intuition of why the neural network performs worse in high SNR regimes? I'd expect a neural network be superior in all conditions, given it is sufficiently large and regularized appropriately.
7. Could you include the results from reference [24] into your results in Section 4.2.1 to allow a comparison between your and an established method? It is crucial to assess how your approach performs in comparison to existing methods.
limitations: limitations: Limitations are addressed by the authors.
paper_summary: paper_summary: In this paper, the authors develop a semi-structured model for functional data, summing an interpretable linear model with a more general nonlinear functional neural network. The authors validate the improved performance of the combination relative to the individual components on a variety of biomechanics datasets. The authors also discuss how to preserve the interpretability of the semi-structured model with a linear weight surface by projecting the neural network predictions into the subspace of the linear model.
strength_weakness: strength_weakness: The paper accomplishes what it sets out to do fairly well, though the contributions of the paper through providing evidence for new phenomena, novel insights, or new algorithmic ideas seem much more limited.

In terms of the structured part of the model, the linear regression with the functional encoder and decoder bases for function -> function regression, I assume that this approach has been used before but I did not see it in any of the explicit references. Can the authors comment on whether this component is original to their work or whether this component is building on what already exists in the literature?
questions: questions: One aspect I did not see mentioned in the paper was the actual choice of basis functions for encoder and decoder. These should certainly be specified along with any justification or tuning of hyperparameters involved (e.g. bandwidth parameters?).

Fig2a and 2b present two different potential options for constructing this semi-structured model, might it be worth evaluating the performance of structure a)? Would we expect the performance to be similar, what are the drawbacks?

When fitting the parameters of the linear model, it is mentioned the large computational cost. I did not find the explanation of how the scalable implementation circumvents this to be particularly clear. Also, even for solving the linear system could one not use more sophisticated matrix free methods such conjugate gradients with the kronecker product matrix, or for example SVRG which would converge quickly even when the number of data points and features is large?
limitations: limitations: I think the limitations are sufficiently discussed.
paper_summary: paper_summary: This paper extends the application of semi-structured networks to functional data. The orthogonalization technique makes this method more scalable than existing methods.
strength_weakness: strength_weakness: The authors emphasize the interpretability of the proposed method, but they have not demonstrated this property in their experiments. Besides, the FFR $\lambda^+$ can only explain the linear part of the target function $y$, but most of the time, we hope to explain the nonlinear interaction in the neural network.
questions: questions: Q1: The authors claim that FFR can serve as "a functional version of a residual connection "(lines 141-142) and show that the FFR component can empower FNN for better model performance, as shown in Figure 5. Why do we not add the residual design to the FNN directly? Comparing the model performance between FNN, FNN with residual network design, and FNN+FFR in Figure 5 could answer this question.

Q2: Under the discretization setting, what are the differences between time series modeling and functional networks?
limitations: limitations: The limitations are well discussed, but the authors have not stated the GPU specification in Appendix D.2

Comments: title: Major concerns resolved
Updated review
comment: Dear Reviewer DyRG,

Thank you for following up on our response to your revised rebuttal. We appreciate that you have raised the score above the acceptance threshold and clearly outlined what was required from us to earn your recommendation for acceptance. We thank you again for your suggestions to change the notation, add details on experimental details, and include an illustrative example. We will incorporate these into the revised manuscript, which we believe will further strengthen our paper.
Thank you for the rebuttal. Given the updates you say you will make to the paper, I have raised my score to a 5.
Dear Reviewer DyRG,

Thank you once again for your thoughtful comments and for clarifying the criteria for recommending acceptance. We hope that our previous response has addressed all your questions and would like to know if there is anything else we can provide at this point.
Dear Reviewer Ta2r,

Thank you for reading and acknowledging our rebuttal. We would like to briefly comment on the statement

> model interpretability requires domain knowledge in biomechanics to perceive, which refrained the general audience from intuitively understanding the power of the model.

While it is true that the interpretability **of the results** requires domain knowledge (which applies to every field, not just biomechanics), the interpretability of the model itself does not. The coefficient surfaces estimated by our model can be interpreted like any other bivariate effect (similar to those in spatial applications --- however, instead of latitude and longitude, we provide a relationship between $y(t)$ and $x(s)$). 

We assume that this is what the reviewer meant, but we wanted to clarify this point to ensure nothing was left unaddressed while we still have the opportunity to respond to comments. 

Thank you again for your time reading our paper and providing the review. Please let us know if there are any remaining concerns that we should address.
Thank you for the authors' reply, which addressed much of my concern. This paper and the authors' replies both demonstrated a solid technical and theoretical foundation. However, as the author has mentioned, model interpretability requires domain knowledge in biomechanics to perceive, which refrained the general audience from intuitively understanding the power of the model. Thus, I will maintain my score unchanged.
Dear Reviewer SfMu,

Thank you for your thoughtful feedback and for taking the time to review our rebuttal. We appreciate that you adjusted the score in response to our rebuttal. We will ensure that all explanations and results from the rebuttal are included in the revised version of our manuscript. Additionally, we will dedicate a separate section to discussing the reasons why the semi-structured approach boosts performance in some cases but not in others. As stated in the general rebuttal, we believe your review has allowed us to further improve our paper through additional clarifications and experiments, and we are thankful for this input.
Thanks for your efforts in providing additional results, in particular verifying study [24] in your experimental setup, and for justifying your study's suitability to NeurIPS. I must admit that I am not an expert of this field and still have difficulties in assessing the relevance, but your arguments appear sound to me.

The additional results provided in your rebuttal are convincing. They help me to further understand where your model is superior; A thorough interpretation and discussion of these results would be helpful for readers. That is, for what reason does you semi-structured approach boost performance in some cases but not in others?

Assuming the explanations and results from the rebuttal are carefully woven into the manuscript, I am raising my score from 4 to 6 but lower my confidence further from 3 to 2, as I do not feel qualified to provide a strong and well justified assessment of this work. Good luck!
Dear Reviewer DyRG,

Thank you very much for reading our rebuttal and your prompt response. We also appreciate the revised review, which will help us to improve the clarity of our paper. Below we respond to the raised questions point-by-point:

> How much tuning has been done for the pure deep-only network baseline from Section 4.2.2? 

That is a very good question, and we can understand your concerns well. However, note that our previous response also applies to the application in Section 4.2. The quoted sentence “we use a tuned InceptionTime [16] architecture from the biomechanics literature [24]” is given in Section 4 and applies to both subsections. In other words, we did not tune the deep network part (for both the deep-only and the semi-structured model). We will make this more clear in a revised version of the manuscript.

> I would be a bit worried that the findings are very dependent on the choice of architecture of this model? E.g. does it have any increased capacity over the deep part of the semi-structured network? 

This is another good question that the reviewer raises. Note, however, that the deep network is the same for the “deep-only” model and the semi-structured network. 

> And does this architecture have suitable inductive biases for fitting linear functions (maybe residual connections which add a linear transformation of the input to the output)?

Good point. We can confirm that the architecture is suitable for fitting linear functions. A large part of the variation in these applications is explained by the (linear) function-on-function regression, which the InceptionTime model can also capture. This is also apparent from the fact that the orthogonalization has a large impact on the explained relationship (Figure 3), from predictions in Figure 8 in the Appendix and further confirmed in [24], where the deep network can represent a similar function space as the function-on-function regression (at least for this specific application).

We will make these points more clear in a revised version and thank the reviewer again for bringing up these points.

> The presentation could be made clearer in a couple of areas. In particular, (1) Equation 3 is the first time that the “double function call” notation of e.g. $\lambda^-(X)(t)$ is used. Having a double function call makes sense given that there is a function which takes X as input and outputs another function. However it is not clear to me why this is then not used in Equation 1, where you could presumably replace $\mu(t)$ with $\mu(X)(t)$? 

We thank the reviewer for pointing this out. The reviewer is absolutely correct and $\mu$ is also a double function with the first argument being the data $X$. In line 101, we defined $\mu$, writing “An FFR for the expected outcome $µ(t) := \mathbb{E}(Y(t)|X=x)$" and explicitly dropped the dependence (by considering $X$ to be fixed with realization $x$). We agree, however, that this might lead to confusion and will add the second argument where appropriate to be consistent. 

> And e.g. in the last part of Section 2.2, why write $h^{(L)}(t)$ instead of $h^{(L)}(X)(t)$? Would it be possible to make this clearer? 

Again, the reviewer is correct. We simply tried not to overload the presentation but agree that this should be consistent and will update the notation. Thank you for pointing this out. 

> And also (2), when functional data analysis is introduced in Section 1 and 2, it would be helpful if an example application was described

We agree and will add an example from the field of biomechanics (predicting ground-reaction forces with acceleration curves) and neuroscience (predicting muscle movements from brain signals).
Hi, please see my updated review which hopefully makes it clearer what needs to be done for me to recommend acceptance