Mixed effects in machine learning – A flexible mixedML framework to add random effects to supervised machine learning regression
Authors: Pascal Kilian
Sangbeak Ye
Augustin Kelava
Venue: Accepted by TMLR ()

Abstract: Clustered data can frequently be found not only in social and behavioral sciences (e.g., multiple measurements of individuals) but also in typical machine learning problems (e.g., weather forecast in different cities, house prices in different regions). This implies dependencis for observations within one cluster, leading to violations of i.i.d. assumptions, biased estimates, and false inference. A typical approach to address this issue is to include random effects instead of fixed effects. We introduce the general mixedML framework, which includes random effects in supervised regression machine learning models, and present different estimation procedures. A segmentation of the problem allows to include random effects as an additional correction to the standard machine learning regression problem. Thus, the framework can be applied on top of the machine learning task, without the need to change the model or architecture, which distinguishes mixedML from other models in this field. With a simulation study and empirical data sets, we show that the framework produces comparable estimates to typical mixed effects frameworks in the linear case and increases the prediction quality and the gained information of the standard machine learning models in both the linear and non-linear case. Furthermore, the presented estimation procedures significantly decrease estimation time. Compared to other approaches in this area, the framework does not restrict the choice of machine learning algorithms and still includes random effects.

Published in Transactions on Machine Learning Research (02/2023)
Mixed effects in machine learning A flexible mixedML framework to add random effects to supervised machine learning regression
Pascal Kilian
wesoul.iikanDuunitteeeengg
Methods Center University of Tübingen
Sangbeak Ye
wungbeak.ye@wnni-tteeee
Methods Center
Universily of Tübingen
Augustin Kelava
augustin. grnnatii..kdaaaa..eennttttnnnnne
Methods Center
University of Tübingen
Reviewed on OpenReview: https: // http:///epereeeeee.emmee net/ forum? id= 28sortt
Abstract
Clustered data frequently be found not only in social and behavioral sciences (e.g., multiple measurements of individuals) but also in typical machine learning problems (e.g., weather forecast in different cities, house prices in different regions). This implies dependen- cies for observations within one cluster, leading to violations of independent and identically distributed assumptions, biased estimates, and false inference. dress this issue is include random effects instead of fixed effects. We introduce the general mixed effects machine learning framework (mixedML), which includes random effects in su- pervised regression machine lcarning models, and present different estimation procodures. A segmentation of the problem allows to include random effects
typical approach to ad-
additional correction to the standard machine learning regression problem. Thus, the framework can be applied on top of the machine learning task, without the need to change the model or architecture, which distinguishes mixedML from other models in this field. With a simulation study and empirical data. sets, we show that the framework produces comparable estimates to typ- ical mixed effects frameworks in the linear case and increases the prediction quality and the gained information of the standard machine learning models in both the linear and non-linear case. Furthermore, the presented estimation procedures significantly decrease timation time. Compared to other approaches in this area, the framework does not restrict the choice of machine learning algorithuns and still includes random effects.
1 Introduction
Clustered data, for example, students in schools, house prices in neighborhoods patients in hospitals are typical situations in the social and behavioral sciences, economics, and medicine (see [Sijders & Bosker 2011). It can be expected that observations within the same cluster share similaritics that don't exist between observations in different clusters. If these similarities between the observations (or individuals) are not taken into account, a serious violation of the independent and identically distributed (i.id..) assumption has to be expected, lcading to biased estimates, false inferences, etc. Throughout this paper, we use the example of predicting a student's test score. After including information on the student level (level 1), result of unobserved information
like age cognitive ability, the prediction residuals can be

---

Published in Transactions on Machine Learning Research (02/2023)
about the students. These residuals are assumed to be ii.d., but within-school (cluster) similarities violate the independence assumption. The within-school dependency can be described by school-level (level 2) information like average cognitive abilities. On this level, unobserved information (like class sizes) might also inffuence prediction results.
In order to account for the dependencies of clustered data, one way is to introduce fixed effects (e.g., dummy variables/onee encoding, which describe the membership to a data cluster beyond the available cluster information). However, this leads to rapid increase in the number of parameters and to a loss of degrees of freedom. In the past, another way to account for the dependencies has been to include random effects. In this scenario, for each cluster, a random draw from a parametric distribution (i.c., normal distribution) is conducted, which represents the cluster effect in the measurement. In other words, each observed value is decomposed into a part which is shared by all observational units for the same cluster and another part that represents the deviation of an observational unit from the cluster (mean). The random effects approach implies that only those paramcters are estimated which belong to the parametric distribution (e.g., one mean and one distributional parameter for the normal distribution). As result, the loss of degrees of freedom is limited. In the school example, this means on level 2, use the observed school information (e.g. average cognitive abilitics) as fixed effects that influence the students' test score prediction. The effect of this school information is the same for all students (dependent on the average cognitive abilities of their respective schools). But for each school, a random draw from a parametric distribution (as a realization of a random variable) describes the deviation of the expected school influence on the prediction (accounting, for example, for different class sizes / student-teache scores). In this sense, it is also helpful to think about the specific set of schools, included in the data, as random subset of all possible schools.
ratios that are not observed but are expected to influence test
In this paper, we will describe how random effects can be included in the more general setting of machine learning algorithms. In contrast to self-contained models, we present a framework that can take existing models (without random effects) and extend them (by fino-tuning) by random effects. We will also discuss and introduce different estimation procedures for the variance components of this framework. In the next sections, we introduce the significant related work in this area and describe the contribution of our framework to the existing literature. In section 2 we introduce mixed effects models and the notation for the paper. In section g we describe the proposed framework followed by the estimation in section 4 Following the properties of real data set, we use simulated data for evaluation in section 5 and apply the framework on real data in section
6
Related work The idea of adding random effects to prediction algorithms is, in principle, known. However, previous research mostly focused on tree- based methods for the (non-linear) prediction of the fixed effects part of the problem. Fokkema et al. (2018) presented the (gencralized) lincar mixed effects model troes (GLMM tree). Similar results can be found as (generalized) mixed effects regression trees or random forests in Hajjem et al.] (2011: 2014 2017] or Sela & Simonoff] (2012). Also, Nestler & Humberg (2021) proposed a cross-sectional regression tree and added regularization (LASSO) and more random components. Most of (EM) procedure, where standard mixed model frameworks are incorporated (from here on referred to as out-of-the-box - OOB implementation). This approach is widely known as the Random Effect- Simonoff 2012). Some frameworks (e.g. Hajjem et al. 2017 Xiong et al. 2019) did not use OOB functions, but rely on Maximum Likelihood Estimation (MLE)- based RE-EM. In the framework Mixed effect machine learning (MeMI., Ngufor et al.] 2019) a more general approach suggested, where (in theory) the choice of the machine learning model for the fixed effect prediction is free, and, for the random effects, a GLMM model is used. While they pointed out that, in theory, all kinds of machine learning models can be used, the implementation and application still is restricted to tree- brsed models (with an indicator variable) and OOB frameworks for the random part. The same idcas can be found in Xiong et al. (2019). Here, deep learning architecture (from the field of computer vision) was applied to receive representations of images. For those representations, an EM algorithm is used to estimate the mixed effects model. Mandel et al. (2022) developed a model which substitutes the lincar predictor and therefore bears the highest resemblance to the mixed effects machine learning (mixedML) framework described here. Using simulated data and data set to predict depression and anxiety levels of schizophrenia
these approaches are based on an Expectation-
(RE-EM) tree from (Sela &
GLMMss with a food-forward noural network
2

---

Published in Transactions on Machine Learning Research (02/2023)
they showed the advantages of their model. We refer to this paper for a very thorough derivation of model that combines machine learning and mixed-effect modeling. While we absolutely recommend the recent research results for application and as a presentation of different models in this area, we show with this paper that random effects can be considered on the basis of a broad range of cxisting models, even after they have been trained. In this sense, this framework a generalization of the above results that includes most of the introduced methods. Therefore, in this paper, we do not aim to introduce new model that competes with the mentioned research. Instead, we unify the methods in one framework and add new estimation methods within the framework well as provide more fiexibility for the choice of the ml approach.
Contributions Our framework differs from the mentioned literature in the flexibility of the machine learning regression procedure. This is because are not presenting model that combines specific machine learning algorithm with random effects, but rather framework that uses placeholder for the model that is used for the fixed effects part. Further, our framework merges and complements varions estimation methods presented previously.
Therefore, our contribution to the literature is three-fold. First, we combine different approaches into one framework. Here, we refer to the estimation procedures. In general, and also in our framework, these are based the introduced RE-EM idea (which was applied RE-FM trees to tree-based models). As presented, both MLE (similar to Hajjem et al. 2014 Xiong et al. 2019 and OOB methods were introduced within this approach. Besides these methods, our framework also provides Restricted Maximum Likelihood estimation (REML) which is a common approach to estimating variance and covariance components in mixed effects models. In a simulation study, those estimations procedures are compared, especially with respect to computing time which decreases significantly for the MLE/REML procedures compared to OOB.
Second, our framework provides great flexibility in the estimation of the fixed effects. The framework can directly be applied to wide range of supervised regrossion machine doep learning models and thus increasing their predictive power (Sela & Simonoli] 2012). A very important aspect here is how the models are trained within each FM iteration. The models only need to be adjusted with respect to a slightly changed target variable in each iteration (comparable to transfer learning). This means that a trained model only needs to be finc-tuned in each iteration, which is a considcrable time-saving when using model-dependent convergence criteria. From this, another enormous advantage of the framework can be derived. While the framework allows to train a model from scratch, an already specified and trained model can be used and adjusted under consideration of random effects for the reason described above.
Third, we use the simulation and real data to access the benefits of the framework in the sense of model specification. Here we show that for prediction quality, it is not necessary to specify non-lincarities like interactions in the model (the traditional way in psychometrios) as long as the machine learning procedure is capable of modeling the non-lineat relationship.
Through the above points, we complement the existing mothods. Specifiod as special cases, the framework can also serve as summary of the mentioned research (e.g. (ree-based methods can still be used as specification). Taken together, these enhancements will enable researchers to investigate a possible prediction improvement in the mixed model domain (under consideration of different implemented estimation procedures) after having completed the analysis in the machine learning domain, using all the time and work previously invested [hyperparameter tuning, training of the model, ...). Tn that sense, the framework can be applied on top of existing models and analyses. With this, the advantages of the usage of mixed effects become available a broad range of machine learning rescarch.
2 Mixed effects regression models
2.1 Linear Mixed Effects (LME) Model
Using the notation of Laird & Ware 1982 model can be described as follows:
Demidenko (1987).
the traditional mixed effects regression
y y= 2,u; + te
[1,
(1)
3

---

Published in Transactions on Machine Learning Research (02/2023)
where y, is the (ny x 1) vector of observed dependent variables (vector of test scores of the ny students in school j, from the introductory example) for each cluster å (ny is the sample size within a cluster j), X, is the (nj x p) matrix of observed variables of ny observational units, & is the (p x 1) dimensional vector of unknown regression coefficients (parameters), Zj is a (n, x pr) design matrix, Vj is a (p, x 1) vector of unobserved random variables (random effects) with My (0 E), and Ej is the (ny x 1) vector of normally distributed residual variables with E ~ (onnn... I.,)
Note that in X,, collect the level 1 and level 2 covariates columns (observed student (level 1) and school (level 2) information). In other words, the first p1 columns describe the observed covariates of the Ry observational units from cluster j, and the remaining P2 columns describe the observed covariates which give information on the cluster j (being constant for the n2 observational units form cluster j). Pr is the number of observed covariates for which random effects are included. Without loss generality and for the ease of notion, we will assume that the covariates are ordered, so that the first p, covariates are assumed have cluster-deenndent (random) effects.
Furthermore, we use the notation új X3 + Zzū; for expected values and define for an individual obscrvation of the cluster j:
Dsj Dig + ūj
with the respective row ž1 = (1, Biy) in the design matrix Zj. consisting of a bias/ hassintereept term (= 1) for the random intercept and the feature vector Rij = æzi [1 pr]. The estimated random effects (i.e., realizations of the latent/randon variables) are ú (ū,,.. ..., a' with új (0g, ... Upeans) for cluster j. Then y) is conditionally normal distributed
yi . Ny, fxed Vj),
fxxd
(3)
V, = Z, + a Inj-
(4)
In matrix notation we can write (over all J clusters): = x3 + Zu te with v = (V/, block matrix
"i,...,,/ U/)y and the design
21
==
(5)
ZJ
Variance and covariance components in LMEs are typically estimated by the EM algorithm, based on either MLF REML. As elaborated in Raudenbush & Bryk (2002), MLF estimates are asymptotically consistent and efficient. However, for small sample sizes on level 2 (i.e., number of clusters; which is not the case in our paper), MLE estimates of the variances and covariances of the random effects will be underestimated. In order to avoid this issue, REML estimates can be obtained which give unbiased estimates and take into account the unreliability of the fixed effects. Again, we would like to emphasize that this is an issue for small sample sizes given in social and behavioral sciences (e.g., with a number of clusters < 30). As a result of this, in practical situations estimation of the random effects variances and covariances relies on REML, and estimation of fixed effects relies on MLE in so-called multilevel modeling.
2.2 Non-linear mixed effects models
As a generalization of the lincar mixed effects model, the non-lincar mixed effects model specifics the fixed effects term as a non-linear function of the population parameter B as follows:
= fa(,) + 2,43 + +5,5J E JJ.
(6)
As consequence, we have a closed-form analytic expression for the marginal mean and covariance terms of the model, Ely,] - fx(X,) (following the conventional notation of non-linear regression model).
Due to these properties, this model is referred to as non-linear marginal models Demidenko (9877 or as population-averaged models (Liang & Zeger (986)) Zeger Liang] (1986)). ) because the marginal function of the response variable is expressed as a function of population-leve covariates. This specification differs from the fully non-linear mixed effects model in that the random effects not included in the non-linear
4

---

Published in Transactions on Machine Learning Research (02/2023)
kernel, wherein this model is also called the partially non-linear mixed effects model (Davidian & Giltinan (1955)) In non-linear mixed effects models, estimation of the models relies on the marginal distribution of the response variable. However, as we assume an explicit expression of the distribution of the mean and covariance structure of the response variable, there is no need for integration.
The model for the dependent variable then takes a conditional form, y y) +x) Zz t Without loss of generality, we assume that Ej and Vj are normally distributed. The non-linear mixed effects model can be written the following:
プ N(F Lng))
(7)
and fx((;) = =yfixed as before.
3 Mixed effects models for machine learning
While the fixed effects approach can be implemented straightforwardly using one-hot encoded variables for the cluster membership, it is not obvious how to implement the random effects. In the next section, we introduce the framework mixedML. to implement the random effects.
3.1 Mixed effects approach mixedML
If the data contains features on both the observation level (level 1) X1 NXP1 and the cluster level (level 2) X2 E Jxpx we include the level 2 features with fixed effects in the level 1 prediction. For that we combine the matrices to obtain X = [X with X₂ E NRx, where row i of X₂ is row j of X2 for all in cluster j. Note that this does not correspond directly to a mixed model with level two covariates since the level 2 covariates only determine the cluster deviations from the overall intercept, instead of inftuencing the cluster-specific slopes. In our model, the spuster-specifc slopes are only determined by the centered random effect. From here on X is seen as feature matrix that might contain level 2 features, which are constant across observations within the same cluster.
Instead of estimating a linear relationship between X and y via X for the fixed effects, we use potentially non-linear function f(X) to model this relationship via y = f(X) + Zu +e and then add the linear random effects with the same assumptions as in section 2
To paramcterize the function f, we use a machine learning model, referred as ml fised(0) and add the random effects (cluster differences), which can be seen as a linear correction. This results in
y mi (X) x)++ + Zu t
(8)
or in the cluster specific notation (compare to i = m0 (X,) + Z,új. For the estimation we split y into a fixed and random part y y fixe + y rand with the cluster specific realizations u (u'.... ) of the random ifffects/laten variables v.
+ E where fixed = y - Zu and yand = fixed = y - y fixed
Before we describe further details in the next section, we give general idea, following related ideas for tree- based models (e.g. Hajjem et al.) 2011 2014 Sela & Simonoff 2012) and Hajjem et al. 2011 Xiong et al. (2019h. We train mliixed(e) by ignoring cluster effects (with the target y) to get a first estimate for y nxed (the prediction gfixed of the model). With this we estimate à based e fixed Then update y fixed = y Zä using the current estimate é and re-train mlsxed(0) with the updated target variable fixed, This process is iterated until some convergence critcria are met or the maximal number of itcrations is reached. Since, for every ileration, a (potentially complex) machine learning algorithm needs to be trained, the question of training time arises. To provide some clarity here, in each iteration the model is re-used and fine-tuned to an adjusted prediction task. In the first itcration, we train the model for a task (),)). In the subscquent iterations, we update the trained network of the last iteration and tune the parameters for the updated task (X, fixed) (comparable to transfer learning).
5

---

Published in Transactions on Machine Learning Research (02/2023)
4 Estimation
To get an estimate for y fixed we learn the parameters O of fixe((()) (e.g. by SGD methods) and predict yfixed m1 fxed(() (X). If is known, common supervised machine learning regression methods can be trained and applied to predict yfxed, including the special case of u = (and thus y yy where random effects are ignored (typical case). Since the random part the model (and therefore u) is unknown, we need to find estimates á to use in (3.1) to define the machine learning task. But to get an estimate for the random part of the model, we need the estimate gfixed for [3.1 This motivates an alternating, iterative process like the EM-algorithh where estimmaees of the current fixed random parameters are used for the
respective updates (see e.g. Sela & Simonoff. 2012 Hajjem et al. 2014 Xiong et al. 2019
4.1 EM
Following common techniques for the EM-algorithm (Lindstrom & Bates 1988 Wu & Zhang 2006 Laird & Ware 1982 (similar in Hajjem et al. (2011)),
use the EM algorithm
estimator for the
variance components a and Σ (for both, MLE and REMI... MLE and REMI, estimates can be obtained by using the respective projections PJMLE- Vj-1 and PJREML = Vj V, (X"XV
-x-x))xXX X (X))
(see
e.g. Lindstrom & Bates 1988), where V, is defined as in 4 Given the estimates de 1) (t- and ú (t-1)
from the last iteration (t - 1), we can (re)train mi (t)
fixed (X) using the target y = yj Z, ú (t-1)
and
(t) calculate 81 . and Vj( (tt-1) (evaluated at (((- as
((-1) = -, Z) + 0²I, 1 (1) (( 1) Z3 V 11-1
yj y ml fxed (Xi)
yj - m1 fxxd (X,) Z
The new estimates /MLE/REMI and (f+1) MLE/REMI. are calculated as
(++1)
(())
(tyT i
-
((( 21 (TT(11)
DELE/REML 21
MLE/REML
+
==1
(OSILE/RETL) )²=
rs 3 + o
(t)T (t)
2(r 211-1)
2(t-1) (I o2t (2to((-1)p((t1
JML /
N 1=1
Note that instead of inverting V,, Vj I,..., J we can invert E and use V,- 1- (2" zj
where (z Z, + - is usually smaller (number of random effects squared) than V, (ng x n2).
Convergence is monitored by the generalized log-likelihood (GLL)
GLL(0,,,lu.)) = Σ (o: + +u u -'uu; + log | + log [o²
(9)
We implemented this approach and presented the algorithm in Table [ Note that the separated estimation of random effects and fixed effects in each iteration t gives great flexibility for both parts. For the fixed effects, we can implement any supervised regression machine learning algorithm that learns to predict y fixed and produces the estimate) ppretcciooiiiiionn fixeed For the random elfects, we can also use standard LME methods (e.g. MixedLM() from the statsmodels module (Seabold & Perktold 2010)) in python, or Imer() from Ime4 [Bates et al. 2015) package R) where the dependent/tane ttarget variable defined as rand(() y fixe((1) in iteration As mentioned before, we refer to this implementation OOB (see Appendix for the algorithm).

---

Published in Transactions on Machine Learning Research (02/2023)
Table 1: Implementation of the MLE/REML based EM estimation.
Algorithm
t= 0
bised = U. Ev(o) = 1,8 (0) = and train fixed((0) (X) Initialize 4,(0) =0, =0,Vj = b..., J. (0)
+ fixed
V(0)
(())
1-1-1
for all j
V 1)) + 7
8 (1 2), / (1-1)-1 ml (1)
Exed (X3)
(1) = v, =1 nixdd (x,) 4
MLE: PJMLE = V
REML: PAREML- V.
Vj Xx (-) X
(0
MLE/REML
2==1
(t) (t)T
1 - E(t 23 1/ p(t1)
RELE/REMI Z, (t)()
o
MMLE/REML
(1)
Ej=1 [ )
119 (t) 2(1-1) trace
+ de
trace 1 (1- 2((-1) p (tt-1)
MMLE/REMI
(())
Bixed = Zie(e)
train (1 (X)
M(()
fxee
-> fixed
()
Note that both algorithms include an isolated and complete training (using convergence criteria) of the machine learning procedure (ignoring random effects) in the first part of the first iteration. This provides great flexibility in the choice of the machine learning procedure and the respective training process. Within the optimization, the machine learning model can be learned from scratch, but the process can also be started with pre-trained models, which leads to carly stops according to convergence criteria. In this case, the existing pre-trained model will be fine-tuned lo consider random effects. In every subsequent iteration, parameters will be updated according to updated expectations of the random effect. We will refer to those three estimation methods as MLE, REML, and OOB.
4.2 Prediction
The problem of predicting an unseen data point can be divided into two tasks. In the first task, we want to predict the outcome Yey of new observation i' in the cluster j, and the cluster known from the training data (j [I.....]]] Vl). We call this task within sample prediction. In the second task, we predict a new outcome where we have information about the cluster (the level 2 features), but the cluster itself might not be one of the clusters considered during training. We refer to this task as out of sample prediction. The predictions are set to
nut ffxeed m (x) (Xvr); within 02j ml ( (Nvtz) + Zūj, Ue
(10)
(11)
where Z is specified as above and ún is the realisation of v for the known cluster j. Note that we use the very simple approach of using the information of random effects if it is present. Random effects could also be predicted for unknown clusters as in Xiong et al. (2019). For simplicity of the framework introduction, we have refrained from including this in the paper to focus model comparisons with and without the consideration of random effects. In the test score example, the within sample prediction is the test score prediction of student in a school that was included in the training data. Here we know the schools' realization of the random effects following the population (of possible schools) distribution and we can use this information. In the out of sample prediction we only know the observed information of the student's 's school and thus can only use the common fixed effects.
5 Simulation studies
To evaluate and compare the models, we conducted a simulation study. The simulated data sets follow the properties of a real data set. from social sciences. We give the relevant settings in this section. Further details
7

---

Published in Transactions on Machine Learning Research (02/2023)
can be found in Appendix B
For sample sizes used J == 200 with nj sampled between 22 and 25 (giving N = 5707). The dimension of the feature space is pt 10, P2 = 3, and includes random effects for all level predictors (p,- = - pt+titercettbbaas
For the covariance matrix Σp of the pr + 1 random effects (level 2 latent variables) we assume Un = for n # m and set o2 1, the latent variable vector.
1, Vm [0, P.l.. By setting the variance of the random effects one, we identify
Cluster specific realizations uj of the latent variables are sampled as u, ~ ~N(0,,,)) for j = 1, I....JJ For the residuals we assume Eg = I with o, = 3.5, and sample c <~ (O, (c E IR
We set the intercept/biiss 8o = 5 and all other parameters to 1 and obtain B1 E Rei (level 1 parame- ters/weights), Bg E IR (level marameters/weightt)) With the chosen values, together with oe, we achieve an overall explained variances of y of around 0.77 (lincar) and 0.82 (non-linear), which would already be considered as high but plausible (at. least. for repeated measures) in the social sciences.
To show the downward compatibility of the framework, we simulate the linear case by ylin B Zu-t. To show the benefit of more complex relationships, we simulate the non-lincar data set by including quadratic terms of the first five level 1 features and define Xg = (2f11..., xis) and Bg E R sampled as the other coefficients. This gives
Ynlin = 3o + x X, + Zu t
(12)
5.1 Evaluation and comparison metrics
In the analysis, we always specify and train the machine learning algorithm first and report the results (m1). As a fixed effects comparison, we specify the same model but include the one-hot encoded cluster-membershjp membership as predictor (hot). We then use the respective all introduced estimation methods (MLE, REML and OOB). Following this procedure from here on, we only introduce the ml method which the other models based on.
model within the mixedMT. framework and train with
In the evaluation and comparison of the framework, we focus on two aspects. First, we compare models with regard to parameter estimation and estimation time. Second, we compare the prediction qualities for new in-sample observations (wilhin the cluster of the observations is known from the training data) and out- sa-ll observations (out - all information is given, but it unknown the cluster one of the training clusters). For the prediction quality, we use the aat--maan-squurc--rree (RMSE). Again, the test score example, an in-sample observation (wilhinn refers to student who goes to a school that is known from training (and thus the respective estimated mean deviation can be used). Tn an out-of-sample observation, the student's 's school is one of all possible schools, and apart from the common fixed effects we have no further information. The differences in the according RMSE values are according to whether or not this information can be used. Note that the ml and hot models don't include the random effects. This means the within and out scores are the same for the ml model (a within-sample observation does not provide further information). For the hot there is no real out score because only known clusters are represented in the encoding.
5.2 Models and procedure
In this paper, the focus is to compare approaches regarding the consideration random effects. In order to perform the comparisons on reasonable machine learning models, hyperparameters for those models are tuned scparately (for details, see Appendix C). The same models (with the found specifications) then are used as pure machine learning models (m1) and within the mixedML framework.
In the linear case, we specify the ml model to be linear regressor and use a linear feed-forward neural network (NNim no hidden layer) and compare it to traditional LME model (statsmodels), specified the same way. Note that NN is an unusual and not very efficient way to implement linear regression, but we decided to use a neural network for the simplest (linear) case to examine a model from a model class which can be extended to general cases without further ado (only by changing the architecture). We compare the fixed effects/ /weitts of the problem, the variance components, the prediction quality, and the different est imtimation times for the algorithms.
8

---

Published in Transactions on Machine Learning Research (02/2023)
In the non-linear case, we first estimate two LME models. The first model uses only the original (linear) features (TMMBia); the second (non-lineau) LME. knows about the simulated quadratic terms and they are included as input (LMEntna). The latter will serve as a baseline (in the ground truth sense) for flexible machine learning predictors. Here only the original features are used and the question at hand is the non- linear relationship can be modeled. The flexible ml models in the non-linear case are a 3-layer feed-forward neural network (NNatin with 5 and 3 hidden units) and a Support Vector Regressor (SVR with a radial basis function kernel).
The model specifications are shown in Table 2
Table 2:
The different models considered for the evaluations.
LMB
Lincar Mixed Effects model with X
pME
Linear Mixed Effects model where the specified non-linear
included in X
(we know the exact (simulated) non-linear relationship, see (2)
We use Mi from statsmodels with the conjugate gradient optimizer
ml:
X00
O-hidden-layer neural network with linear output activation as the linear regression special case with Adam optimizer (Ir = 0.01).
Mllin
2-layer neural network with 5 and 3 hidden units (4 and 4 units for the hot). We specified ReLU activation for the hidden units and linear output activation and used tensorflow-keras for the implementation with Adam optimizer (Ir = 0.01) and mean- squared-error loss.
SVR
Support Vector Regressor implemented in sklearn (for hyperparametors see Ap- pendix
pure Fet
mixedML. implementation of the EM algorithm (Est: MLE REML) the out-of-
the-hox approach (Est: OOB, from statsmodels) with as the fixe(())) model.
In order to obtain measures of the stability of the parameter estimates, we performed a 50-fold cross- validation for all models, that means in every fold we estimate the model on a training set of a random 98%-2% split and evaluate it on the respective test set. We repeat this procedure 50 times so that every observation is exactly once in the test set, then means and standard deviations are reported. Run times for single folds are reported with the results (2,8 GHz Quad-Core Intel Core 17). For ncural networks, we implemented the convergence criteria of loss differences < 0.1 for 8 conseculive epochs. As mentioned earlier, for the EM algorithm, we use the same procedure based on the GLL (9 for all models.
5.3 Results
See Table the results of the linear problem. The estimated parameters and variances of the models are in line with the traditional model (LME), indicating an unbiased parameter estimation. In terms of prediction quality, all models and estimation methods show similar results on the training and test data where a significant increase in prediction quality can be noted when the information about the cluster membership is used ( (within) compared to observation from unknown clusters. In the pure machine lcarning models (m1), the cluster membership cannot be used, so there is no within score comparison. The out scores are in line with the pure machine learning prediction results. The fixed effects machine learning approach (hot) should be compared to the train- and test- within scores, since the cluster membership used as fixed effects. For both scores, we models. Even compared to the pure machine learning models (m1), the one-hot encoded cluster membership only leads to little increase in quality. While parameter estimates and prediction qualitics seem to be equivalent. for the three estimation methods, the estimation time (for one fold) differs dramatically, with the biggest mean difference between the out-of-the- box (OOB) estimation and the MLE/REML procedures (At 233.62s (MLE) and At = 228.87s (REML)). Note that the estimation times for our implemented MLE/ /REML estimations are also faster than the traditional LME estimation time.
decrease in the prediction quality of the hot model compared to the other
The non-linear results are reported in Table 4 All parameter estimates are stable, but two main differences can be seen. When the linear traditional model (LMEn)) is compared to the traditional model where
We refer to this model non-linear distinguish from the linear LME and hecause the non- lon-linear terms are considered as features. Using those features, the estimated relationship, and the model itself is nevertheless linear.
9

---

Published in Transactions on Machine Learning Research (02/2023)
Table 3: Results for the linear problem (data).
LM
NN
NNMLE
NN
NN
NN
Pixed effects
bias
5.00
4.89 (0.01)
4.89 (0.03)
4.88 (0.05)
4.89 (0.04)
X1.1
1.00
0.97 (0.01)
0.98 (0.03)
0.99 (0.05)
0.97 (0.04)
X1.2
(0.01)
(0.04)
(0.06)
1.00 (0.03)
1.00
0.98 (0.01)
0.98 (0.03)
0.98 (0.05)
0.98 (0.05)
X1.4
1.00
(0.01)
(0.03)
(0.05)
(0.05)
1.00
1.03 (0.01)
1.04 (0.03)
1.03 (0.06)
1.03 (0.06)
X1.6
1.00
0.88 (0.01)
0.87 (0.03)
0.88 (0.05)
0.88 (0.06)
X1.7
1.00
0.89 (0.01)
0.89 (0.03)
0.88 (0.04)
0.90 (0.04)
X1.8
1.00
1.01 (0.01)
1.01 (0.03)
1.01 (0.05)
1.01 (0.04)
X1.9
(0.01)
(0.05)
(0.03)
1.00
1.06 (0.01)
1.06 (0.03)
1.07 (0.05)
1.07 (0.05)
X2.1
1.00
0.85 (0.01)
(0.04)
(0.04)
0.86 (0.05)
X2.2
1.00
1.10 (0.01)
1.10 (0.03)
1.09 (0.04)
1.10 (0.04)
X2.3
1.00
1.02 (0.01)
1.02 (0.03)
1.02 (0.04)
1.02 (0.04)
Variance components
Var(bias)
1.00
(0.02)
(0.02)
(0.02)
1.07 (0.03)
Var(m1)
1.00
0.78 (0.02)
0.78 (0.02)
0.78 (0.02)
0.79 (0.02)
Var(Ps)
1.00
1.05 (0.02)
1.05 (0.02)
1.05 (0.02)
1.06 (0.02)
Var(ira
1.00
0.89 (0.02)
0.89 (0.02)
0.89 (0.02)
0.90 (0.02)
Var(ma)
1.00
1.08 (0.03)
1.08 (0.03)
1.07 (0.03)
1.08 (0.03)
Var(ms)
1.00
0.90 (0.03)
(0.03)
0.93 (0.02)
0.94 (0.02)
Var(mr)
1.00
1.04 (0.03)
1.04 (0.03)
1.06 (0.03)
1.07 (0.03)
Var(wy)
1.00
(0.02)
(0.02)
(0.02)
(0.03)
(ar(us)
1.00
0.81 (0.03)
0.84 (0.03)
0.86 (0.03)
0,87 (0.02)
(mo)
1.00
1.20 (0.03)
1.20 (0.03)
1.21 (0.03)
1.22 (0.03)
(ar(
1.00
1.24 (0.03)
1.24 (0.03)
1.23 (0.03)
1,25 (0.03)
Cov (mean)
0.00
0.01 (0.00)
0.01 (0.02)
0.01 (0.02)
0.01 (0.02)
3.50
3.51 (0.01)
3.51 (0.20)
3.50 (0.19)
3.50 (0.20)
Prodiction RMSEwithin (test) RMSF (test)
4.04 (0.18)
4.04 (0.18)
4.04 (0.19)
4.04 (0.18)
4.80 (0.28)
4.78 (0.24)
4.80 (0.28)
4.80 (0.28)
4.80 (0.28)
4,80 (0.28)
4.80 (0.28)
time (s)
43.10 (3.88) 266.81 (18.34)
33.19 (2:28)
37.94 (3.59)
Notes: #1/2,²² confficients (á) for the respective features of level 1/2, var(.): estimated variances of the (pr 1) random effects, aa residual standard deviation, train/test sets scoording to 50-fold cross-validstion (means standard deviation), root mean squared (RMSE) subecripts: withim: with the corrections of the random effect realizations known for that cluster, out: ignoring the cluster (out of sample), mf only the pure mi model, hot the pure mi model with one-hot membrr-pemmrr.hip. time: estimation time for fold seconds.
interactions are included (LMEahm), we can see that the estimated variance components fit the simulated parameters (see Table 3 better for LME casiest to see for the a, estimates, where the linear model overestimates the parameter. Interestingly, the non-linear neural network (NN seems to be equally unable to capture the non-linearity LMF Here the NN estimates are in line with LMEin (note that the estimation method within each EM iteration is the same here). The estimates of the same model but with the MLE/REMP estimation indicate only slightly better fit. For the SVR model, the parameter estimates are in line with the baseline model LME again with very few advantages for MLE/REMI.
Those findings are also reflected the prediction quality. Again it can seen, that the neural network on the level of the lincar LME. The SVR prediction results can be compared with the bascline model (ground truth). Note that in the baseline LME,lin all simulated non-linearities (interactions) are specified, whereas the SVR model only uses the original features input and then models the non-linearity. At least in the sense of prediction, precise model specification is thus not necessary. For the estimation times, we see again, that our MLE/REML REEM estimation is significantly faster than the out-of-the-bor bhx implementation.
10

---

Published in Transactions on Machine Learning Research (02/2023)
50-fold cros-valldattion and standard deviation), time: estimation time for one fold raudom effects 82 de residual standard deviation, train/ teet sets according
SVRREML REMM
0.78 (0.04)
1.23 (0.04)
1.25 (0.08)
0.65 (0.02)
1.26 (0.03)
1.15 (0.03)
0.83 (0.03)
1.10 (0.04)
1.10 (0.03)
1.06 (0.03)
1.09 (0.04)
-0.01 (0.02)
4.08 (0.2 26)
86 86.48 (2.94)
SVRMLE
0.75 (0.04)
1.22 (0.04)
1.24 (0.08)
0.65 (0.02)
1.25 (0.03)
1.14 (0.03)
0.82 (0.03)
1.09 (0.04)
1.08 (0.03)
1.05 (0.03)
1.08 (0.04)
-0.01 (0.02)
4.08 (0.26)
82.51 (1 (.52))
SVR ROB
0.67 (0.08)
1.21 (0.08)
1.23 (0.08)
0.57 (0.03)
1.19 (0.03)
1.11 (0.03)
0.77 (0.04)
1.03 (0.03)
1.01 (0.03)
0.95 (0.03)
1.04 (0.04)
-0.01 (0.02)
4.10 (0.27)
256 956.97 (5.58)
Table Results for the non linear problem (data),
NML biin
0.74 (0.09)
1.46 (0.07)
1.48 (0.04)
0.72 (0.04)
1.35 (0.04)
1.20 (0.05)
0.87 (0.04)
1.06 (0.04)
1.06 (0.04)
1.01 (0.01)
1.19 (0.05)
-0.00 (0.03)
4.65 (0:32)
42.47 (2.92) 44.34 (2.90)
NN
NYMLE plm
0.71 (0.06)
1.45 (0.05)
1.47 (0.04)
0.71 (0.03)
1.34 (0.04)
1.19 (0.04)
0.86 (0.04)
1.05 (0.04)
1.05 (0.04)
1.00 (0.04)
1.18 (0.04)
-0:00 (0.03)
4.65 (0:32)
Notes: api coofficients (g) for the respective features, var()); ostimated variances the (P)
0.62 (0.03)
1.43 (0.04)
(0.04)
0.57 (0.04)
1.26 (0.04)
1.14 (0.04)
0.79 (0.04)
1.01 (0.04)
0.93 (0.04)
0.86 (0.04)
1.11 (0.04)
00.01 (0.03)
4.68 (0.29)
287. 75 (20.38)
I.MFminn
0.37 (0.02)
1.09 (0.03)
1.07 (0.03)
0.50 (0.02)
0.91 (0.03)
1.01 (0.03)
0.79 (0.03)
1.03 (0.03)
0.85 (0.03)
0.87 (0.03)
0.96 (0.03)
00.00 (0.00)
3.56 (0.01)
46.11 (5.61)
I.Fi
0.62 (0.03)
1.43 (0.04)
1.47 (0.04)
0.57 (0.04)
1.26 (0.04)
1.14 (0.04)
0.79 (0.04)
1.01 (0.04)
0.93 (0.04)
0.86 (0.04)
1.11 (0.04)
-0.01 (0.00)
4.68 (0.01)
48 45 (3.80 (3.85)
Cow (mean)
Var(#1o)
Var(zo)
Var(e1)
Var(22)
(2)
Var(24 (24)
Var(es
Var(er)
Varlez (27)
Var (2x)
Vax (20)
(s)
Lime
11

---

Published in Transactions on Machine Learning Research (02/2023)
Table 5: Prediction results for the non-linear problem (data).
RiShii
RMS
RMSE
RMSE
LME
lin
4.25 (0.01)
5.72 (0.01)
5.39 (0.39)
5.72 (0.45)
nlin
3.15 (0.01)
4.75 (0.01)
4.09 (0.26)
4.75 (0.33)
NN plin
DOB
4.27 (0.02)
5.74 (0.01)
5.42 (0.39)
5.74 (0.45)
MLE
4.21 (0.01)
5.75 (0.04)
5.39 (0.39)
5.75 (0.48)
REML
4.20 (0.01)
5.75 (0.04)
5,39 (0.39)
5.74 (0.47)
5.74 (0.02)
5.74 (0.02)
5.74 (0.45)
5.74 (0.45)
hot
5.59 (0.03)
5.76 (0.44)
SVR.
OOB
3.68 (0.01)
5.22 (0.01)
4.75 (0.30)
5.22 (0.36)
MLE
3.65 (0.01)
5.24 (0.01)
4.75 (0.30)
5.25 (0.37)
REML
3.65 (0.01)
5.24 (0.01)
4.75 (0.30)
5.25 (0.37)
ml
5.24 (0.01)
5.24 (0.01)
5.25 (0.37)
5.25 (0.37)
hot
5.17 (0.01)
5.23 (0.38)
Notes: train/test sets according to 50-fold cross-validation (means and standard deviation), root mean squared emoor (RMSE) subscripts: within: with the corrections of the random effect realizations known for that cluster, out: ignoring the cluster (out of sample), ml only the pure ml model, hot the pure ml model with one-hot mesttrr-mmmershhpp, time: estimation time for one fold in seconds.
6 Empirical evaluation
In order test the framework on real data, we use a data set introduced the textbook Snijders & Bosker (2011) (with further reference to c.g. [Banndsma & Knuver 1989 Knuver & Brandsma 1993). The data contains the result of a language Lest (larget variable) of 3758 students grouped in 211 dilferent classes/schon (one class per school). Further information on the student level (level 1) is available, where we use the students' IQ and their socioeconomic status (SES). On the school level, we use the school mean IQ (sIQ), and the school mean SES (sSES). See also within- and between-groute group regression, for example, in Snijddrr & Bosker (2011) for this particular example. We use the same data set with the same preprocessing as the textbook, meaning that IQ and SES centered (before missings were removed, which leads small shift in the centering), then the school means were calculated.
6.1 Models and procedure
Using this data, we model the relationship of the results of the language test dependent on the students' IQ and SES with the TQ and SES school means (sTQ and sSES) as level variables. We use random slopes (TQ and SES) and random intercept. Additionally, that this is non-linear relationship between the four single predictors and the target. If all interactions are first calculated and then used as predictors, the relationship becomes linear
expect interactions between all of the variables. Note
Bi = 9o + 1110u + 3,SES, + faslQ, + 3,sSES, + (SESS x 1QB) + BoSSS x xsQ2, + (SES, x sSES,) + Bs(IQQ x sSES,) + (IQ x slQ,) + SI0 x sSES,) + Z,uj + Egj.
For this linear relationship, where interactions are specified the model, we use a single layer feed-forward neural network as m]ixxe(())- For the non-lincar approach, where interactions have not boen specified in advance, we use a Support Vector Regressor (SVR)
Ví ml (TQ,,,
SES,, &Qjjs &SFS,) Zju; Ezj-
For both models, apply the REML estimation. Note that the matrix Z is the same for both scenarios, with
1
1Q
SE811
1
IQ2j
SC82)
Z,
(13)
IQnji
8e8nii
12

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Results of a mixedML model with a single layer feed-forward neural network (linear) and all
interactions included as predictors.
book results
Est. (sd)
fixed effects
int
41.63 (0.26)
41.67 (0.26)
SES
0.17 (0.012)
0.17 (0.02)
IQ
2,23 (0.06)
2.24 (0.06)
sSES
-0.09 (0.04)
-0.10 (0.06)
sQQ
0.82 (0.31)
0.92 (0.07)
SES
+0022 (0.01)
-0,02 (0.01)
SES sSES
0.00 (0.00)
0.00 (0.01)
SES sQQ
0.02 (0.02)
0.03 (0.02)
$SES sQQ
(0.04)
(0.07)
sSES IQ
0.00 (0.01)
0.00 (0.02)
IQ sIQ
-0.8 (0.08)
-0.06 (0.04)
variance components
Var(int)
8.34 (1.41)
9.31 (0.83)
Var(SES)
0.00 (0.00)
0.01 (0.00)
Var(IQ)
0.17 (0.07)
0.42 (0.05)
Cov(int, SES)
0.00 (0.00)
-0.01 (0.03)
Cov(int, 1Q)
-0.94 (0.20)
-0.91 (0.10)
Cov(SES, TQ)
0.00
-0.02 (0.01)
de
6.11 (0.95)
6.07 (0.48)
Notes:
SES: socloecooomic status, #SES/sIQ school means, x denotes interactions that were included as predictors. The
variance components refer to the variances and covariances the random effects. The REMI-EM algorithm was used for estimation. a, residual standard deviation, book results in Table 5.3 inijderss & Bosker 2011
Table 7: Prediction results for the empirical data.
RMSE
RMSE.ut
RMSEtest
RMSEtest
NN
5.82 (0.03)
6.90 (0.12)
6.31 (0.53)
6.88 (0.58)
NN (ml)
6.84 (0.08)
6.84 (0.08)
6.88 (0.58)
6.88 (0.58)
NN (hot)
6.05 (0.04)
6.38 (0.52)
SVRREML
5.80 (0.01)
7.09 (0.03)
6.31 (0.54)
7.08 (0.69)
SVRREML (ml)
6.77 (0.01)
6.77 (0.01)
7.08 (0.69)
7.08 (0.69)
SVRREML (hot)
6.44 (0.01)
6.55 (0.62)
Notes: NN interactions are specified, SVR REML only original features, train/tue /tstt sets according to 50-fold
cross-validation (means and standard deviation), root mean squared error (RMSE) subscripts: within: with the corrections of the random effect realizations known for that cluster, out: ignoring the cluster (out of sample), ml only the pure ml model, hot the pure ml model with one-hot chuster-membershipp
Again, hyperparameters of the pure machine learning models were tuned separately (Appendix C and we then perform 50-fold cross-validation to report means and standard deviations.
6.2 Empirical results
Parameter estimates for this model are shown Table f compared to the results in Snijders Bosker (2011) (Table 5.3). From this comparison, we again conclude that our REML estimation method is in line with traditional methods, even for real empirical data.
Since we don't get the individual parameters (of the features and their interaction) in the SVR model, we will use the predictive quality as the metric for comparison. The question here relates to whether a more flexible machine learning model incorporates information into a prediction that would otherwise have lo be specified directly (e.g., as interactions). The prediction results are shown in Table 7 Here we can see that the SVR can produce the same prediction results, even though the non-lincarities (intcractions) are not specified. Note also, that the SVR here suffers from overfitting (when the train and test results are compared); which should be avoided by hyperparameter tuning, which was not included in our analysis.
13

---

Published in Transactions on Machine Learning Research (02/2023)
7 Conclusion and discussion
With this paper, we aimed to, first, motivate include random effects due to the chance of increased predictive power and inference. To reach this goal, we combined existing approaches into a more general and flexible framework, which does not limit the choice of machine learning procodures that can be used. Here, every machine learning predictor can be specified outside of the framework and can then be used for the fixed effects. The framework allows to specify machine learning model-specific convergence criteria, which is highly rocommended since the fine-tuning of the model in every EM iteration reaches the convergence fast.
The second goal was to include different estimation procedures, especially adding the REML estimation, and compare parameter estimates and the predictive quality in simulation study, The results show that in the linear case, the framework is able to produce the same parameter estimates as in traditional LME models, but decrease cstimation time. Our /EELL estimations decrease the estimation time compared to the often-used OOB implementation significantly and also decreases the time compared to the traditional LME method. The lower estimation time of MLE/REML compared to OOB can only be explained by the applied variance and covariance estimation, since for both methods the machine learning fixed effects estimation is the same. Whereas in MLE/REMP only the variance components are estimated, the OOB estimates full LME model. Here random effects and fixed effects are estimated, even though the fixed effect estimation was already performed by the machine learning procedure and thus are zero in the remaining model. As result, the OOB fixed effects estimates are consequentially zero, but nevertheless, they had to be estimated, since an empty (no fixed effects) model can not. be specified. This might lead to a significant and unnecessary increase in cstimation time. In the non-lincar case, the framework shows good prediction results even if non-linearities like interactions are not specified. This is a big advantage since correct model specifications are almost always impossible and the robustness of procedures is desirable. In our analyses, as expected, there is little difference between the MLE and REML estimates. Nevertheless, we can and would generally recommend the REML estimator as it is robust to the number of clusters, as described in section 2.1
As the third goal, the performance should be evaluated not only on simulated data but also on real data set. Here the paramcter estimates are in line with the traditional estimates, indicating an unbiasod estimation. Furthermore, the same prediction results of a model where interactions are specified can be achieved without the need to overspecify a model or find the right interactions and non-linearities by hand, but by using a flexible framework for the fixed effects.
7.1 Limitations and future work
Relevant open questions should be investigated in further, detailed simulation studies. This refers especially to the area of sample size, both general sample size and different combinations regarding the number of clus- ters and included observations. Open questions in this area refer to the efficiency of the estimation in regards to the dependence of the data size, well a possible dependence (especially of the comparison models like hot) on different cluster sizes (e.g. underrepresented clusters). In our studies, although no balanced design was chosen, the cluster sizes are nevertheless in the same order of magnitude. The dependency on sample size also becomes relevant in the combination of a large number of clusters, with a in general small sample size. Here the fixed effects approach (here as hot) might suffer from a mismatch in parameters to be estimated and available observations. Another point that should be investigated in these studies is the convergence of the EM algorithm and to what extent this depends on different simulated scenarios.
Since the framework decreases estimation time, even when compared to the traditional models, it would be interesting to investigate if this advantage further increases with increasing sample size. Then the mixedML specification that intends to mimic LME (NNIa) can be an alternative for the psychometric community to estimate linear mixed effect models for big data sets. Further, the option of predicting the random for unobserved clusters as in Xiong et al. (2019) should be included in a general framework. The next steps will be to implement the generalizod mixed effects model allowing for link functions to non- normal target distributions (see Ngufor al. 2019). This will allow for different output activation functions, thus including classification problems.
14

---

Published in Transactions on Machine Learning Research (02/2023)
Acknowledgments
This work was supported by the German Research Foundation (EXC 2064, project number 390727645) Cluster Excellence "Machine Learning New Perspectives for Science".
References
Douglas M. Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting Lincar Mixod-Effects Models Using Ime4, Journal of Statistical Software, 67(1), 2015. ISSN 1548-7660. doi: 10.18637/js.v066..1
H.P. Brandsma and J.W Knuver. Effects of school and classroom characteristics on pupil progress language and arithmetic. International Journal of Educational Research, 13(7):777-788, 1 1989. ISSN 08830355. doi: 10.1016/000 (1883-0335(89)9000888
Chih Chung Chang and Chih Jen Lin. LIBSVM: A Library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3), 2011. ISSN 21576904. doi: 10.1145/1961 189.119199999999
Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences. Routledge, 5 2013. ISBN
9781134742707. doi: 10.4324/99788555 (0.4324/97902087755577
Marie. Davidian and David M. Giltinan. Nonlinear models for repeated measurement data.
Chapman Hall, 1995.
ISBN 9780412983412.
URL
com/
Eugene Demidenko. Mized Models. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., Hoboken, NJ, USA, 1 1987. ISBN 9781118651537. doi: 10. 10.1002/9788111865115 UtLpatt://doi..iie.
ccm/10.1002//971111888
M. Fokkema, N. Smits, A. Zeileis, T. Hothorn, and H. Kelderman. Detecting treatment-subgroupp interactions clustered data with generalized lincar mixed-effocts model trees. Behavior Research Methods, 50(5): 2016 2034, 10 2018. ISSN 1554-3528. doi:
Ahlem Hajjem, François Bellavance, and Denis Larocque. Mixed effects regression trees for clustered data. Statistics & Probability Letters, 81(4):451-459, 4 2011. ISSN 01677152. doi: 10.1016/..pl1.20001..0.. URL htttp:://inkinggiete ewsevier. /ttpa///iairrer.eeea////a//////a
Ahlem Hajjem, François Bellavance, and Denis Larocque. Mixed-effects random forest for clustered data. Journal of Statistical Computation and Simulation, 84(6):1313 1328, 6 2014. ISSN 0094-9655. doi: 100000000949655.200 2012.74153
URL
Ahlem Hajjem, Denis Laroeque, and François Bellavance. Generalized mixed effects regression trees. Statis- tics & Probability Letters, 126:114-118, 2017. ISSN 01677152. doi: 10. 20.1006//..p1.201.02....3 URL
https://linkinghuut elserierrerrrrre http///////.ee///////////////000
Anja W.M. Knuver and Hennie P. Brandsma. Cognitive and Affective Outcomes in School Effectiveness
Research. School Effectiveness and School Improvement, 4(3):189-204 1993. ISSN 0924-3453. doi: 10. 0011080000224445900000322
Nan M. Laird and James H. Ware. Handom-Effectt Models for Longitudinal Data. Biometrics, 38(4):963, 12 1982. ISSN 0006341X. doi: 10.2307/2529877 cressref
URL http:://www.toor.oorgg//////
jstor. /tt.oo...r.//taaae/e///997777010
Kung Yoe Liang and Scott L. Zeger. Longitudinal data analysis using generalizod linear models. Biometrika, 73(1):13-22 1986. ISSN 0006-3144. doi: 10.108//IO0ETT/T..... URL https://academicc.og
15

---

Published in Transactions on Machine Learning Research (02/2023)
Mary J. Lindstrom and Douglas M. Bates. Newton Raphson and EM Algorithms for Linear Mixec-Effects Models for Repeatedd-eessures Data. Journal of the Americun Stalistical Associalion, 83(404):1014-0222 12 1988. ISSN 0162-1453 doi:
19911119191188....... 1ttp:/11...000//1214455..5 198.. 10478693
URL http:://ww..aandfonlln com/
Francesca Mandel, Riddhi Pratim Ghosh, and Ian Barnett. Neural networks for clustered and longitudinal data using mixed effects models. Biometrics, 2 2022. ISSN 0006-341X. doi: 10.1111/biom. 13615. URL
https://onlinellbrayy
1111/biom. 13615
Steffen Nestler and Sarah Humberg. A Lasso and Regression Tree Mixed-Effect Model with Random Effeets for the Level, the Residual Variance, and the Autocorrelation. Psychometrüka, 8 2021. ISSN 0033-3123. doi: 10.1007/s11336-021-008000
URL https://link: httpr/geer..cm/0..11o001/11323-2
Che Ngufor, Holly Van Houten, Brian S. Caffo, Nilay D. Shah, and Rozalina G. McCoy. Mixed effect machine learning: A framework for predicting longitudinal change in hemoglobin Alc. Journal of Biomedical
Informatics, 89:56 67, 2019. ISSN 15320464. doi: 00.1016/1:bb.2018.0900 URL https://linking::: elsevier. aatavv/cccooo/erreeee/////////21
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Leorning Research, 12, 2011.
Stephen W Raudenbush and Anthony Bryk. Hierarchical linear models: applications and data analysis methods. SAGE Publications, 2nd edition, 2002.
Skipper Seabold and Josef Perktold. Statsmodels: econometric and statistical modeling with python. In 9th
Python in Science Conference, 2010.
Rebecca J.
Sela and Jeffrey S. Simonoff. RE-EM trees: data mining approach for longitudinal and clustered data. Machine Learning, 86(2):169 207, 2012. ISSN 0885-6125. doi: 10.10077/119994-111555555 URL
http:://iinkssppe
com/ 10. 1007 000994 011-5258-1
Tom A. Snijders and Roel J Bosker. Multilevel Analysis: An Introduction to Basic and Advanced Multilevll Modeling. Sage, London, 2nd edition, 2011. ISBN 978144625433..
Hulin Wu and Jin-Tine Zhang. Nonparametric Regression Methods for Longitudinal Data Analysis. John
Wiley Sons, Inc., Hoboken, NJ, USA, 2006. ISBN 9780470009673. doi: 10.1002/04470009955.
Yunyang Xiong, Hyunwoo J. Kim, and Vikas Singh. Mixed Effects Neural Networks (MeNets) With Appli- cations to Gaze Estimation. In 2019 IEEE/CVE Conference on Computer Vision and Pottern Recognition (CVPR), pp. 7735-7744. IEFE, 2019. ISBN 978-1-7281-3293- doi: 10.1109
Scott I. Zeger and Kung-Yee Liang. Longitudinal Data Analysis for Discrete and Continuous Outcomes. Biometrics, 42(1):121, 3 1986. ISSN 0006341X.
doi: 10.2307/253124
16

---

Published in Transactions on Machine Learning Research (02/2023)
Appendix
A
out-of-the-box implementation Algorithm B
OOB Algorithm for the implementation with standard mixed model methods (out-of-the-box)
A Agorithm: OOB implementation
t=0 Initialize a,(0) 0, Vj 1,,., J. 1(a)
Fized
and train ml (X) (() + (())
fxed
fixdd set E(U) = y D(a)
Alxdd
L+1 specify the LME as
C(t-1) = X,3 Zu
be trained with mixed model framework (out of the box) U(e)s Ee(e) and 02
Update v(()
fxoe
(ū(e)
train (f) (X) )- = Axed + Qee; and glxed
set C(e) = y fxed
Simulation details
To evaluate and compare the models, we conducted simulation study. We fix the number of clusters to 200, the number of observations per cluster ny to sampled value between 22 and 25, the number of level 1 predictors include random effects for all level
(attributes, features) p1 and the number of level predictors p2 predictors (p- p.).
We simulate the level feature vector for every observation i E [l...., N] as random draw from the pr-dimensional multivariate normal distribution (0, 1xx) with
diag 11s =1
and covariance values of 0.1 or 0.3 according to a random draw of the uniform distribution U0.1 (with threshold 0.5). Note that these chosen covariance values correspond to typical low and medium effect sizes that usually found in the social and behavioral sciences [Cohen 2013).
We repeat this process for the level 2 feature vectors with the according pz dimensional covariance matrix Ezs. Note that level 2 features are equal for observations within the same cluster.
For the covariance matrix E, of the p. + random effects (p, slopes/ glopes/weightts weights plus random intercept/biat bias), Oum.t = 0 for n # m. For the then diagonal covariance matrix of the random effects, we assume the identity matrix (setting o²_ 1, Vm [0, prl...,,]]]. By setting the variance of the random effects one, we identify the latent variable vector. To simulate the cluster specific realizations of the latent variables we draw samples ~ NO, E,)
for j I,.., and define (ui,..., "
For the residuals we assume Σ. 0.1 with a, = 3.5, and sample e N'O0 E.) (e RM).
We set the intercept/ 3 and all other parameters and obtaín 81 E R (level perameters/weighhe 32 E RR (level 2 parameters/wwighhe variances of y of around 0.77 (linear) and 882 (uon-linear).
With the chosen values, together with de, we achieve an overall explained
Calculate the target
Putting together, obtain the linear case
Vlo 8o + 1-3 Zu
For the non-linear case, we include the quadratic terms of the first five level features and define
X, = (,,
and B, R sampled as the other coefficients. This gives
Valin + XB8 X,9, Zu
17

---

Published in Transactions on Machine Learning Research (02/2023)
C Hyperparameter tuning
Before being used the mixedMT. framework; the hyperparameters of pure machine learning models are tuned for the respective data sets. We performed an exhaustive grid search with 3-fold cross-validation. For the linear simulation data (NNim), we only tune the learning rate. In the non- non-linear case (NNBa); we also tune the number of hidden units in the two layers. For SVR we use the sklearn (Pedregosa et al. 2011) implementation which is based libsvm Chang & Lin (2011). We tune the kernel (linear or radial basis function (rbf)), the regularization parameter C (inverse proportional typical regularization parameters range), and the rbf-kernel parameter ganiaa.
NN
NNa hot
NNulin
NNaiin hot SVR
SVR hot
Simulations
learning rate
0.01
0.01
0.01
0.01
hidden layers
2
2
units layer 1 units layer 2
4
4
kernel (linear/rbf)
rbf
rbf
C gamma
38
0.001
0.001
Emprirical data
learning rate
0.01
0.01
kernel (linear/rbf)
rbf
rbf
C
gamma
0.001
0.001
18


Reviews:
[{"review_id": "Oplf1JylXQ", "reviewer": "Reviewer_JZWV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the work is in the computational efficiency of the algorithm, which in addition to the flexibility of modeling frameworks that can be used, is making it attractive tool for modeling the data where samples tend to form the clusters. And such applications are numerous, so the potential for impact is there.\n\nHowever, one of the weaknesses is that empirical evaluation doesn't seem compelling. Some of the evaluation setup choices are not justified, or at least not clearly presented. Claiming that model (hyperparameters) are not tuned because 'the focus is to compare approaches regarding the consideration of random effects' does not sound compelling. It might cast the doubt that mixed models are just capitalizing against not properly fitted baseline. Similarly, using the Neural Network to 'simulate' linear model is peculiar, and potentially affecting the computation times, as implementation of simpler model like Linear Regression is expected to be at least a bit more efficient. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper a new algorithm for fitting the mixed effect regression model is described and evaluated. Motivation for the approach is enabling flexibility of using arbitrary function approximator, like for example Neural Networks, to model the fixed effect patterns. The algorithm is based on Expectation-Maximization procedure, where parameters of random effects are estimated in E-step, and fixed effect part is optimized in M-step. Mixed effect regression problem is described and formally introduced, along with the derivations, and modeling assumptions on the form of random effects which allows for efficient computation of the solution, in particular uncorrelated multivariate normal distribution. The approach is empirically evaluated on linear and nonlinear synthetic examples, as well as in the application on language test grades of thousands of students, coming from few hundred different schools. The main takeaways appears to be superior computational efficiency with comparable results, agains the out of the box approach.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some suggestions, which I believe would help to convey the messages more clearly:\n- Since 50 fold crossvalidation procedure was performed, I guess it wouldn't be much more effort to also perform the hyperparameter tuning of baseline ML model. \n- Even though Mean Squared Error is a valid measure, the Square part is making differences more pronounced. So maybe is more fair reporting Root Mean Squared Error instead.\n- Transfer learning was just mentioned in the pass, without giving more details or references on how it would be achieved, or what would be the pros/cons, etc. If its really relevant for the paper, its useful to elaborate on it more.\n- Presentation is clear for the most part, and even though in general paper is reading easy, there are some sentences which require effort to decipher. For example sentence \"Nested data induce similarities between observations when they belong to a cluster.\" sounds a bit circular because its stating that observations within a group are similar because they belong to the group. It would be more informative if it explains how the similarity is induced, or what the implications of this similarity are. \n- Nonlinear mixed model formulas in section 2.2 are missing and equation numbers.\n\nAnd few questions:\n- Does first equation in section 2.2 implies that fixed effect mapping(s) $f_j$ are (potentially) different for every $j$?\n- Why covariance matrix of random effects $\\Sigma$ doesn't have subscript $j$? Isn't it possible that covariance differs among the groups.\n \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YzXwTD79Wk", "reviewer": "Reviewer_4AE4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors nicely describe the problem in the introduction. I wish they would expand further and use a running example to help explain the various components of their framework.\n- The authors perform extensive experiments that compare against natural alternatives (e.g., one-hot encoded fixed effects).\n\nWeaknesses (see more discussion below):\n- While I understand that novelty is not a consideration for this review, and the authors are honest in stating they \"do not aim to introduce a new model that competes with the mentioned research,\" it is not clear to me what generalizable insights the authors are claiming to contribute. The authors state they would like to \"point out existing results and show that hierarchical data structures... can also be considered as an addition to one's own analyses\" but unfortunately I'm not sure what this means. It would be helpful for the authors to clarify what they think a reader can takeaway from their study after reading the paper given that they are replicating previous methods (this also ties into the next weakness)\n- I think there are a number of ways in which the clarity of the presentation could be improved. \n  - Most importantly, some key elements of the framework are never discussed. As far as I could tell, there is no discussion of the difference between ML and REML other than the differing formulas provided in Equation 4.1. Further, I don't believe the acronym REML is defined.\n  - While the experiments are thorough, I found the presentation of the results (particularly the tables) hard to parse. I have some suggestions for how the authors could improve this so that it is easier for readers to identify takeaways from experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the authors present a framework for learning mixed-effects models built around machine learning algorithms used to model the fixed-effects. It largely builds on prior work which developed expectation-maximization (EM) procedures for fitting the mixed effects models. Most of these works have used tree-based models as the core machine learning algorithm, while in the present work the authors allow for arbitrary machine learning algorithms to be used (including SVMs and neural networks). The authors then perform simulated experiments where they compare a variety of baselines (including models which ignore cluster structure in the data, and models which use cluster indicator-based fixed effects instead of random effects), as well as to ablated components of mixed effects models produced by their framework. These results show that, when well-specified, the mixed effects models are significantly better at predicting on new data from a previously observed, known cluster (i.e., cluster was in the training data), and that all approaches are comparable when data comes from unseen clusters. They then apply their framework to school language assessment dataset and demonstrate that they recover very similar coefficients when compared to those recovered by classical methods", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it would be most helpful for the authors to add more discussion surrounding the details of their framework. This would help clarify what makes the framework interesting (weakness 1) and is also important so that readers can fully understand (weakness 2.1) the framework and navigate the choice between REML and ML.\n\nCurrently, the authors do a good job of describing mixed effects models in Section 3. However, when describing the estimation framework in Section 4, I think the manuscript would benefit strongly from more discussion. Most importantly, the authors should include a discussion of what REML and ML are (including what REML stands for), and what makes them different. Methodologically, how do they differ (e.g., what are the consequences of the differences in the two projections in Section 4.1)? Are there situations in which we might expect one to be preferable to the other?\n\nIn a related vein, are there any methodological consequences of using non-tree-based machine learning models? Or is the EM estimation procedure exactly the same as with trees? As an example of useful information that the authors provide, when talking about benefits of ML and REML over the OOB version, the authors state that \"the machine learning model can be learned from scratch, but the process can also be started with pre-trained models, which leads to early stops according to convergence criteria.\" This is helpful to know, and I was wondering if there were any convergence results that the authors could cite here (they would likely be model class specific though)? I also think the authors should reiterate why ML/REML have lower runtime than OOB when discussing the results in the experiments.\n\nI also think the paper would benefit from a running example so that different model components can be tied to tangible concepts. This would be especially helpful for describing the difference between the \"within\" and \"out\" settings in a way that will be easy to understand for practitioners.\n\nIn my view answering these types of questions is important for the paper to deliver a complete account of the proposed method so that users can better navigate the framework.\n\nOn a minor note, I have suggestions for improving the presentation of the experimental results. Currently, Table 3 in particular is a large data dump, and it is hard to pull out important results from it. To help declutter the table, I think that performance on training data is less interesting (unless the authors can justify within the text otherwise) so it could be omitted or placed in an appendix. Within the table there is  nothing visual to help make items in the table stand out (e.g., bolding superior results). It might also be more effective to have visualizations for some of the data instead of using a table, such as a bar plot of the fixed effects estimates with error bars for example.\n\nAdditionally, I think a reader should be able to compare the performance of models (the \"prediction\" section of Table 3) on the test data by looking at a single row. So for the within [test] and out [test] rows, all methods should have their own column. At present, methods (such as pure ml and hot) are stratified across rows and columns. Instead, have the pure ml methods and the hot methods be their own columns. To draw out the significance of the fact that some of these models cannot make more accurate \"within\" predictions, the performance of all models on the within [test] data should be listed instead of blanked out. To make sure I understand correctly, couldn't I use a pure \"ml\" model to predict on a within-sample? The prediction won't differ from the value the model would predict if the cluster were not known (i.e., the within and out performances will the same, correct?), but the performance comparison with other methods that can account for the cluster is useful to see.\n\nOther minor notes:\n\n- It is confusing for \"ml\" and \"ML\" to both be used as acronyms that refer to different things\n- Regarding sample sizes for the simulated experiments: I expect a priori that one-hot encoding will suffer most when sample sizes for some groups are very small. And I also expect pooling all the data (i.e., the pure ml model) will perform poorly (stratified, e.g., by worst performing cluster) when, say, one cluster is over represented in the training data. If not including simulated cases to demonstrate this, could the authors add more explicit discussion of when one-hot encoding (or more generally considering the random effects as fixed effects) might fail?\n- The authors make many loose connections to transfer learning, but it isn't obvious to me. I think if the authors are going to lean into this connection, they should explain it explicitly at some point in Section 2.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4lFfB9304D", "reviewer": "Reviewer_uHhv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed framework has the flexibility to be easily modified to use different ML methods. \n- Simulation and real data results illustrate the promise of the framework.\n- The writing style is good and balanced between using active and passive voice.\n\n**Weaknesses**\n\n- Properties and limitations of the framework not discussed.\n- Some issues related to writing format and notation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a general and flexible mixed-effects approach that incorporates random effects into supervised regression machine learning approaches to handle the dependency issue in clustered data. Herein, the framework has no restriction on the choice of the machine learning method. In addition, the framework shows promising results with decreased estimation time and improved prediction compared to other existing methods. The authors test their framework performance on simulated and real data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes**\n- Acronyms/abbreviations: I found it confusing to switch between using the abbreviation and then using the full term and re-defining the abbreviation. When abbreviating a term, always spell out the full term the first time it appears/is used, followed by the abbreviation in parentheses ( e.g. Maximum Likelihood Estimation (MLE) ). Then any time you need to mention/use Maximum Likelihood Estimation, use only MLE.\n\n-  Define/state what dose i.i.d and mixedML stand for.\n\n-  Add the unit of time: There is a row for the time in many tables but never mention if it is measure in hours, minutes or seconds. E.g. use  time (s) if it is measured in seconds instead of time.\n\n- Fix/Add space between the table and the caption/title. In some tables, there is a good space (e.g. Table 1), some not ( e.g. Table 2).\n\n-  Use proper punctuation after an equation (i.e. add a comma or period): \n\n    - If you continue writing after an equation (i.e. add another equation or where ...), add a comma. E.g. after eq(8), add a comma and after eq(9), add a comma. \n\n    - Add a period if the sentence ends in a displayed equation. E.g. add a period at the end of the eq(10).\n\n-  Make sure to use text environment when using a word (non-math symbol) inside math mode in the subscript and/or superscript. E.g  use this format  $y^{\\text{fixed}}$ and $\\text{NN}_{\\text{lin}}^{\\text{OBB}}$,   \n \n      instead of $y^{fixed}$ and $\\text{NN}_{lin}^{OBB}$\n\n-  Whenever mentioning a function or a programming language name, use {\\sf ..... }.  Eg. use this format  $\\sf python$,  and $\\sf{ lmr()}$ instead of  python, and $ lmr()$.\n\n-  Be consistent if you choose to hyphenate/not to hyphenate words (e.g. either use mixed-effect or mixed effect).\n\n- If possible/applicable: run more simulations (see questions for details).\n\n- If possible/applicable: describe how you split the data (see questions for details).\n\n**Questions**\n\n- If there are no computation issues (time or convergence), why the simulation seems run only one time with one sample size? It might be useful to repeat the simulation like 5 times or more with different sample sizes. Then report the average values of parameter estimation, time, ..., etc. to see if there are any properties or limitations.\n- Is the data split into train/test? If yes, how did you split it?\n\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors and reviewers engaged in a lengthy discussion on the merits of the paper, claims, evidence, and presentation. The authors made several revisions to the paper during the review process as a result of those conversations and highlighted the changed that they made to the manuscript in blue text. After this discussion the muscript is much improved and the reviewers recommended that the manuscript be accepted.
Decision: Accept (poster)