Using Random Effects to Account for High-Cardinality Categorical Features and Repeated Measures in Deep Neural Networks
Authors: Giora Simchoni
Saharon Rosset
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep neural networks
high-cardinality
categorical features
random effects
mixed effects
likelihood
regression

Abstract: High-cardinality categorical features are a major challenge for machine learning methods in general and for deep learning in particular. Existing solutions such as one-hot encoding and entity embeddings can be hard to scale when the cardinality is very high, require much space, are hard to interpret or may overfit the data. A special scenario of interest is that of repeated measures, where the categorical feature is the identity of the individual or object, and each object is measured several times, possibly under different conditions (values of the other features). We propose accounting for high-cardinality categorical features as random effects variables in a regression setting, and consequently adopt the corresponding negative log likelihood loss from the linear mixed models (LMM) statistical literature and integrate it in a deep learning framework. We test our model which we call LMMNN on simulated as well as real datasets with a single categorical feature with high cardinality, using various baseline neural networks architectures such as convolutional networks and LSTM, and various applications in e-commerce, healthcare and computer vision. Our results show that treating high-cardinality categorical features as random effects leads to a significant improvement in prediction performance compared to state of the art alternatives. Potential extensions such as accounting for multiple categorical features and classification settings are discussed. Our code and simulations are available at https://github.com/gsimchoni/lmmnn.

Using Random Effects to Account for High-Cardinalle Categorical Features and Repeated Measures in Deep Neural Networks
Giora Simchoni
Saharon Rosset Department of Statisties
Department of Statistics
Tel Aviv University
Tel Aviv University
Tel Aviv, Israel, 69978 gsimchoni@tauex. tau ac i1
Tel Aviv, Israel, 69978 sahar ontauex tau, ac i1
Abstract
High-cardinality categorical features are a major challenge for machine learning methods in general and for deep learning in particular. Existing solutions such as one-hot encoding and entity embeddings can hard scale when the cardinality is very high, require much space, hard to interpret may overfit the data. A special scenario of interest is that repeated measures, where the categorical feature is the identity of the individual object, and each object is measured several times, possibly under different conditions (values of the other features). We propose accounting for high-cardinality categorical features random effects variables in a regression setting, and consequently adopt the corresponding negative log likelihood loss from the linear mixed models (LMM) statistical literature and in- tegrate it in a deep learning framework. We test our model which we call LMMNN simulated well real datasets with single categorical feature with high cardinality, using various baseline neural networks architectures such as convolu- tional networks and LSTM, and various applications e-commerce, healthcare and computer vision. Our results show that treating high-cardinality categorical features random effects leads to a significant improvement in prediction performance compared to state of the art alternatives. Potential extensions such as accounting for multiple categorical features and classification settings discussed. Our code and simulations are available at htttp:::///// /ttps://github http://gttthut...eeg com/ www.uuww.oo.ggiiicoo..........o. wwwgggssiimnnnneeeeemme 1/mmmnn
1 Introduction
In recent years deep neural networks (DNNs) have served as the method of choice for learning complex non-linear relations between features of large datasets and for hard prediction tasks. Yet, despite their advanced machinery such as stochastic gradient descent (SGD) and convolutional layers, DNNs (as other machine learning tools) do not readily lend themselves to handling categorical features of high cardinality. Such features are often seen in modeling tasks, a good recent example would be Lin et al. [16] who used electronic medical records (EMR) of hospital patients to predict hospital readmission, and had to deal with the patient's diagnosis as a categorical feature with thousands of levels. In recent review Hancock and Khoshgoftaar [13) surveyed the approaches taken by DNN practitioners for turning the levels of categorical features into numeric form, which the most used approaches also appear as the go-to solution in one of the most cited handbooks of machine learning by Géron 9.. See Section for more detailed discussion of these approaches.
one-hot encoding (OHE) and entity embeddings. OHE and embeddings
different approach for handling high-cardinality categorical features that of linear mixed models (LMM), see e.g. McCulloch et al. [18. In the LMM framework statisticians differentiate between
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

fixed effects (FE) which are regular parameters to be estimated from the data and random effects (RE), which are parameters that are not directly estimated, but rather treated as random variables, and their distribution is estimated instead. This mechanism is often used to deal with high cardinality categorical variables or "cluster" indicators. In a typical setting a statistician would be interested in the effect few designed experimental conditions have on a dependent variable, e.g. the effect of treatment A versus treatment B on a patient's blood pressure. The treatment effect would be considered fixed effect, while the effect of the city in which a subject resides- - categorical feature with possibly thousands of levels - would be considered a RE variable, often having normal distribution N (0. of) with of, being an unknown "variance component" to be estimated (see Section 2 for a formal description). Another common example of a RE variable is the subject of the experiment herself, in repeated measures setting, where subject is treated to both treatments A and B. In such a setting a statistician would usually deem a subject's effect as random, i.e. coming from a possibly infinite population of subjects, and use proper LMM machinery handle such variable.
In this paper we are interested in examining modern big-data predictive modeling domains, where DNNs would be natural choice of modeling approach, but where high dimensional categorical variables or repeated measures are an important aspect of prediction. Examples we pursue might include: predicting the gaze direction of person's 's eyes from repeated facial images of the same individuals [24]. detection of diabetic retinopathy from retinal fundus photographs where each patient is photographed for both left and right eyes (1I], prediction of a person's 's fluid intelligence from dozens of features including a person's ' job which can take hundreds of levels [14] and predicting the price of an Airbnb rental with host being one of tens of thousands [15]
To address such challenges we seek a simple extension to DNN which allows to treat high-cardinality categorical features as RE and ultimately improve the network's ' prediction performance in a regression setting. We do this by generalizing the MMMM-framworr negative log likelihood (NLL) function and treating it as natural loss function for the entire network, minimizing both the "fixed" and "random" " parts, and improving the bottom line test mean squared error (MSE). We demonstrate the superiority of our method over using existing methods such as OHE, entity embeddings and standard LMM via R's Ime4 package (Bates et al. (2). as well as previous attempts at integrating RE in DNN (MeNets, Xiong et al. (22)) on both simulated and real datasets from various applications such as healthcare and computer vision. Finally we discuss future directions, such as using RE in DNN in a classification setting and with more complex covariance structures such kriging.
2
A Brief Tour of Linear Mixed Models
The canonical linear model assumes:
y == + e, where X is the n x p model matrix where the ith row is x4, 3 E TRP is a vector of model parameters to be estimated and E I is normal i.d noise, i.e. ~ N (0. I?)), where of is variance parameter and I is the n x n identity matrix.
(1)
A linear mixed model treats the 3 parameters fixed effects, and allows additional data to enter the model in the form of a n x q matrix Z and a vector of random effects b E R.:
XB + Zb +
(2)
Here b are random variables, typically assumed to have multivariate normal distribution N (0, D) where D is a x g positive semi-definite matrix holding usually unknown variance components to be estimated, let these be 0, so D could be written as D(0). The structure of this covariance matrix is up to the researcher but there are typically simplified structures used. It is further assumed that there no dependence between the normal noise and the random effects, i.c. cov (c, 0.
Let us write the vector of all variance components as to = [o?- 0). How to estimate 8. The statistical approach of choice is maximum likelihood estimation (MLE). We write the marginal distribution of as N (X3, V) where V(e) = ZD( + Io and from here it is straightforward to see that the log likelihood is:
1(3, (vv = - (v (w-
X3)
log [V() 2 log 2n
(3)
The MLEs for 3, & maximize 1(3, kly) equivalently minimize the negative log likelihood (NLL) in (3). Typically maximizing the likelihood is difficult with given constraints on & (which are non- negative variance components), and to get less biased estimates for v alternatives methods such as

---

restricted maximum likelihood estimation (REML) are preferred. For full details see e.g. McCulloch al. [8]
Of particular interest is the random intercepts model, in which a single categorical random variable, such "subject" "city", is assumed. If this so-called "clustering" variable has q levels, then Z is an indicator matrix holding at entry [é, if observation belongs to cluster and zero otherwise. In this case it is often assumed that the random effects for this variable are independent of one another, hence: D = I. In this setting ( Zz + o² is block diagonal matrix and its inversion can be avoided in B). More on that in Section 3
How to predict dic in a machine learning scenario, where (X, Z, y) are typically split into training and testing sets (Xer, Zer, Yor) and (Xte Xte, Yte)? In general one would use y conditional distribution:
ûtc B +zab,
where 3 = (X/,V-1Xxr))
Xxr) V-1 yer are the estimated fixed effects once the estimated variance components db are input into V. and:
i D V-1
ytr XtrB)
(5)
is the so called best linear unbiased predictor (BLUP), as are not actually parameters to be estimated, but random variables be predicted.
Sometimes, however, (4) not possible such as in the case of the random intercepts model, where Zte holds levels of the random variable unseen before, e.g. new subject. In this case it is customary to y's marginal distribution and predict ôte be Xte i.c without the random part. Finally, still under the random intercepts model, it can be shown that for a given level j and corresponding random effect by. the computation can be simplified to avoid the inversion of V giving:
nooz 02 + +n (Dtra XirB,),
6,
(6)
where (o2, ag) are the estimated variance components, ny is the no. of observations in cluster j and Sony and &j are the true and predicted mean values of in cluster respectively.
3 LMMNN: Our Proposed Approach
Let us change (2
into:
y - + (Z)b+
where / and y are non-trivial functions which DNNs are suitable for, e.g. non-linear and involving interactions. We propose to use NLL natural loss function, where X is replaced by the DNN outputs f(X):
1 Vl. -
NLL(J. g. + - (v- (
log V(g, +b) + log 2n,,
(8)
where (gg vb) of (()))()) + Note that f and are kept general as possible, to allow any acceptable DNN architecture, including convolutional and recurrent neural networks. See Figure f a schematic description of our approach which we call LMMNN, in the case f and g are approximated with simple multilayer perceptron.
We propose using existing DNN machinery, mainly SGD, to fit the model and its variance components. To be more specific, we implemented a custom NLL loss layer where at each epoch (8) is calculated on a small batch typically of size 30-50 observations and auto-differentiation is handled by Keras 4 An alternative could be using explicit formulas for the variance components & derivatives in case V ($) is simple:
ONLL
0V Zi
fx)) (- tr (v (v-1 - ØV
f(x)) V -
(y-
- v
(9)
dbp
2
0V where
is (Z) AD(0) gZ)) for the 0 variance components and / for o².
og

---

The main challenge in implementing mini-batch SGD on (8) is the inverse and determinant of Vgg. 16) which may not decompose into sums on batches. An important exception is the main setting of interest in our paper in the random intercepts model, when (() 7 and Vgg, 1b) is a block-diagonal matrix. If all blocks are identical, such that each level of the categorical feature has m observations (X3.8) Un) where
1.... 9. we can write ((+) diag(V1. ..... Vg) where each V3 block is of size m x m and ( = Jm + =o?Im where Jm is m x m all 1s matrix. This means we can write the inverse in (8) as block diagonal as well, ((b) 1 = diag(V; Vi= :..., Vg-1), and the log determinant in 8 as a sum of log determinants: log (v+) - Σ j= log Vil- The NLL in (8) can now be written as a sum: NL <byy) = E (وg f (X,))VV, (uy - f (X))) + 2 log [V,| + 2 log 2m. Most importantly, the gradient in y can be decomposed into a sum of gradients over mini-batches of size
ONLL O
? Σ
0V, Du V (y) ff
1 tr 2
2V, Oeb
}
2 (0) ( (Xx)) VJ
(10)
If all blocks are not identical but small (the typical case), we can still make sure that mini-batches consist of each level j's ny observations. For more general cases the mini-batch approach (where inverse and determinant are calculated on the mini-batch) still seems to work well in practice, some related theory can be found in Chen et al. 3 see Section 6 for further discussion more LMM scenarios.
For prediction, we accommodate as well:
The (Xte) 9 (Zee) b,
(11)
where f and 9 are the outputs of the DNNs used to approximate of and g. and are the predicted random effects with the modified BLUP: b D(V) (Z+r) V(g. V() 31 ytr (X,r))
Note further that in the likely case where is simply the identity matrix and we are interested in single clustering variable in the random intercepts model, we can also avoid inversion of V in the loss function as described in [18. as well as in computing the modified BLUP:
b,
(Jerss - f (Xxr))
(12)
4 Related Work
4.1 One-Hot Encoding and and Entity Embeddings
As Hancock and Khoshgoftaar 13 state, OHE is often a "first step". and the standard method for handling categorical features. If of length n is a categorical feature of q distinct levels, OHE would add binary features 21, ....... zg. one for each level, with 2μ if observation i has level / in feature v, and 0 otherwise. The main advantages of the OHE approach are that it deterministic, fast and as q can be in the tens hundreds of
highly explainable. On the downside it is hard to scale thousands as in the examples below - burden to any algorithm even when accounting for sparse representation of features. A second disadvantage of OHE features is how little information each feature carrics. Other methods for converting categorical features into numeric can reach a much more meaningful representation.
One such method is entity embeddings (See e.g. Guo and Berkhahn 12]). Entity embeddings start by feeding neural network with feature v after it has been one-hot encoded into q-length vector, and using the network's inherent loss function and back propagation algorithm to learn a oow-dimensional representation of length d (typically d << q) to each of u 's levels. This results in lookup table E of dimensions y x which serves the predictive modeling task and can also later re-used via transfer learning where the representation learned for one task can serve for other tasks, see e.g. Do and Ng [6]. It is also our experience that the entity embeddings approach performs well. Apart from that, efficient implementations make sure it scales well when is large, unlike OHE the resulting

---

Z
X
NLLAF.49)
((X)
Figure Schematic description of LMMNN using a simple deep MLP for fitting f and g. and combining outputs with the NLL loss layer.
representations may carry a lot of information, and the option to perform transfer learning on these is appealing. On the other hand, entity embeddings may consume much space (the E' lookup table), they may need to be learned for each new task and the resulting representations are usually hard to interpret.
4.2 Previous Attempts at Combining Random Effects in DNN
There have been few previous attempts at combining random effects in DNN. The efforts aware of are MeNets by Xiong et [22] and DeepGLMM by Tran et al. (21).
MeNets Inspired by non-linear mixed effects model y (X + Zb) where is some non-linear function, Xiong et al. [22) and Xiong et al. [23) propose the following model to learn fixed effects 3 and random effects b:
B (X)bb+
= B
(13)
There two main differences between 13 and our 77: First, the Z RE features matrix is missing, rather the RE features are learned as the output of the neural network to standard input X, f(X).
Second, (13) uses single neural network to learn both the fixed and random features, where ( is more general and allows different functions f.g for fixed and random features respectively. In particular, (7) allows for g to be the identity function which may certainly be appropriate for high cardinality categorical features and repeated measures and gives intuitive meaning to b.
In order learn A and b the authors variational expectation maximization (V-EM) combined with SGD: E-Step which 3, are updated while minimizing the standard squared loss with DNN, followed by M-Step where the variance components & are updated so to maximize NLL loss similar to Hence, an additional critique of MeNets could be that they essentially use two loss functions for a single task, whereas we use NLL as a single loss function be maximized in the SGD framework, making our implementation simpler and easier to track. In addition, in order to update by each E-step, there is the need to invert V, which is the jth RE covariance matrix of size n, n... This can be hard to scale if one of the categorical feature levels has large n.
Seee Section and the UKB dataset, when one of the levels of the categorical feature "job" has over 29,000 observations being unemployed and MeNets inverts 29,000 29,000 matrix each epoch.

---

Finally we note that MeNets were demonstrated in limited context: the authors put special emphasis on computer vision applications, mainly that of gaze estimation, as a consequence the DNN architec- ture used by the authors is solely a convolutional network, whereas we demonstrate our results on broad class of DNN architectures. Moreover, the categorical clustering feature MeNets used was the subject whose gaze was recorded, with typically only 10-50 subjects (and many repeated measures for each subject). Below we demonstrate McNets in a wider context and compare it to LMMNN, illustrating the superiority of our approach in terms of both predictive performance and computation.
DeePGLMM A different approach taken by Tran et al. 21 is based on very specific mixed effects model, in which each subject is repeatedly measured at different times / for some response ya which can be continuous as well as discrete, as modeled by generalized linear models (GLM). In such a model it makes sense to not only have a random intercept for each subject but also a random slope a, The authors write:
glma) f(z (2)
w w. B0)) + (8(2) ( + +,)'= (2)
(14)
where are the network parameters, z11) and (x are the features expected to have nonlinear and linear effects,, respectively, ((a) and [8)) are the fixed nonlinear and linear effects respectively, and
a = E(y[z), via some link function g. e.g. the logit function for binary y. A Bayesian approach based on variational approximation is used to maximize the likelihood for (1).
We note that (14 is similar to our criterion in 7.. when g is the identity function and y is linear in Z, the RE features matrix. We further acknowledge that our approach is only currently applicable for a continuous y (though see future directions in Section 6 However, the variational approxima- tion algorithm proposed in DeepGI.MM, which combines numerous elements such importance sampling, factor covariance, variable selection as well as choice of priors makes it challenging to implement, let alone use as a "plug-in" for different DNN architectures as we strive to do. Due to the high conceptual and computational complexity of DeepGLMM we were not able to implement and test it below like MeNets- it cannot scale to problems of the size we consider and to diverse architectures.
Results
5.1
Simulated Data
We start with a simple random intercepts model simulation, in which n = 100, 000 and there is a single categorical random variable with q levels (q E {100, 1000, 10000)), such that Z is n x g binary matrix, and b is a q-length vector of i.i.d random effects, sampled from N (0,oī) distribution (oī E {0.1, 1, 10)). The q levels not evenly distributed among the N observations, rather use a multinomial distribution sampling. for more details see our code and description in additional material. There are 10 fixed features in X non non-linearly related to y:
= +
+ X10) +...+ cos(X1 + ..... + X10) + 2 X1 X₂ + Zb+c
(15)
Notice in this case g is assumed be the identity function. We sample X features from U 1- distribution, from N (0, distribution where keep of always. We perform 5 iterations for each oz) combination combinations in total), in which it into training (80% and testing (20%), train our models to predict jtv and compare the bottom- line MSEs in predicting ytu. We compare our model's MSE to those of R's 1me4 package results (i.e. standard LMM), MeNets, OHE, entity embeddings and ignoring the categorical feature in Z altogether. We use the same DNN architecture for all neural networks, that is hidden layers with 100, 50, 25, 12 neurons, ReLU activation and a Dropout of 25% in each, and final output layer with single neuron. The loss we use squared error loss (MSE) for OHE, embeddings and ignoring the RE, and NLL for LMMNN and MeNets (as mentioned above, MeNets uses squared loss for estimating fixed effects and NLL for variance components only). For both losses we use a batch size of 30 and an early stopping rule where training is stopped if no improvement in validation loss is seen within 10 epochs. We initialize both o? of to be 1.0 where appropriate: R's 1me4 and LMMNN, and compare the resulting final estimates for these two methods. All runs are made on Nvidia Quadro P620 GPU on a Windows machine, with Keras 4 and Tensorflow ① with our own NL. loss layer for LMMNN, available on Github.
sample the data, randomly split

---

Table 1: Simulated model with g(Z) 2, mean test MSEs and standard enrors in parentheses. Bold
results are non-in non-inferior to the best result in a paired t-test. Hence, LMMNN is significantly better than all competitors in all scenarios.
Ignore
OHE
Embeddings
Ime4
MeNets
LMMNN
0.1
10a
1.25 (.012)
1.20 (.010)
1.18 (.006)
2.92 (.017)
1.15 (.013)
1.14 (.010)
10a
1.23 (.009)
1.31 (.008)
1.24 (.004)
2.96 (.022)
1.40 (.065)
1.14 (.009)
104
1.22 (.004)
1.54 (.008)
1.56 (.007)
2.97 (.014)
1.51 (.133)
1.17 (.010)
10a
2.17 (.041)
1.23 (.008)
1.21 (.010)
2.93 (.013)
1.22 (.022)
1.09 (.010)
10a
2.16 (.015)
1.39 (.015)
1.32 (.014)
2.94 (.013)
1,42 (.091)
1.14 (.006)
10+
2.14 (.013)
1.68 (.013)
1.68 (.013)
3.17 (.021)
1.66 (.056)
1.27 (.014)
10
10a
10.45 (.38)
1.56 (.044)
1.57 (.039)
2.92 (.012)
1.86 (.156)
1.10 (.013)
103
11.37 (.11)
1.75 (.022)
1.72 (.041)
2.95 (.024)
2.19 (.143)
1.12 (.015)
10
11.31 (.04)
2.34 (.027)
2.20 (.033)
3.37 (.020)
3.29 (.423)
1.32 (.007)
Table 2: Simulated model, estimated variance components average
g(Z)
g ZW
Ime4
LMMNN
Ime4
LMMNN
06
62
63
62
07
62
ob
0.1
10a
2.92
0.09
1.12
0.10
2.92
0.49
1.09
0.15
103
2.91
0.10
1.16
0.10
2.91
3.52
0.92
0.16
104
2.90
0.10
1.17
0.17
2.91
33.8
0.19
0.16
1
10a
2.90
1.01
1.12
1.00
2.91
2.44
1.07
0.41
103
2.90
0.98
1.15
1.00
2.90
32.0
0.84
0.43
10
2.90
0.99
1.26
1.02
2.92
336.6
0.19
0.35
10
10a
2.90
10.13
1.04
9.24
2.89
32.9
1.06
2.21
103
2.91
10.02
1.12
10.01
2.89
337.8
0.75
1.30
10
2.91
10.01
1.34
9.72
2.90
3305.6
0.22
1.98
Table summarizes the test MSE results and Table [ (left) summarizes the estimated variance components results. As be LMMNN reaches the smallest test MSE average and with a considerable gap from the other methods, when standard errors are taken into account. This is particularly true when RE variance of and cardinality g are high. As for the estimated variance components o2 of. LMMNN reaches good estimation for both, while R's Ime4 reaches poor estimation for of without adding appropriate non-linear and interaction terms, resulting in worse prediction performance. We also plot additional material Figure the predicted random effects i vs. the true random effects b for different gs, the category size distribution and also ÿtc vs. yte- Finally Table 1 in additional material summarizes mean runtime and number of epochs.
more challenging scenario when g is some linear transformation Waxd of Z into a lower dimension d:
(16) where we use d 0.1 q and sample W from (-1,11) distribution. b is now a d-length vector of random effects, sampled from N (0, of) distribution (oī (0.1, 10)). We use Keras Embedding layer on Z to learn W, and notice now we cannot avoid inversion of V when calculating the loss in ( or in predicting 5 in (5). V is a N x N matrix, so it is perfectly reasonable to invert it for a batch size of 30. but for predicting i on the entire training set at the end of training this means creating and inverting matrix dimensions 80,000 80,000. practice we sample 10,000 observations from the training set for estimating V/ and this seems to work well, summarized by Table which shows the test MSE scores for all methods. Our method performs the best especially at high cardinality & where the reduction in MSE can go over 80%. We do note that LMMNN struggles
(X1 + +... + X1o) cos(X1 + .... + + X, ZWb+c.
7

---

Table 3:
Simulated model with g(Z) = ZW, mean test MSEs and standard errors in parentheses.
Bold results are non-inferior to the best result in a paired t-test.
of
Ignore
OHE
Embeddings
Ime4
MeNets
LMMNN
0.1
10a
1.42 (.039)
1.22 (.018)
1.19 (.024)
2.91 (.021)
1.25 (.084)
1.13 (.013)
103
4.92 (.345)
1.49 (.033)
1.43 (.035)
2.95 (.020)
1.44 (.061)
1.16 (.013)
104
35.1 (.456)
3.25 (.086)
3.39 (.116)
3.42 (.036)
7.35 (1.9)
1.59 (.023)
1
10a
4.13 (.626)
1.32 (.025)
1.31 (.033)
2.88 (.023)
1.40 (.106)
1.16 (.011)
10a
35.6 (2.78)
2.49 (.151)
2.56 (.355)
2.96 (.045)
7.00 (1.9)
1.19 (.028)
104
334 (18)
9.13 (2.29)
14.4 (2.89)
4.29 (.1)
143.3 (32)
3,43 (.757)
10
10a
32.5 (6.86)
1.74 (.121)
2.43 (.317)
2.90 (.023)
12.0 (3.03)
1.12 (.012)
103
324 (25)
8.94 (.79)
9.27 (.92)
2.96 (.031)
164 (18)
1.20 (.020)
104
3337 (134)
60.1 (4.6)
91.1 (4.5)
13.8 (1.2) 2880 (463)
13.3 (2.0)
finding good 02 of estimates in such a complex scenario, as summarized in Table 2 (right), but its estimates are still substantially better than those of R's Ime4.
5.2 Real Data
We first describe the datasets and prediction tasks used in this work.
Estimating physical activity from self-reported behaviors from the UK Biobank The UK Biobank is an ongoing large scale cohort study where over 500,000 individuals across the UK aged 40-69 were first invited to 22 assessment centers in 2006-2010, surveyed and measured for various behaviors and metrics, such as gender, job, personal habits, current and historical health status (20) Participants' information has been de-identified and strict protocols exist for obtaining and disposing of the UK Biobank data. Several sub-samples of this cohort have also been invited to participate in more expensive and time consuming surveys such as neuroimaging and genome sequencing. One sub-sample of over 100,000 participants wore a wrist-mounted accelerometer for period of days (7). We follow Pearce al. [19] who attempted to predict these accelerometer-basse physical activity (PA) data from much more readily available 14 self-reported behaviors such as no. of hours watching TV and sleeping, and gender. The motivation behind doing so is that physical activity is a strong marker for diseases outcomes such as cancer and respiratory disease mortality, yet is hard to measure. The authors used linear regression and ignored some important features. Of particular interest here is the current job feature, categorical feature with over 300 levels, unevenly distributed across participants. We attempt to improve prediction of physical activity by adding the job feature and using DNN in a LMMNN approach.
Estimating Drugs 1-10 rating from textual reviews from Drugs.com The website drugs.com is an online pharmaceutical database of drugs. providing information, reviews and ratings of thousands of doctor prescribed drugs. Gräßer et al. [10] scraped drugs.com for over 215,000 anonymized reviews and ratings which available in the UCI Machine Learning Repository (8) The authors discretized the 1-10 ratings into "negative", "neutral" and "positive" sentiments and used logistic regression on the text reviews processed to n-grams, predict sentiment in a classification framework. Here predict the rating itself in a regression framework using DNN, with a standard word embeddings layer followed by a LSTM layer and single neuron output. A high-cardinality categorical feature which should improve prediction is the drug itself and there are over 3,600 drugs reviewed and rated in this dataset (so each drug has about 60 reviews average, but these are unevenly distributed).
Landmark localization in CelebA facial images The CelebA dataset (Liu et al. 17 contains 202,599 cropped facial images from 10, celebrities, where each celebrity has between 1 and 35 images, annotated for various attributes and landmarks such as the location of the tip of the nose and mouth. We use two convolutional neural networks to predict the (X, Y) location of the nose and expect that treating facial images from the same individual as repeated measures should improve prediction. Therefore, the celebrity identity itself is the RE feature used here in the LMMNN framework.
8

---

Table 4: Real data features summary table
Dataset
71
Categorical
Input Type
DNN
UKB PA
96K
350
15
job drug
PA
Tabular
MLP
Drugs
215K
3.6K
10K
rating
Text
LSTM
CelebA
202K
10K
218x178x1
identity host
noseX-Y
Images
CNN
Airbnb
50K
40K
196
log(price)
Tabular
MLP
Table 5: Real data 5-CV mean test MSEs and standard enrors in parentheses. Bold results non-inferior to the best result in a paired t-test.
Dataset
Ignore
OHE
Embeddings
McNets
LMMNN
UKB PA
0.812 (.008)
0,816 (.009)
0,817 (.010)
0,811 (.009)
0.809 (.008)
Drugs
2.74 (.032)
2.77 (.005)
2.72 (.051)
2.81 (.031)
2.66 (.006)
CelebA noseX
1.68 (.05)
(.3)
7.6 (.3)
1.54 (.07)
CelebA nose
1.64 (.09)
2.5 (.2)
12.3 (1.1)
1.39 (.04)
Airbnb
0.156 (.002)
0.158 (.003)
0.153 (.003)
0.142 (.002)
Predicting prices of Airbnb rentals The website insideairbnb.com/ scrapes publicly available data from Airbnb rentals under the CCO 1.00license (5). Kalehhasti al. [13] attempted to predict the log price of a rental from various features such as neighborhood, number of rooms and textual features extracted from reviews, overall 755 features. After performing variable selection they used multiple methods including DNN and found support vector regression (SVR) to give the best result in terms of test MSE. Their dataset holds about 50,000 listings with over 39,000 hosts which they did not even consider when performing variable selection. Some hosts publish only rental, while some over 100.
In each dataset we use the same baseline DNN architecture detailed in the original analysis, available. If the original analysis did not include the use of DNNs or did not provide specific details, we use a standard architecture such as the L.STM used for modeling the Drugs dataset reviews. The RE variable is either ignored, goes through embeddings or OHE (if possible computationally) before entering the network, used the McNets framework or used in the LMMNN framework, where we use g(2) Z always. For specifies on baseline network architecture for each dataset see additional material. We perform 5-fold cross validation where in each iteration we use 10% of the train set (which is 80% of the entire data) validation data, and train the network until no improvement in validation loss seen within 10 epochs. All runs are made on Google Colab Pro, with a Nvidia Tesla P100 GPU.
Table 4 describes the datasets general, Table summmaiies the test MSE results and Table 3 in additional material summarizes mean runtime and number of epochs. It can be seen that including the high-cardinality variable in each of these datasets within a LMMNN framework achieves the lowest test MSE. We also refer the reader to additional material Figure 2 where show true test y vs. predicted scatter plots. Comparing to the results of the original papers challenging since not all have supplied data and methods in fully reproducible way and they use different metrics. Notwithstanding, we can verify our results are at least comparable in performance: For estimating physical activity in the UK Biobank, Pearce et al. [19] report achieving test R² of between 14 and 17%, while LMMNN achieves on average 19% (higher is better). For estimating drugs rating from text reviews cannot compare Gräßer et al. [10 results they worked in classification setting. For localizing the nose in the CelebA facial images, we are not aware of any MSE benchmarks but note that the LMMNN result of less than 2 pixels Airbnb rental price prediction, Kalehbasti et al. [15) report achieving a test MSE measure of 05157 for a neural network, while LMMNN achieves on average mean test MSE of 0. 142, which even better than the best result they achieved with SVR, 0.147.
average seems satisfactory. Finally, for the
LMM Extensions
The current paper is focused on regression settings with single categorical feature, e.g. the subject with repeated measurements, wanted to take advantage of the computationally attractive LMM

---

Table 6: Simulated model with g(Z) Z and two categorical features, mean test MSEs and standard emros in parentheses. Bold results are non-inferior to the best result a paired t-test.
01
62
Ignore
OHE
Embeddings
Ime4
LMMNN
0.5
0.5
10
10a
2.18 (.03)
1.45 (.02)
1.34 (.01)
2.97 (.03)
1.13 (.01)
103
104
2.15 (.02)
1.70 (.01)
1.68 (.02)
3.12 (.03)
1.23 (.01)
10
104
2.13 (.02)
1.83 (.02)
1.80 (.02)
3.23 (.03)
1.30 (.00)
0.5
5.0
10
10a
6.73 (.04)
1.66 (.03)
1.57 (.02)
3,00 (.04)
1.12 (.00)
103
10*
6.75 (.04)
2.20 (.03)
2.01 (.03)
3.31 (.02)
1.29 (.01)
104
10a
6.50 (.05)
1.88 (.03)
1.92 (.04)
3.15 (.01)
1.23 (.01)
10
104
6.68 (.12)
2.48 (.03)
2.16 (.02)
3,43 (.01)
1.37 (.01)
5.0
5.0
103
103
11.26 (.19)
1.83 (.02)
1.80 (.07)
2.97 (.02)
1.14 (.02)
103
104
11.33 (.19)
2.36 (.03)
2.11 (.02)
3.32 (.02)
1.30 (.01)
10a
104
11.24 (.09)
3.02 (.03)
2.55 (.000)
3.69 (.02)
1.49 (.02)
feature described in equation 66 of avoiding any matrix inversions in calculating the predicted RE and the NLL loss. However, in further experimenting with LMMNN we note that it seems to perform well with more complex mixed effects scenarios which do necessitate inverting part of the V covariance matrix in equation (8) on each mini-bated during training. Apart from the simulation in which we defined gZ) ZW and used embeddings in the network architecture, other scenarios include having multiple categorical features, the random slopes model which adds a "slope" random effect to each categorical level, and even kriging random fields, where V is assumed to contain spatio-temporal dependencies.
To give the reader flavour for how LMMNN can be generalized to more scenarios, consider situation of two uncorrelated categorical features, having Q1 and d2 levels. Now Z in 7 is an indicator matrix of size (q1 + 92), b of size 91 + 92 distributes before N (0, D), only now D is of size (q1 + 92) x (q1 + 42), and has structure:
o In 0g
Og X The T
(17)
Now & holds three variance components 11 1. and we can write the NLL and its derivatives in (8) and (9) exactly as before. However, notice that V in this case is not necessarily block-diagonal anymore, and LMMNN calculates the inverse on part of it for each mini-batch.
We show in Table 6 the mean test MSE results of simulation similar the first simulation in Section 5.1 where g is the identity matrix, / is in (15). of = 1, n 100, 000, using the same DNN architectures and rules. In fact the only difference from the simulation in Section [. that now have !wo high-cardinality uncorrelated categorical features, we vary (1 Q2, oi Tfe and LMMNN now has to estimate three variance components. As can seen, LMMNN's performance is superior to all other methods. It also estimates the variance components quite accurately when compared to R's 1me4, with a certain cost in runtime, see Tables and in additional material.
Conclusion
In this work we propose to treat high-cardinality categorical features in DNN as RE variables, to avoid the possible pitfalls of overfitting, over-parameterization and scalability issues often encountered when opting to use OHE or entity embeddings. LMMNN introduces a novel negative log likelihood loss function inspired by the well researched linear mixed effects model and its positive effect on DNN prediction performance shown, with a simple implementation in Keras which can be plugged in to almost any regression DNN architecture, and is freely available on Github.
Treating categorical features with many levels in DNN as random effects seems to promising as simulated and real data results show and we intend to further pursue this direction in future work.
10

---

Acknowledgments and Disclosure of Funding
This study was supported in part by fellowship from the Edmond Safra Center for Bioinformatics at Tel-Aviv University, and by Israeli Science Foundation grant 1804/16. UK Biobank research has been conducted using the UK Biobank Resource under Application Number 56885.
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- scale machine learning on heterogeneous systems, 2015. URL httph:ttp////. h tensorflox Software available from tensorflow.org.
org/
[2] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using lme4, Journal of Statistical Software, 67(1):1-48, 2015. doi: 10.18637/jss.v067. i01.
[3] Hao Chen, Lili Zheng, Raed AL Kontar, and Garvesh Raskutti. Stochastic gradient descent in correlated settings: A study gaussian processes. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2722-2733 Curran Associates, Inc., 2020. URL
[4] François Chollet et al. Keras. Intps://keerss 2015.
[5] Murray Cox. Inside airbnb: Adding data to the debate. http://Inssideiirbbb.coo com/
get-the-dar data. html 2021. [Online; accessed 7-April-2021).
[6] Chuong Do and Andrew Y. Ng. Transfer learning for text classification. In Y. Weiss, B. Schölkopf, and J. Platt, editors. Advances in Neural Information Processing Systems, vol- ume 18. MIT Press, 2006. URL htt.:/tppee/rrrreednnnneeeeeeeee htt.gngeweerrr....pppaee///////e httpp//s.../pppp/////////ee .pp
[7] Aiden Doherty, Dan Jackson, Nils Hammerla, Thomas Plötz, Patrick Olivier, Malcolm H. Granat, Tom White, Vincent T. van Hees, Michael I. Trenell, Christoper G. Owen, Stephen J. Preece, Rob Gillions, Simon Sheard, Tim Peakman, Soren Brage, and Nicholas J. Wareham. Large scale population assessment of physical activity using wrist worn accelerometers: The uk biobank study. PLOS ONE, 1-14, 02 2017. doi:
URL
https: sstps:///oi..o /ttp::///ooo.r//10.1
pone 0169649
[8] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http::////ccevvv ics. uci edu/ml
[9] Aurélien Géron. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O' O'Reilly Media, 2019.
[10] Felix Gräßer, Surya Kallumadi, Hagen Malberg, and Sebastian Zaunseder. Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning. In Proceed- ings of the 2018 International Conference on Digital Health, DH '18, page 121-125, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450364995. doi: 10.1 00114///144555 31946777.. URL https:/ttts////i..oo org/10. 1145/3194658 3194677
[11] Varun Gulshan, Lily Peng, Mare Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica Mega, and Dale R. Webster. Devel- opment and Validation of Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA, 316((2)24422222411 12 2016. ISSN 0098-7484. doi: 10. 10.1001/jama.2016 17216. URL https: doi org 10. 1001/ ama. 2016 17216

---

[12] Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables, 2016.
[13] John T. Hancock and Taghi M. Khoshgoftaar. Survey on categorical data for neural networks. Journal of Big Data, 7(1):28, Apr 2020, ISSN 2196- 1115. doi: 10.1186640033--00000000.. URLhttps:: postps://ont //t:://ooii org/ 10. 1186/ 84005770000000005-----
[14] Tong He, Ru Kong, Avram J. Holmes, Minh Nguyen, Mert R. Sabuneu, Simon B. Eick- hoff, Danilo Bzdok, Jiashi Feng, and B.T. Thomas Yeo. Deep neural networks and ker- nel regression achieve comparable accuracies for functional connectivity prediction of be- havior and demographics. Neurolmage, 206: 16276, 2020. ISSN 1053-8119. doi: https: //bb:oog/|..looo...emmmmmeeeeee0 scieette//rriccee////////1111119
URL https: http:://www.sccenceeiie
com/
[15] Pouya Rezazadeh Kalehbasti, Liubow Nikolenko, and Hoormazd Rezaei. Airbnb price prediction using machine learning and sentiment analysis, 2019.
[16] Yu- Wei Lin, Yuqian Zhou, Faraz Faghri, Michael J. Shaw, and Roy H. Campbell. Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory, PLoS ONE, 14(7), July 2019, doi: httpss:///doi 00g/10.0/// 1371/ pone. 0218942
[17] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[18] Charles F. McCulloch, Shayle R. Searle, and John M. Neuhaus. Generalized, Linear, and Mixed Models. John Wiley and Sons, Inc., June 2008. ISBN 978-0-470-00371--1
[19] Matthew Pearce, Tessa Strain. Youngwon Kim, Stephen J. Sharp, Kate Westgate, Katrien Wijndaele, Tomas Gonzales, Nicholas J. Wareham, and Søren Brage. Estimating physical activity from self-reported behaviours in large-scale population studies using network harmoni- sation: findings from uk biobank and associations with disease outcomes. International Journal of Behavioral Nutrition and Physical Activity, 17(1))40, Mar 2020. ISSN 1479-5868. doi: URL httLhhtps://// / org/ 10. / s12966 002122666020-0098-0 020-00937-4
10.1
[20] Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong. Jill Pell, Alan Silman, Alan Young. Tim Sprosen, Tim Peakman, and Rory Collins. Uk biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLOS Medicine, 12(3):1-10, 03 2015. doi: 10.137//jurmal, 1001..000779 URL
https: org 10. 1371/journal. jornnal... pmed. 1001779
[21] Minh-Ngoc Tran, Nghia Nguyen, David Nott, and Robert Kohn. Bayesian deep net glm and glmm. Journal of Computational and Graphical Statistics, 29(1)9771133, 2020. doi: 10. 1080/ 1061 00618600.2019.1637770
htt.::ttt::/ott org/ 10. 101.100000001800 106186001.000.00 2019. 1637747
[22] Yunyang Xiong, Hyunwoo J. Kim, and Vikas Singh. Mixed effects neural networks (menets) with applications to gaze estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[23] Yunyang Xiong, Hyumwoo J. Kim, Bhargav Tangirala, Ronak Mehta, Sterling C. Johnson, and Vikas Singh. On training deep 3d cnn models with dependent samples in neuroimaging. In Albert C. S. Chung, James C. Gee, Paul A. Yushkevich, and Siqi Bao, editors, Information Processing in Medical Imaging, pages 99-111, Cham, 2019. Springer International Publishing. ISBN 978-3-030-20351-11
[24] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance- gaze estimation in the wild. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4511-4520, June 2015.
12


Reviews:
[{"review_id": "fm8wc81syIt", "reviewer": "Reviewer_VuSr", "summary": "The paper presented a methodology for dealing with high-cardinality features extracted from categorical variables by using the one-hot encoding applied to deep learning models. The suggested methodology decomposed the model into two sections: fixed effects (FE) and random effects (RE) and it is based on the negative log-likelihood loss from the linear mixed models for neural networks. Presented results including simulated data and four different real data sets with several dozens of categorical variables. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-\tOriginality: The suggested methodology improves other techniques to deal with high-cardinality data such as MeNets and DeepGLMM. The authors established clearly the differences between their approach and benchmarks. The main difference is in the proposed way to estimate the variance of the random effect component. The related work is properly cited by the authors.\n-\tQuality: Despite authors referenced related work properly, it is not stated why high-dimensional cardinality is a major challenge for Deep Learning models. However, core ideas are properly stated and defined.\n-\tClarity: In the suggested methodology is not clear if the inverse of the matrix defined in equation (8) is explicitly computed on their approach. In fact, lines 121-123 state a fast way to compute the inverse of the matrix on small batch sizes, and then lines 127-129 presented an approximation for faster predictions.\n-\tSignificance: Results are important in terms of computing random effects for categorical high-dimensional problems. However, it could be relevant to present other application areas and to illustrate the limitations of deep learning models under high-dimensional cardinality. Besides, the authors present improvement in terms of MSE and standard errors compared to the benchmark, it could be also relevant to indicate the processing time between models to evaluate the level of resources required for their approach. \n\nQuestions to be addressed: \n- Is the suggested methodology useful for other types of categorical data encoding approaches?\n- What is the relationship between high-cardinality and random effects? Is the approach suitable for low-dimensional cardinality?\n- How do authors can determine the limit between high and lower dimensional cardinality?\n- In the case it is required to compute the inverse of the matrix V, does the suggested approach guarantee the estimated matrix is positive definite?\n- In figure 1 it is introduced the notation NLL(f,φ|y) which is not defined in the paper.\n- What is the difference between equations (6) and (11)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hrs"}, "extra_text": {}}, {"review_id": "H1I_5zo3EH2", "reviewer": "Reviewer_V1Q6", "summary": "This paper proposes to treat high-cardinality categorical features as the random effects of a mixed non-linear model. The proposed approach, called LMMNN, consists of minimizing the negative log-likelihood over both the functional estimates (parameterized as neural networks) and the variance components of the mixed non-linear model. The framework can be seen as an extension of the random intercepts model. The pipeline is eligible for automatic differentiation, though a closed form can be deduced in simple settings. Experiments are performed on both synthetic and various real datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written with smooth progressions. Related works are well discussed to the best of my knowledge, especially section 4.2 which provides an in-depth comparison with closest previous works. The literature on neural networks over spaces of probability measures (closely related to the estimation of the distribution of random effects) or extensions to non-Gaussian random effects could be mentioned.\n\nThe idea to treat categorical features with high cardinality as random effects is pertinent. \n\nHowever, as is well explained in the paper, the setting is currently limited to continuous explained variables and a single categorical feature to ease computations.\n\nResults seem promising, especially in high-dimensional scenarios. However, LMMNN seems to struggle to estimate variance parameters even in the case of the random intercepts model. Is it partly due to the approximate estimation of V? Are variance estimates more precise as the number of samples increases?\n\nI recommend a reject for this paper, on the grounds that the case study and experiments are limited.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and potential negative societal impacts are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "91FUelUFsLH", "reviewer": "Reviewer_LkaW", "summary": "This paper proposes a random effects based objective for deep neural networks. The idea is to modify the standard linear mixed effects objective by replacing the linear predictor with the output of a neural network. The resulting objective can be trained with regular SGD. By modelling a categorical variable (e.g., subject identity) as a random effect, the result is argued to better account for this variable’s influence on the target variable compared to one-hot encoding, and embedding approaches. Experiments on synthetic and real-world datasets demonstrate the feasibility of the proposed approach.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Overall recommendation\n\nI generally appreciate works that revisit classical statistical tools, particularly those that address pertinent challenges in training modern neural networks. However, I have two main concerns:\n- I did not find the proposal insightful – to my understanding, it involves swapping out the linear predictor in a LMM with the output of a neural network. While certainly it is valuable to explicate simple ideas that work well on practically relevant problems, it is hampered by the below point.\n- The problem setting considered is fairly restricted: one can employ only a single categorical feature in conjunction with continuous targets and square loss. \n\nGiven the above, I am not inclined to recommend this paper for acceptance in its present state.\n\n### Major comments\n\nThere are several potentially interesting technical challenges in applying random effects models in a neural context – e.g., moving beyond the square loss, handling multiple categorical variables, modelling hierarchical structure amongst such variables, …. Progress on any of these problems would be welcome. However, the present work is somewhat limited in focus (which the authors are upfront about). In particular, the requirement to use the MSE loss (not the canonical choice in training neural networks for most tasks, barring some recent work [Hui and Belkin, 2020]), and the restriction to using a single categorical feature make the problem setting yet more niche.\n\nThe presentation of the mixed effects model is done in generality. This is appreciated, though it seems that all the subsequent applications use the random intercept setting, wherein Z has a very specific structure. It would be useful in this case to explicate, e.g., the closed-form inverse of the matrix V, which I believe exists since the matrix will have a block structure, with each block submatrix being the sum of a (scaled) all-ones matrix plus an identity matrix.\n\nOne intuitive property of the mixed effects likelihood (Eqn 3) is that, by virtue of the non-isotropic covariance V, it allows for observations within a group to influence each other’s predictions. Concretely, the objective for β can be seen as adding cross-sample terms with weighting given by V^-1. It would have been nice to see more discussion of this, as the point is somewhat implicit in the present text. This is particularly since one could imagine modifying other losses to encode a similar effect.\n\nOne confusion in the framework is that (7) presents g(.) as being an instance of “non-trivial functions which DNNs are suitable for, e.g. non-linear and involving interactions”. However, on L127 and subsequently, it seems that g(.) is the identity function.\n\nIn discussing the relation to one-hot encoding, two claims are made:\n- It is hard to scale when q is large. What is the precise scalability problem when using SGD based optimisation? Surely even the present method will have a linear scaling with the number of possible categories is large?\n- The one-hot features carry little information. What is the additional information that the present method extracts? The paper is not explicit on this point.\n\nOne conceptual point that would be worth clarifying is the following. Eqn (2) can be written as y = A w + ε, where A = [X Z] is an n x (p + q) matrix, and w = [ β; b ] is a (p + q) dimensional vector. The matrix A can be seen as adding a one-hot encoding to each input sample’s features. One typically posits a prior distribution N(0, σ2) on β, for which the MAP estimate is implemented by L2 regularisation. One could imagine performing regularised least squares (or indeed, full Bayesian linear regression that marginalises out both β and b) on the matrix A; it is worth commenting on how this differs from the present solution.\n\nIn discussing the relation to embedding based methods, it could be useful to comment on modelling relational data (e.g., (user, item) interactions). Here, there is a natural “clustering” and repeated interaction structure in the data (as each user will typically interact with multiple items). The canonical approach to these problems is to apply some form of matrix factorisation, which can be seen as learning user and item embeddings. There is also work on using random effects style models in such settings [Hoff, 2005]. How do these relate to the present work?\n\nIt is not clear how precisely the Embedding approach is implemented: in contrast to the one-hot encoding (which is clear), how does the training objective employ the embedding associated with a given category?\n\nWhen discussing the prior DeepGLMM work, one claim is that the latter is more challenging to train, owing to its use of variational inference. Presumably one could similarly compute MAP/MLE estimates of the underlying model? It was not too clear how the model itself fundamentally differs from the present one.\n\n\n### Minor comments\n\n- Consider using boldface to denote matrices and vectors – it would make parsing some of the equations easier\n- L41: “a categorical feature with possibly thousands of levels — would be considered a random effect variable, often having normal distribution N(0, σ2)” → confusing, not clear how a categorical variable can follow a normal distribution\n- L77: cov → \\mathrm{cov}\n- Eqn 3: consider using \\top, not ‘\n- L93: _{tr}, _{te} → _{\\mathrm{tr}}, _{\\mathrm{te}}\n- Figure 1: it is not entirely clear what f(X) corresponds to in this figure. Is it the weights on the edges going into the sink node?\n- Suppose σb → 0; does the present model reduce to use one-hot encodings?\n- Another way of allowing for nonlinearity would be to allow f(X) to follow a Gaussian process. How would this compare to the present approach?\n- Would it be interesting to apply a standard LMM to features X that are extracted from a pre-trained neural network? What would be the pros and cons of this strategy?\n- Some comments on the relation of the random intercept model to multilevel modelling could be useful. See also [Hoff 2009], [Gelman, 2004; Sec 6].\n\n### References\n\n[Hui and Belkin, 2020] Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks. arXiV.\n\n[Hoff, 2005] Bilinear Mixed-Effects Models for Dyadic Data. JASA.\n\n[Hoff, 2009] A first course in Bayesian statistical methods. Springer.\n\n[Gelman, 2004] Analysis of variance—why it is more important than ever. The Annals of Statistics.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1ITXYaTEiGU", "reviewer": "Reviewer_avbJ", "summary": "The logic of this paper is clear and the experiment is complete, but it can be further improved. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "By considering the problem of high-cardinality categorical features are hard to scale, require much space, and are hard to interpret or may overfit the data, this paper proposes the LMMNN model, which adopting the corresponding negative log-likelihood loss from the linear mixed models statistical literature and integrates it in a deep learning framework. They test their model on simulated as well as real datasets with a single categorical feature with high cardinality, using various baselines neural networks architectures such as CNN and LSTM, and various applications in e-commerce, healthcare, and computer vision. The experiments on the real datasets demonstrate the application value of their model. In addition, the logic of the paper is clear and the supplementary materials are complete. All in all, this is a meaningful study.\n\nHowever, I still have some concerns as follows:\n\n1.\tLine 14: convolutional networks -> convolutional neural networks (CNN)\n2.\tAs far as we know, complex DNN generally have better feature screening capabilities, but the DNN they used as baseline models were relatively simple. How the authors demonstrate that their method is superior to more advanced network architectures in improving performance.\n3.\tThe authors should improve their language writing to avoid grammar mistakes.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 days"}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the area chair, I recommend that this paper be accepted and published. The main reason is that the paper is genuinely useful to practitioners, because it addresses an issue that arises in many applications and the solution is clear and simple.

A paper like this one should not be rejected because the authors have failed to do some work that reviewers would like to see. That work should be in future papers. The authors are clearly knowledgeable and competent. The technical work is of good quality and correct. Practitioners and future researchers can use and build on this.

In the final version, hopefully the authors will address the last part of the following comment from an expert reviewer: "High-cardinality categorical variables are pervasive in practice. An effective, generic, clearly expressed solution to coping with such variables would certainly be a welcome contribution to the literature. This is true even if the solution has modest technical novelty; indeed, it would be a virtue if one could arrive at a simple solution based on existing ideas. One could make a case that the authors' solution is effective. However, I am less sure about generic and clearly expressed..."
Decision: Accept (Poster)