Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a random effects based objective for deep neural networks. The idea is to modify the standard linear mixed effects objective by replacing the linear predictor with the output of a neural network. The resulting objective can be trained with regular SGD. By modelling a categorical variable (e.g., subject identity) as a random effect, the result is argued to better account for this variable’s influence on the target variable compared to one-hot encoding, and embedding approaches. Experiments on synthetic and real-world datasets demonstrate the feasibility of the proposed approach.
main_review: ### Overall recommendation

I generally appreciate works that revisit classical statistical tools, particularly those that address pertinent challenges in training modern neural networks. However, I have two main concerns:
- I did not find the proposal insightful – to my understanding, it involves swapping out the linear predictor in a LMM with the output of a neural network. While certainly it is valuable to explicate simple ideas that work well on practically relevant problems, it is hampered by the below point.
- The problem setting considered is fairly restricted: one can employ only a single categorical feature in conjunction with continuous targets and square loss. 

Given the above, I am not inclined to recommend this paper for acceptance in its present state.

### Major comments

There are several potentially interesting technical challenges in applying random effects models in a neural context – e.g., moving beyond the square loss, handling multiple categorical variables, modelling hierarchical structure amongst such variables, …. Progress on any of these problems would be welcome. However, the present work is somewhat limited in focus (which the authors are upfront about). In particular, the requirement to use the MSE loss (not the canonical choice in training neural networks for most tasks, barring some recent work [Hui and Belkin, 2020]), and the restriction to using a single categorical feature make the problem setting yet more niche.

The presentation of the mixed effects model is done in generality. This is appreciated, though it seems that all the subsequent applications use the random intercept setting, wherein Z has a very specific structure. It would be useful in this case to explicate, e.g., the closed-form inverse of the matrix V, which I believe exists since the matrix will have a block structure, with each block submatrix being the sum of a (scaled) all-ones matrix plus an identity matrix.

One intuitive property of the mixed effects likelihood (Eqn 3) is that, by virtue of the non-isotropic covariance V, it allows for observations within a group to influence each other’s predictions. Concretely, the objective for β can be seen as adding cross-sample terms with weighting given by V^-1. It would have been nice to see more discussion of this, as the point is somewhat implicit in the present text. This is particularly since one could imagine modifying other losses to encode a similar effect.

One confusion in the framework is that (7) presents g(.) as being an instance of “non-trivial functions which DNNs are suitable for, e.g. non-linear and involving interactions”. However, on L127 and subsequently, it seems that g(.) is the identity function.

In discussing the relation to one-hot encoding, two claims are made:
- It is hard to scale when q is large. What is the precise scalability problem when using SGD based optimisation? Surely even the present method will have a linear scaling with the number of possible categories is large?
- The one-hot features carry little information. What is the additional information that the present method extracts? The paper is not explicit on this point.

One conceptual point that would be worth clarifying is the following. Eqn (2) can be written as y = A w + ε, where A = [X Z] is an n x (p + q) matrix, and w = [ β; b ] is a (p + q) dimensional vector. The matrix A can be seen as adding a one-hot encoding to each input sample’s features. One typically posits a prior distribution N(0, σ2) on β, for which the MAP estimate is implemented by L2 regularisation. One could imagine performing regularised least squares (or indeed, full Bayesian linear regression that marginalises out both β and b) on the matrix A; it is worth commenting on how this differs from the present solution.

In discussing the relation to embedding based methods, it could be useful to comment on modelling relational data (e.g., (user, item) interactions). Here, there is a natural “clustering” and repeated interaction structure in the data (as each user will typically interact with multiple items). The canonical approach to these problems is to apply some form of matrix factorisation, which can be seen as learning user and item embeddings. There is also work on using random effects style models in such settings [Hoff, 2005]. How do these relate to the present work?

It is not clear how precisely the Embedding approach is implemented: in contrast to the one-hot encoding (which is clear), how does the training objective employ the embedding associated with a given category?

When discussing the prior DeepGLMM work, one claim is that the latter is more challenging to train, owing to its use of variational inference. Presumably one could similarly compute MAP/MLE estimates of the underlying model? It was not too clear how the model itself fundamentally differs from the present one.


### Minor comments

- Consider using boldface to denote matrices and vectors – it would make parsing some of the equations easier
- L41: “a categorical feature with possibly thousands of levels — would be considered a random effect variable, often having normal distribution N(0, σ2)” → confusing, not clear how a categorical variable can follow a normal distribution
- L77: cov → \mathrm{cov}
- Eqn 3: consider using \top, not ‘
- L93: _{tr}, _{te} → _{\mathrm{tr}}, _{\mathrm{te}}
- Figure 1: it is not entirely clear what f(X) corresponds to in this figure. Is it the weights on the edges going into the sink node?
- Suppose σb → 0; does the present model reduce to use one-hot encodings?
- Another way of allowing for nonlinearity would be to allow f(X) to follow a Gaussian process. How would this compare to the present approach?
- Would it be interesting to apply a standard LMM to features X that are extracted from a pre-trained neural network? What would be the pros and cons of this strategy?
- Some comments on the relation of the random intercept model to multilevel modelling could be useful. See also [Hoff 2009], [Gelman, 2004; Sec 6].

### References

[Hui and Belkin, 2020] Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks. arXiV.

[Hoff, 2005] Bilinear Mixed-Effects Models for Dyadic Data. JASA.

[Hoff, 2009] A first course in Bayesian statistical methods. Springer.

[Gelman, 2004] Analysis of variance—why it is more important than ever. The Annals of Statistics.
limitations_and_societal_impact: Limitations are adequately discussed.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes to treat high-cardinality categorical features as the random effects of a mixed non-linear model. The proposed approach, called LMMNN, consists of minimizing the negative log-likelihood over both the functional estimates (parameterized as neural networks) and the variance components of the mixed non-linear model. The framework can be seen as an extension of the random intercepts model. The pipeline is eligible for automatic differentiation, though a closed form can be deduced in simple settings. Experiments are performed on both synthetic and various real datasets.
main_review: The paper is well written with smooth progressions. Related works are well discussed to the best of my knowledge, especially section 4.2 which provides an in-depth comparison with closest previous works. The literature on neural networks over spaces of probability measures (closely related to the estimation of the distribution of random effects) or extensions to non-Gaussian random effects could be mentioned.

The idea to treat categorical features with high cardinality as random effects is pertinent. 

However, as is well explained in the paper, the setting is currently limited to continuous explained variables and a single categorical feature to ease computations.

Results seem promising, especially in high-dimensional scenarios. However, LMMNN seems to struggle to estimate variance parameters even in the case of the random intercepts model. Is it partly due to the approximate estimation of V? Are variance estimates more precise as the number of samples increases?

I recommend a reject for this paper, on the grounds that the case study and experiments are limited.
limitations_and_societal_impact: The limitations and potential negative societal impacts are discussed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presented a methodology for dealing with high-cardinality features extracted from categorical variables by using the one-hot encoding applied to deep learning models. The suggested methodology decomposed the model into two sections: fixed effects (FE) and random effects (RE) and it is based on the negative log-likelihood loss from the linear mixed models for neural networks. Presented results including simulated data and four different real data sets with several dozens of categorical variables. 
main_review: -	Originality: The suggested methodology improves other techniques to deal with high-cardinality data such as MeNets and DeepGLMM. The authors established clearly the differences between their approach and benchmarks. The main difference is in the proposed way to estimate the variance of the random effect component. The related work is properly cited by the authors.
-	Quality: Despite authors referenced related work properly, it is not stated why high-dimensional cardinality is a major challenge for Deep Learning models. However, core ideas are properly stated and defined.
-	Clarity: In the suggested methodology is not clear if the inverse of the matrix defined in equation (8) is explicitly computed on their approach. In fact, lines 121-123 state a fast way to compute the inverse of the matrix on small batch sizes, and then lines 127-129 presented an approximation for faster predictions.
-	Significance: Results are important in terms of computing random effects for categorical high-dimensional problems. However, it could be relevant to present other application areas and to illustrate the limitations of deep learning models under high-dimensional cardinality. Besides, the authors present improvement in terms of MSE and standard errors compared to the benchmark, it could be also relevant to indicate the processing time between models to evaluate the level of resources required for their approach. 

Questions to be addressed: 
- Is the suggested methodology useful for other types of categorical data encoding approaches?
- What is the relationship between high-cardinality and random effects? Is the approach suitable for low-dimensional cardinality?
- How do authors can determine the limit between high and lower dimensional cardinality?
- In the case it is required to compute the inverse of the matrix V, does the suggested approach guarantee the estimated matrix is positive definite?
- In figure 1 it is introduced the notation NLL(f,φ|y) which is not defined in the paper.
- What is the difference between equations (6) and (11)?

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The logic of this paper is clear and the experiment is complete, but it can be further improved. 
main_review: By considering the problem of high-cardinality categorical features are hard to scale, require much space, and are hard to interpret or may overfit the data, this paper proposes the LMMNN model, which adopting the corresponding negative log-likelihood loss from the linear mixed models statistical literature and integrates it in a deep learning framework. They test their model on simulated as well as real datasets with a single categorical feature with high cardinality, using various baselines neural networks architectures such as CNN and LSTM, and various applications in e-commerce, healthcare, and computer vision. The experiments on the real datasets demonstrate the application value of their model. In addition, the logic of the paper is clear and the supplementary materials are complete. All in all, this is a meaningful study.

However, I still have some concerns as follows:

1.	Line 14: convolutional networks -> convolutional neural networks (CNN)
2.	As far as we know, complex DNN generally have better feature screening capabilities, but the DNN they used as baseline models were relatively simple. How the authors demonstrate that their method is superior to more advanced network architectures in improving performance.
3.	The authors should improve their language writing to avoid grammar mistakes.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: As the area chair, I recommend that this paper be accepted and published. The main reason is that the paper is genuinely useful to practitioners, because it addresses an issue that arises in many applications and the solution is clear and simple.

A paper like this one should not be rejected because the authors have failed to do some work that reviewers would like to see. That work should be in future papers. The authors are clearly knowledgeable and competent. The technical work is of good quality and correct. Practitioners and future researchers can use and build on this.

In the final version, hopefully the authors will address the last part of the following comment from an expert reviewer: "High-cardinality categorical variables are pervasive in practice. An effective, generic, clearly expressed solution to coping with such variables would certainly be a welcome contribution to the literature. This is true even if the solution has modest technical novelty; indeed, it would be a virtue if one could arrive at a simple solution based on existing ideas. One could make a case that the authors' solution is effective. However, I am less sure about generic and clearly expressed..."
Decision: Accept (Poster)