LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models
Authors: Mohammadreza Nemati
Zhipeng Huang
Kevin S. Xu
Venue: Accepted by TMLR ()

Abstract: Some of the simplest, yet most frequently used predictors in statistics and machine learning use weighted linear combinations of features. Such linear predictors can model non-linear relationships between features by adding interaction terms corresponding to the products of all pairs of features. We consider the problem of accurately estimating coefficients for interaction terms in linear predictors. We hypothesize that the coefficients for different interaction terms have an approximate low-dimensional structure and represent each feature by a latent vector in a low-dimensional space. This low-dimensional representation can be viewed as a structured regularization approach that further mitigates overfitting in high-dimensional settings beyond standard regularizers such as the lasso and elastic net. We demonstrate that our approach, called LIT-LVM, achieves superior prediction accuracy compared to the elastic net, hierarchical lasso, and factorization machines on a wide variety of simulated and real data, particularly when the number of interaction terms is high compared to the number of samples. LIT-LVM also provides low-dimensional latent representations for features that are useful for visualizing and analyzing their relationships.

Published in Transactions on Machine Learning Research (09/2025)
LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models
Mohammadreza Nemati
mohammadree...emai@aaee.cc
Deportment of Computer and Data Sciences
Case Western Reserve Universily
Zhipeng Huang"
hhipeng..uanngee Waaffug edu
Department of Mathemalics and Compuler Science Suffolk University
Kevin S. Xu
ksu2.0case. edu
Department of Computer and Data Sciences Case Western Reserve University
Reviewed on OpenReview: https: opw.reviee.....e...mmmoo net/ forum? 3uW5neESul
Abstract
Some of the simplest, yet. most frequently used predictors in statistics and machine learning use weighted linear combinations of features. Such linear predictors can model non-linear relationships between features by adding interaction terms corresponding to the products of all pairs of features. We consider the problem of accurately estimating coefficients for interaction terms in linear predictors. We hypothesize that the coefficients for different interaction terms have approzimate low-dimensional structure and represent each feature by latent vector in a low-dimensional space. This low-dimensional representation can be viewed as a structured regularization approach that further mitigates overfitting in high-dimensional settings beyond standard regularizers such as the lasso and elastic net. We demonstrate that our approach, called LIT-LVM, achieves superior prediction accuracy compared the elastic net, hierarchical lasso, and factorization machines on wide variety of simulated and real data, particularly when the number of interaction terms is high compared to the number of samples. LIT-LVM also provides low-dimensional latent representations for features that are useful for visualixing and analyzing their relationships
1 Introduction
Some of the simplest, yet most commonly used models in statistics and machine learning model an example's target variable using a weighted linear combination of its features 31, Ø2...., Tp. These linear prediction models, such as linear regression and logistic regression, are favored for their spoed, simplicity, and inter- pretability. In high-dimensional settings where the number of features p is close to even exceeds the number of examples n, regularization techniques such the lasso (Tibshirani 1996) and elastic net (Zou & Hastie 2005), have become essential to prevent overfitting and enhance prediction accuracy unseen test data.
The primary limitation of linear predictors is their reliance on linear combinations of features, which reduces their flexibility. To enhance their flexibility, a common approach is to incorporate polynomial features, which create new features using products of the original features. Perhaps the simplest case involves computing all second-order interaction terms corresponding products T, for j k, which describe the interactions between pairs of different features. In classical statistical settings where p is much smaller than n, including these (2) interaction terms does not complicate the estimation process, the total number of features would
*Research partially conducted while Huang was Case Western Reserve University.

---

Published in Transactions on Machine Learning Research (09/2025)
Features
Interactions
B
1
B2
B3
[4]
flat
1012
013
14
023
2z4
34
X11
X12
X13
X14
X12 X21 X22
X11X13
X11
X12X13
X12
X13X14
X
X21
X22
X23
X24
int
X21
X21
X22
X22
X2
P
p
Latent Feature Representation Structured Regularizatio: Contribution
n1 x
2
Z2
Interaction Matrix
Z
82
Z 11
Z12
0
01
0
13
014 4
12
yy
12
221
Z22
0
0
023
0
24
2/1
Z
0
0
0
04
Z31
Z32
024
0
0
0
014
3
Z41
Z42
p x
X d
Z4
Figure 1: Our proposed framework for linear predictors with interaction terms. We arrange the coefficient vector for the interaction terms Otat into an upper triangular matrix @ Our main contribution is to impose an appronimate low-dimensional structure on θ using a structured regularization. We represent each of the p Teatures by latent vector 2j in dimensions, with < p, and minimize the deviation between 0j* and a function of zj and 2 such the dot product zz Zy. In this toy example, p == and 2.
be on the order of p², which still quite low compared to n. However, as p gets closer to n, p² quickly surpasses n, and even stronger regularization may be needed to prevent overfitting.
Unlike the original features, interaction terms have additional structure, as they denote relationships between two features. The coefficients for these interaction terms be arranged in p x p coefficient matrix Q where entry 0yk denotes the cocfficient for the interaction term TyJk. Prior rescarch for interaction terms has typically assumed hierarchical structure such that an interaction wetween features j. k can only be present in the model if either feature j or k is in the model (Bien et al. 2013). We do not make such an assumption and further hypothesize that the cocfficient matrix @ has an approsimate low-dimensional structure, so that we can represent each feature j by a vector zj in d-dimensional space, where d < p.
structured sparsity
Our main contribution is structured regularization approach for estimating the coefficients for interaction terms in linear predictors using an approximate low-dimensional structure, shown in Figure I in addition to traditional regularization techniques such as the elastic net. We call our proposed approach LIT-IVM, denoting linear and interaction terms with a latent variable model, and is applicable to many types of linear predictors, inchuding lincar regression, logistic regression, and the Cox proportional hazards (Cox PH) model Coox 1972)- LIT-IVM fits the space of models between elastic net with interactions, which provides sparse but unstructured interaction cocfficient estimates; sparse factorization machines (Atarashi et al.) 2021 J which use exact low-rank factorizations of the interaction coefficients that are too rigid; and nou-linear predictors, which are more expressive but not interpretable linear predictors with interactions.
Our proposed structured regularization for the interaction cocfficients offers two main benefits. First, it improves estimation accuracy for the interaction coefficients, leading to better predictive accuracy, especially as grows relative to n, which we demonstrate simulation experiments and across many real datasets for both regression and classification settings. Second, provides low-dimensional latent representations of the features that are useful for visualizing and analyzing their relationships, which we demonstrate in an application for modeling donor-recipient compatibility in kidney transplantation.
2

---

Published in Transactions on Machine Learning Research (09/2025)
2
Background
2.1 Linear Predictors
Linear predictors comprise some of the simplest models used in statistics and machine learning, They offer the advantage of being far more interpretable than complex non-linear predictors, though sometimes at the cost of reduced prediction accuracy. Linear predictors generate a prediction ÿ the dependent variable or target y as a function of lincar combinations of the independent variables or features 21, ,,,,..,,,
1 = [(B 3121 + ... + Byxp)) f(8" x),
(1)
where By is intercept term, and 31. ........ 3p the coefficients that quantify the contribution of each feature. They are combined to form the coefficient vector 8 = [So .,, Spl. and the features are combined with a dummy variable of 1 for the intercept to form the feature vector = (I,
Estimation In scenarios where the number of examples n significantly exceeds the number features p. the coefficients or weights 3 can typically be accurately estimated using the maximum likelihood estimate (MLE). However, in many application scttings, particularly within the biomedical domain, is often on the same order p even smaller, leading situation where the MLE can result in overfitting results in models that perform well on training data but poorly unseen test data. To counteract this, regularization methods like the lasso (Tibshirani 1996), which uses an f penalty, and clastic net (Zon & Hastie 2005), which uses both & and &g penalties, are often employed to reduce overfitting and enhance the generalization ability of the model. These regularizers tend to favor estimated coefficient vectors that are sparse and have smaller magnitudes.
overfitting. Such
To capture non-linear relationships between features, linear predictors can be extended by incorporating interaction terms. Adding all second-order interaction terms involves adding all products Zy with j < k as additional features. The coefficients for these (g) interaction terms are stored in the interaction matriz A:
Interaction Terms
0
11
013
1.,
023
02.
(2)
0p 1.P
Let the vectors Onat 1012, 013......................... 11. pp.... ..... 0 0p-1.pl and Xint [x] 2143,..., "1 (pp.1,11711)) denote the flattened vector representations of O and the interaction terms. Then, equation [ can be written as
9 = f(3 x + Xime) ((8Tx))
(3)
where 3 [[, Paat] and x = [x, Xma] denote the augmented coefficient and feature vectors, respectively.
With the addition of interaction terms, the size of the augmented coefficient vector B on the order of p², regularization even more important to prevent overfitting, Since interaction terms represent relations between different features, their coefficients O may possess some additional structure that is not present in the coefficients 3 for the raw features. In this paper, we propose approximate low-dimensional structure for ④ that leads to structured regularization for interaction coefficients.
2.2
Related Work
2.2.1
Interaction Terms and Interpretability
Lincar predictors with interaction terms have long boen used as an interpretable approach for modeling non-linear relationships between features. Interaction terms can provide the same benefit for generalized additive models (GAMs), noted by Lou al. (013)). who proposed the GA2M model to extend GAMs
3

---

Published in Transactions on Machine Learning Research (09/2025)
with pairwise feature interactions, significantly enhancing model accuracy while maintaining interpretability. Higher-order interaction terms can also be added, e-g., in the scalable polynomial additive models (SPAMs) proposed by Dubey et al. (2022). where the maximum order of the interaction terms controls the flexibility of the model. Limiting the maximum order controls the trade-off between flexibility and interpretability, with second-order or pairwise interactions lypically representing the "sweet spot" " weew the two (Dubey el al. 2022
One challenge when adding interaction terms to linear predictors and GAMs is how to handle large number of interaction terms. If we add all pairwise interactions to model, the p features produce (3) interaction terms, which is often a larger number than the number of samples using lasso or elastic net penalties. One approach to avoid this is interaction detection: selectively add only the interactions that pass some statistical test. Lou et al. (2013) adopt this approach and propose a computationally efficient method to rank interactions. Their approach scales to p in the thousands, which results in millions of possible pairwise interactions. Another approach to assume that the interaction matrix O possesses some low-dimensional structure, such as low rank. This is the assumption made by factorization machines, which we discuss in Section 2.2.3
and can lead to overfitting even when
Another line of research on interaction terms focuses on structured sparsity using the notion of heredity.
Under the strong heredity structure, the interaction coefficient 05k * 0 only if 83 # 0 and B& # 0, implying that both the features j, must be selected in order to select their interaction. Several regularization methods that incorporate strong heredity or a relaxed version termed weak heredity have been proposed (Zhao el al., 2009: Yuan et al. 2009; Choi et al.) 2010 Radchenko & James, 2010 Bien et al.) 2013). The hierarchical lasso Bien et al. 2013) is probably the most similar to the methods consider in this paper, it also uses elastic net penalty, where the & portion enforces the strong weak heredity structure. The heredity assumption simplifies the discovery of interaction terms, as one can first select a smaller number of features through their main effects 3 and then consider only interactions between the selected terms, but does not assumc any low-dimensional structure on the interaction matrix.
2.2.2 Latent Variable Models
Latent variable models (LVMs) are used extensively in statistics and machine learning and aim to uncover latent variables representing hidden factors not directly observable but inferred from available data. An LVM particularly relevant to this paper is the latent space model (LSM) proposed by Hoff et al. (2002) for graph-structured data. Under this model, conditional independence of node pairs is assumed, given the latent. positions of the nodes in Euclidean space. In the most commonly used form of the LSM, the log odds of edge forming between two nodes i and j is given by og + Q1 -llz. 2yll2, where Tay denotes obscrved covariales about the node pair, 2, and 2, are the latent positions of nodes and j in d-dimensional latent space, and og and are scalar parameters. This model implies a greater probability of an edge between nodes that are closer together in the latent space.
Huang& & Xu 2022) proposed a novel LSM to analyze human leukocyte antigen (HLA) compatibility from data on kidney transplant outcomes. They first fit a Cox PH model with interaction terms using an & penalty. They propose to improve the estimates of the interaction coefficients using an LSM for the HLA compatibility network. Their proposed approach uses two-step estimation: first optimizing the be-eeallzedd Cox PH model's partial likelihood to estimate interaction coefficients and then optimizing the LSM likelihood to refine the estimates. On the other hand, our approach uses joint estimation over the model cocfficients and latent vectors and can be applied broadly to many linear predictors and regularizers.
2.2.3 Factorization Machines (FMs) and Related Models
The first FM model was proposed by Rendle 2010). It assumes that each feature has d-dimensional latent vector z, such that θ - =zz". This forces 4 to have an ezact low-rank structure. Sparse FMs (Xu
et al. 2016 Pan et al. 2016 Atarashi et al. 2021 further constrain θ to be sparse; we discuss these models
further in Section 3.3
Factorization of interaction terms has also been applied to models with time- varievvrrnngggg coefficients through the factorized structured regression (FaStR) model proposed by Rügamer al. (002). FMs have been used for
4

---

Published in Transactions on Machine Learning Research (09/2025)
modeling in recommender systems, and the extension to time-varying coefficients in FaStR enables modeling in time-aware recommender systems where user behaviors change over time. FMs have also been extended to higher-order interaction terms in a scalable manner, including higher-order FMs (Blondel et al. 2016) and additive higher-order FMs (Rügamer) 2024). which combine the benefits of GAMs with the scalability of lactorized terms.
FMs and other models with factorized coefficients differ from our proposed LIT-LVM approach in two main aspects. First, they are heavily concerned with scalability when using interaction terms and are often applied to data with both large and p with sparse feature matrix X (Rendle 2012 2013 On the other hand, our focus is on improving accuracy of estimation and prodiction, not scalability, so our approach is not more scalable than directly modeling interaction terms. Second, FMs and other models with factorized coefficients assume that o ezactly low rank, whereas we assume that O has an opprozimate now-dimensional structure (possibly low rank, but not necessarily). This a subtle, yet important distinction, and we provide more details in Section B.
3 Model Formulation and Estimation
3.1 LVM for Interaction Terms
We propose to improve the estimation accuracy of the coefficients for interaction terms by introducing an IVM for these coefficients. We make underlying assumption that the p x p interaction coefficient matrix O has approdimate low-dimensional structure captured by the matrix p x d matrix Z. (Note that this does not imply a low-dimensional structure for the n x data matrix X itself.) We discuss two possible models in the following, a low rank and a latent distance model. Other continuous TVMs or matrix factorizations could also be used instead, such as non-negative matrix factorization Lee & Scung 1999..
Low Rank Model This model hypothesizes that the interaction matrix @ may exhibit an approximate low rank structure. Under this model, the interaction coefficient for any pair j, such that j k is given by
Ojs 27 2x Efk:
(4)
where Ejk is zero-mean error lerm for the pair j, k representing the deviation from the low rank structure. This model is closely related to factorization machines, which assume that 0, zz 2x. ie., an exact rather than approximate low rank structure.
Latent Distance Model This model is inspired by the LSM of Hoff et al. (2002), where the distances between latent vectors is of interest. Under this model, the intcraction coofficient for any pair j, k such that defined as
0₁ = a. |z, 2 E
(5)
where the intercept ag denotes the baseline interaction level. The latent distance model has negative effect for distance that features with more positive values of interactions are placed closer together in the latent space and more negative values of interactions are pushed farther apart. This type of negative effect on distance has also been used for matrices with both positive and nogative entries by Huang & Xu (2022) and Nakis et al. (0233 and is often more interpretable than models with a positive effect on distance (Huang et al. 2022
3.2
Estimation Procedure
We propose to estimate the model parameters 3, which denotes the augmented coefficient vector combining B and the flattened version of @ as described Scction 2.1 by minimizing the total loss function
Liotal = Cpred + Xr XtCivm:
(6)
where 7, and x are hyperparameters controlling the strength of the traditional regularization and the IVM-structured regularization, respectively.
5

---

Published in Transactions on Machine Learning Research (09/2025)
The initial term, Epred, addresses the conventional loss associated with the linear predictor, such mean- squared error for linear regression, log loss for logistic regression, and negative log partial likelihood for the Cox PH model. The second component, Lrog, is associated with standard regularization techniques for linear prodictors, such as the clastic net (Zou & Hastic 2005). The third component, Livm is associated with the LVM for the interaction lerm weights. This component ensures that the estimated interaction coefficients Ô align with the hypothesized approximate low-dimensional structure.
Depending on the choice of linear predictor, regularizer, and LVM, the form of the total loss function will differ. For example, for the case of linear regression with elastic net regularization and low rank LVM, the total loss would be
Lrotal (,
lly x
+ + +1 + + ZZT
mean squared error
not regularizer
low rank IVM
Our Contribution Our main contribution is the third term Livm = -|E in equation 6 denoting the deviation from a low-dimensional structure. c is the matrix of error terms between the interaction coefficients Ojk and reconstructiog ions from their latent representations Zj, Zb- By minimizing squared distances between the interaction coefficients and their reconstructions from diw-dinensional representations, we encourage the interaction coefficients to have an approzimate low-dimensional structure. Our structured regularization approach can be easily modified for settings where not all interactions are of interest, we describe in Section A.1
with various learning rates [0.005, 0.01, 0.05, 0.1] our optimizer for the total loss function. To handle the E penalty, we use the proximal gradient method (Parikh & Boyd 2014), where the proximal operator corresponds to the soft thresholding operator. The other terins are all differentiable, with derivatives computed by automatic differentiation in Py The mnttaaiiaatios of B and Z are drawn from normal distributions: 3, ~ NO, 1) and žyk ~ N(0, 1). The time complexity of our estimation procodure per epoch is O(np2), as we discuss in Section A.2
Optimization We use Adam Kingma & Ba 2015
3.3 Comparison with Related Methods
Nuclcar Norm Regularization One approach to estimating low rank and sparse matricos is by adding nuclear norm and & penalties. Such an approach has been used in robust principal component analysis (PCA) (Candès et al. 2011 h to estimate matrix of the form M Lo + So, where Lg low rank and So is sparse.
More closely related to our work, Richard et al. 2012 and Zhou al. (0013) propose approaches also using nuclear norm and & penalties to estimate single matrix S that is simultaneously sparse and low rank. If we applied these penalties to linear regression with interaction terms, it would result in the loss function
ly x1 + A 11111111 + A
n
where |0]. = -0),=E,,((0) denotes the nuclear or trace norm consisting of the sum of the singular values of @ This approach has the advantage of being convex; however, the disadvantage is that it requires repeated computation of singular value decomposition as part of the optimization routine, which has O(p") complexity. If we adopted this approach for our setting rather than introducing the latent variables Z, then time complexity per epoch would be O(np"), higher than the O(np²) of our LIT-LVM approach. Additionally, are also interested in the latent vectors z, themselves, which can also help to interpret the structure of the interaction matrix O
Sparse Factorization Machines Another approach to estimating low rank and sparse matrices is sparse
FMs, which have been developed to improve the performance and interpretability of FMs (Xu et al. 2016 Pan et al. 2016 Atarashi et al. (021).. Sparse FMs model the interaction cocfficients by O ZZ' such that O is sparse. Since Z p x d with d < p. sparse FMs constrain O to have at most rank d, i.c., an exact low-rank representation. On the other hand, our proposed LIT-IVM approach, when used with low rank
6

---

Published in Transactions on Machine Learning Research (09/2025)
d2
wlasdk
elacticc net
1.0
10
0.9
Noan RaSE Ing scler
Moa ALC
0.0
10"
0.7
10
0.6
0.5
p:40
p:30
p:60
p:70
P:20
p:30
p:40
p:50
p:60
(a) Linear regression with low rank penalty
(b) Logistic regression with latent distance penalty
Figure/ 2: Results of simulation experiments on linear and logistic regression where the true latent dimension for the interaction weights θ is drrue = 2. a Lower RMSE and (b) higher AUC denote higher prediction accuracy. Error bars denote one standard error. Our proposed LIT-LVM approach, which uses structured regularization, significantly outperforms the elastic net with interaction terms, a traditional regularization approach, as p increases for fixed n.
model, penalizes the deviation from a low rank representation using the penalty |0 - IZ resulting in O being approdimately rather than exactly low rank and enabling greater modeling flexibility. The trade-off for the greater flexibility is higher time and space complexity as we need to work with the entire p x p matrix O rather than just with the p x d matrix Z. In the limit X - oo, the LIT-LVM estimate of θ approaches that of sparse FM, any deviation from the low rank structure is heavily penalized.
4 Simulation Experiments
We begin with some simulation experiments to investigate the effects of LIT-IVM as we change the number of features p with fixed number of examples n. As p increases, the model becomes more difficult estimate. Code to reproduce all experiments is available at https:///iithb. -
4.1 Linear Regression with Low Rank
We construct feature matrix X with dimensions p. Each data point x; NO, From the feature matrix, then construct the matrix of interaction terms Xint, which has dimensions (5). The target vector y is then generated by y X,3 + Xint + The entries of the weight vector B, 10, Z has dimensions x dirue with dirue < so that θ possesses the approximate low rank structure in equation with entries Zjk- (0, and εソą ~ A00 0.1). Finally, the observation noise ~ ((0,0.01)
Results The results for fixed 1,000, true latent dimension derue 2, and four different values of p are shown in Figure 2a We compare our proposed LIT-IVM approach using a low rank model to elastic net with interaction terms. For 40, e has (10) = 780 entries, which is less than n, and both approaches are quite accurate. At p 50, + has (5)) 1, 225 entries, which now exceeds n, and observe a phase transition where the RMSE of clastic net increases significantly compared to our LIT-LVM approach. As continues to increase, LIT-LVM continues to outperform the elastic net with interactions.
we repeat the experiment for true latent dimensions of 5 and 10 and observe similar results. We also evaluate LIT-IVM for + drcee and find that, although the accuracy drops, it drops less when choosing d too high rather than too low. Furthermore, its accuracy still higher than clastic net even with misspecified d.
In Section B.1
7

---

Published in Transactions on Machine Learning Research (09/2025)
4.2 Logistic Regression with Latent Distance
In this experiment, we generate the feature matrix and interaction terms in the same manner as in the linear regression experiment in Section 4.1 For logistic regression, each element of the response vector y is sampled from a Bernoulli distribution with success probability given by logistic( x + xine)))) + T)s, where logistic(z)= = Itt-x. We inject sparsity into the augmented coefficient vector 3 by first generating a dense version gdense. We then set cach clement 3gense to 0 with probability given by the Gaussian function =xp(-()jense))//// /o?), where og is a parameter that adjusts the level of sparsity. This ensures that larger elements in gdense have lower probability of being zeroed out.
1
To generate the dense augmented coefficient vector gdense = [gdenae 0dause), we sample entries for the weight vector gdenze from ogense ~ NO, 1). The interaction matrix 2 @dense is constructed using the latent distance model in equation The matrix e. with entries Ejk ~ *N oi) denotes the deviation from the latent distance model's assumed structure. This formulation captures the interaction through the distances in the latent space bet ween each feature, where the matrix @dense is flattened into O before it is sparsified. We also model noise in the response variable using a Gaussian distribution such that T ~ NN of). We set the variances of the Gaussian distributions to of = 0.0001, og = 0.1, and oz = 0.01.
Results
The results for fixed n 1,000, true latent dimension derue 2, and five different values of p are shown in Figure 26 Similar to the results from the linear regression experiment, we observe a phase transition where the accuracy of elastic net decreases significantly compared to our approach. The phase transition occurs at roughly p = 30, lower than in the linear regression experiment. As continues to increase, our LIT-LVM approach continues to outperform elastic net, with prediction accuracy decreasing much slower than in elastic net.
In Section B.2 we repeat the experiment for true latent dimension of 5 and 10 and observe similar results. We also evaluate the performance of LIT-IVM for misspecified d, with the same findings as for lincar regression: choosing d too high is better than too low. We also demonstrate superior performance of LIT-IVM over elastic net persists in even higher-dimentt dimensionll settings up to p 500.
Comparison with Sparse FMs In Section B.3 we conduct simulation experiments comparing the performance of LIT-IVM with sparse F We simulate data in the same manner except with a low rank structure rather than a latent space structure, to better match the assumption of FMs. Likewise, we use LIT-LVM with a low rank model. We find that FMs perform worse when θ is noisy or exhibits higher sparsity (deviation from exact low rank). This indicates that factorizing the interaction coefficients, as FMs do, can degrade performance when the underlying assumption of exact low rank structure is not met. In contrast, the approximate low rank structure assumption in LIT-LVM offers greater flexibility in modeling the interaction coefficients, ensuring more robustness and higher accuracy.
Effects of LVM Hyperparameter x Next, we investigate the sensitivity of LIT-LVM to the hyperparam- eter λr which controls the strength of the IWM-structured regularization. On one extreme, Å = 0 reduces LIT-LVM to a standard regularized linear predictor, while very large X enforces a strict low-dimensional structure that approaches factorization machines as X - xc.
First, we consider the simulation with true latent dimension d and low noise in the latent variables, where Eja, the deviation from low rank structure, has variance oj 0.1, so that the interaction matrix remains approximately low rank. Figure Ba shows the mean AUC for varying λ while holding the other regularization hyperparameters (A1,)2 fixed. When A = 0, the model can only learn a purely sparse linear predictor with no latent structure penalty, which leads to relatively low predictive accuracy. On the other hand, at extremely high Xe, LIT-LVM approaches a factorization machine, which also underperforms here because it enforces an overly strict exact low-rank assumption. Notably, there is a middle range of X that yiclds the highest mcan AUC, indicating that partially constraining interaction terms to follow a latent structure can improve prediction while allowing some flexibility for deviations.
To avoid differences in performance caused by the optimizer, we implement sparse FMs using LIT-IVM with 000. We show Section C.3 that this value sufficient. (Recall that LIT-IVM approaches a sparse FM in the limit as x + -o.))
8

---

Published in Transactions on Machine Learning Research (09/2025)
My=0.01. Ay=0)
0g=0.01, Au-
0.9
My-0.01. A1-000
Mg-0.01 A
0g 01, A1
M1=0.1.AA=0)
6.70
Clymb.3, A)
0.8
0.7
0.60
0.55
0.5
10
10
21 101
10'
10'
30*
(a) Low latent variable noise: o 0.1
(b) High latent variable noise: ob
Figure 3: Mean AUC for different values of the LVM regularization strength A simulated data experiments using logistic regression. (a) The simulation with low noise (a) = 0.1) peaks in AUC at an intermediate ' confirming the benefit of the IVM regularization term. bb) The simulation with high noise (o) = 4) has
destroyed the latent structure, and thus shows minimal benefit from the LVM regularization.
assess the robustness of LIT-IVM under severe violations of the low-rank assumption, we repeal the same simulation but increase latent variable noise variance to oî = 4, effectively destroying the underlying low-rank structure. As shown in Figure 3b] the best performing A is near or 0.01 which corresponds to applying no little LVM penalty. This is not surprising because at such high noise, the interaction matrix no longer possesses low-rank structure so larger values of d degrade the performance.
5 Real Data Experiments
5.1
OpenML Datasets
We conduct experiments on a wide variety of publicly available datasets from OpenML (Vanschoren et al. 2013 to verify how well our structured regularization for the interaction matrix applies to real data. We consider both regression and binary classification tasks using linear and logistic regression, respectively. We consider 12 regression datasets and 10 classification datasets, described in Section C.I. For each task and dataset, we compare the accuracy of five different predictors, all with an clastic net penalty, listed below in increasing order of flexibility:
Elastic net with no interactions: the base linear predictor (linear or logistic regression) applied directly the features.
2. Sparse factorization machines (FMs): factorized form for the interaction coefficient matrix that assumes exact low rank structure.
3. LIT-LVM: proposed approach that assumes approximate low rank structure for the interaction coefficient matrix.
Hierarchical lasso: includes only interaction terms zyzı where at least one of the features (main effects) Σÿ or 2g is also included. This corresponds to assumption of weak heredity.
Elastic net with interactions: includes interaction terms between all pairs of features.
We divide the data into 50/50 train/test split for both classification and regression tasks. We tune hyperparameters using a grid search, selecting the set of hyperparameters that minimize the mean RMSE for regression tasks and maximize the mean AUC for classification tasks over five splits within the training set. Subsoquently, we evaluate prediction accuracy on the held-out test set. We repeat this process using five different train/tes splits calculate the standard errors for the prediction accuracy. Details on hyperparameter tuning are provided in Section C.1.2

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Mean RMSE + standard error over 5 different train/test splits for OpenML regression datasets. Lower RMSE denotes higher prediction accuracy. Highest accuracy for each data set is denoted in bold, and entries within 1 standard error are denoted in italic. Our proposed LIT-LVM approach has the highest accuracy 11 of the 12 datascts and is within 1 standard error on the 1 remaining dataset.
Elastic Net No Interactions
Elastic Net with Interactions
Hicrarchical
Dataset Name
LIT-LVM
Sparse FM
Lasso
boston
4.99 + 0.12
4.39 + 0.23
4.40 + 0.26
5.01 + 0.63
4.51 + 0.17
bike_see scharing_demann
148.40 + 1.08
144.48 + 1.06
144.47 + 1.05
156.65 + 1.07
144.56 + 0.39
diamonds
0.28 + 0.00
0.26 0.00
0.25 0.00
0.28 + 0.00
0.28 + 0.00
fri_ Sri_c4_500__ 500050001 10
0.88 + 0.01
0.00
1.00
0.88 0.02
medical oodical_chargee charges
0.24 0.00
0.00
0.00
0.20
0.20
nyc_ _yc_taxi_ggr green pol
0.50 0.00
0.00
0.00
0.00
0.27 0.00
30.55 0.08
0.71
24.24 0.63
29.05
28.41 0.06
rmftsa rmftsa_ladata ladata
1.86
1.00
1.03
1.85
6.49
socmob
23.37
17.17
16.55
23.56
17.52 1.05
space_ga
0.14
0.13
+.00
0.10
0.14 + 0.00
superconduct
17.72 0.07
15.17 0.19
14.32 0.07
17.70
18.42 + 0.05
wind
3.30 + 0.01
3.28 + 0.03
3.17 + 0.02
3.28 1288:00 0.01
3.47 + 0.02
Table 2: Mean AUC + standard crror over 5 different train/test splits for OpenML classification datasets. Higher AUC denotes higher prediction accuracy. Highest accuracy for each dała set is denoted in bold, and entries within 1 standard error are denoted in italic. Our proposed LIT-LVM approach has the highest accuracy on 9 of the 10 datasets and is within 1 standard error on the I remaining dataset.
Elastic Net No Interactions
Elastic Net
Hierarchical
Dataset Name
LIT-LVM
Sparse FM
with Interactions
Lasso
Bioresponse
0.791 + 0.013
0.791 + 0.004
0.808 + 0.003
0.801 + 0. 0.004
0.808 + 0.004
clean1
0.900 = 0.003
0.922 + 0.004
0.947 + 0.002
0.900 = 0.005
0.934 = 0.003
clean2
0.944 = 0.001
0.943 = 0.003
0.985 0.001
0.955 = 0.001
0.978 = 0.001
eye_movements
0.586 + 0.004
0.608 + 0.002
0.607 0.002
0.586 = 0.004
0.604 0.005
fri c4 fri__4__000 10004_0000100000
0.666 + 0.011
0.576 + 0.020
0.666 + 0.010
0.651 + 0.012
0.666 + 0.010
fri c4_ fri_c4_1000_100 100
0.672 = 0.009
0.564
0.689 = 0.008
0.674 = 0.014
0.678 = 0.009
hill-valley
0.711 + 0.020
0.547 0.027
0.734 : 0.034
0.699 + 0.023
0.715 + 0.020
jannis
0.814 = 0.000
0.826 + 0.001
0.827 0.001
0.801 + 0.001
0.821 + 0.001
jasmine
0.831 + 0.004
0.836 + 0.003
0.837 + 0.003
0.835 + 0.005
0.832 + 0.003
tecator
0.969 = 0.004
0.959 + 0.006
0.974 0.003
0.970 + 0.004
0.970 = 0.003
The regression and classification results shown in Tables 1 and respectively. Our LIT-LVM approach achieves superior accuracy over the suite of datasets for both tasks, and by a large margin on some datasets, such as ayc_taxi_green for regression and clean_2 for classification. As we observed in the simulation experiments, our proposed LIT-LVM approach achieves the highest gains compared to elastic net with interactions on datasets with high p² /n ratios (shown in Tables and 5 in Section C.1.1), including superconduct for regression and fri_ 1_____4_100______111111111111110 ___10010100 __0 for classification. This also agrees with well-known results for other regularization methods such as elastic net, where the benefit is mainly achieved when the ratio p/n approaches or exceeds 1. With the inclusion of (?) interaction terms, it is the ratio p² /n that is important. We also note that, without the structured regularization of LIT-LVM, interaction terms often hurt the accuracy of lincar predictor, as shown by classification datasots such as hill-valley, where elastic net without interactions far exceeds the accuracy with interactions. Such results were also observed by Bien et al. (2013) in their experiments comparing elastic net without interactions, with interactions, and hicrarchical lasso. We find that the hicrarchical lasso's prediction accuracy is generally in between that of clastic net with and without interactions, as one might expect. The weak heredity assumption provides a stronger regularization compared to elastic net with interactions, which allows hierarchical lasso to perform
Results
10

---

Published in Transactions on Machine Learning Research (09/2023)
1ky-0.0.Au=1.01
Avm1.01
0.93
14=00.01, A--0.11
0.68
-=-001AA-11
14.00.01, Ac-100)
0p-01.
0.92
Aya2.00
0p11A
0.91
0.90
6.60
0.89
0.88
6.54
10
10-2
10:1
10a
10a
162
10a
0
10-
10-2
10:1
100
10
10'
10*
(a) OpenML. dataset clean1
(b) OpenMT. dataset fri_cc__00______
Figure 4: Mean AUC for different values of the IVM regularization strength X on real OpenML classification datasets. a The clean1 dataset peaks in AUC at a moderate X between 0.01 and 0.1, while the more
difficult fri fri____________ _405000__50000 100 dataset benefits from stronger LVM regularization with X in the range of to 100.
better, but not as well as LIT-LVM, on datascts with high p²/n ratios. On all datascts, we find that our proposed optimization procedure for LIT-IVM converges, shown in Section C.2
Additionally, we find that FMs are not competitive with our proposed LIT-LVM approach in accuracy, despite having higher latent dimension up to d = 50 tuned by grid search, compared to LIT-LVM with fixed d = 2. We believe that this is due to its assumption that the interaction terms have an exact low rank structure, rather than an approximate one as in LIT-IVM. comparison of LIT-IVM and FM for varying d is provided in Section
Effects of LVM Hyperparameter \ The effects of X on mean AUC in the clean1 dataset are shown in Figure [a From Table 2 we see that LIT-IVM already attains a substantially higher AUC on clean1 than the other baselines. We observe that moderate values of X again yield the best result, reinforcing that some, but not excessive, emphasis on latent structure
exploit the potential low-rank signal in the data. The performance another data set, fri_c4_5 100 is shown in Figure 4b From Table 2 we see that the mean AUC of FM and LIT-LVM are closer on this dataset than on clean1, which indicates that LIT-LVM is indeed imposing a low-rank structure similar to FM, but less rigid, and thus, still manages to outperform it. Consequently, we observe a higher value for X to maximize AUC compared to clean1. Nevertheless, the overall pattern remains similar: a moderate X still achieves or exceeds the best possible performance from an FM, which is a purely factorized method, or a model with just an elastic net penalty and no factorization.
5.2 Survival Prediction and Compatibility Estimation for Kidney Transplantation
We now turn to a biomedical application that partially motivated the development of our LIT-LVM approach. We aim to predict the outcomes of kidney transplants in terms of the time duration until the transplanted kidney eventually fails. This known as the time to graft failure or survival time. Survival times are influenced by many factors, including the compatibility of the Human Leukocyte Antigens (HLAs) between the donor and recipient (Opelz al. 1999: Foster et al. 2014). HLA compatibility also impacts the accuracy of survival time predictions (Nemati et al. 2023
Data Description This study used daía from the Scientific Registry of Transplant Recipients (SRTR). The SRTR data system includes data all donor, wait-listed candidates, and transplant recipients in the US, submitted by the members of the Organ Procurement and Transplantation Network (OPTN). The Health Resources and Services Administration (HRSA), U.S. Department of Health and Human Services provides oversight to the activities of the OPTN and SRTR contractors.
We follow the inclusion criteria and data preprocessing steps outlined by Nemati al. 2002 on the SRTR kidney transplant data. Each sample includes demographic and clinical information of both donors and recipients, such age, gender, race, and BMT. Additionally, we incorporate data on the HLA types of donors
11

---

Published in Transactions on Machine Learning Research (09/2025)
and recipients. Since it is the compatibility between the HLAs of the donors and recipients that is the important factor influencing survival times, it is the interactions between features representing donor HLAs and recipient HLAs, called HLA pairs, that are of interest. We refer to these HLA pairs using uppercase letters for the donor and lowercase for the recipient, similar to Huang & Xu (0022) e.g., A1_a2 a2 denotes that the donor has HLA-A1 and the recipient has HL.AA2. We further describe the construction of these interaction terms in Section
Our dataset incorporates three specific HLA loci, which can be interpreted groups of features: HLA-A, HLA-B, and HLA-DR. We construct separate interaction matrices for each locus. Furthermore, the interaction matrix considers only interactions between donor and recipient HLA types. This targeted approach enables us to focus on the critical aspect of how donor-recipient HLA interactions impact graft survival time. We describe how our proposed LIT-LVM model can be formulated with such targeted interactions in Section A
Over 70% of the examples are censored, mostly denoting transplants that have not yet failed. Thus, we utilize the Cox Proportional Hazards (Cox PH) model, linear predictor commonly used for survival analysis, rather than standard regression models to account for the censored examples. We divide the transplant data into training and test sets, with a 50/50 split. We select the set of hyperparameters that maximize the mean Cox PH partial log-likelihood using 5-fold cross-validation on the training set. We evaluate prediction accuracy on the test, set using Harrell's concordance index (C-index) (Harrell et al. 1982), which is a ranking-based metric that measures only discrimination ability, and integrated time-dependent Brier score (IBS) (Graf et al. 1999), which measures both discrimination and calibration of predictions.
Results We compare the C-index and IBS of five variants of the Cox PH model: one with an elastic net (EN) penalty, one with our proposed LIT-LVM using a latent distance model, one using PCA to obtain low-dimensional latent, embeddings for reconstructing interaction weights after using the elastic net, and one with sparse FMs. We also provide a Random Survival Forest (RSF) model [Ishwaran et al., 2008) as another reference for the accuracy achievable by a nonlinear predictor that is typically one of the best performers among survival prediction algorithms.
Table 3 shows that LIT-LVM outperforms the all other variants of Cox PH models in dis- crimination and even the much more complex random survival forest. (RSF) on the C-index (0.630 vs. 0.629), yet also achieves the best cal- ibration with the lowest integrated Brier score of 0.143. The discordance for RSF, having the second-best C-index but worst IBS (0.156) is not surprising, as the C-index measures rank- ing only, whereas the Brier score also measures calibration, and tree ensembles such as RSF have been found to suffer from poorer calibra- tion than Cox PH models in other work (Steele)
Table 3: Mean C-index and mean Integrated Brier Score (IBS) = 1 standard error for survival time prediction over 10 train /test splits. Highest accuracy for cach data set is denoted in bold, and entries within standard error are denoted in italic.
Model
Mean C-index
Mean IBS
Cox PH (EN)
0.627 = 0.001
0.149 = 0.000
Cox PH (LIT-LVM)
0.630 + 0.001
0.143 + 0.000
Cox PH (PCA)
0.626 + 0.001
0.149 + 0.000
Cax PH (FM)
0.623 = 0.004
0.149 = 0.000
RSF
0.629 + 0.001
0.156 + 0.000
ct al. 2018). Notably, the 0.006 (== 4%) reduction in IBS that LIT-LVM achieves over the clastic net Cox is an order of magnitude larger than the < 0.001 dillerences among the other Cox PH variants, underscoring a clear calibration advantage. The difference in IBS between LIT-LVM and the other models is further explained in Section C.4.2 by examining the Brier score over time. LIT-IVM is the only model that combines superior discrimination with superior calibration.
HLA Latent Representations One of the primary objectives of this study is to capture latent representation of donor and recipient HLAs to model their compatibilities, as greater compatibility is associated with longer survival times. To achieve this, we visualize HLA compatibilities through their distances the latent space. Initially, we train an clastic net model and used its weights to initialize , in LIT- LVM. To initialize the latent positions for different HLA loci, we use the multidimensionall scaling approach proposed by Huang& & Xul (2022). As comparative baseline, we construct HLA latent representations using
12

---

Published in Transactions on Machine Learning Research (09/2025)
Doner Recipient
Donar Reciplent
0.10
0.08
0.05
0.04
DX2
202460f1
0.00
D42
cnd
DRL
0.00
006
9810
ond
DXI
0.05
2931
2024
-0.02
84403
011424,
D4:2
0.10
002
DALE
0.16
10g
-0.04
10g
DR
DEL
400
-0.06
-0.10
-0.05
0.00
0.05
0.10
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
(a) Latent space representation using LIT-IVM
(b) Latent space representation using PCA
Figure 5: Latent space representation of HLA-DR pairs. Green and magenta dashed lines denote HLA pairs associated with the best (most compatible) and worst (least compatible) outcomes, respectively. In the LIT-LVM latent space, compatible HLA pairs are positioned closely and incompatible pairs are widely separated, whereas PCA fails to preserve this property.
PCA the weights of HLA pairs learned by Cox PH with elastic net regularization, denoted by Cox PH (PCA) in Table 3.
We plot the latent positions of donor and recipient HLA-DRs in Figure 5 and draw dashed lines between the three HLA pairs with the most negative weights and most positive weights, according to the Cox PH model with only elastic net penalty. In the Cox PH model, negative weights are associated with a lower risk of the occurrence of an event, so they are indicative of higher donor-recipient compatibility. Thus, we expect that HLA pairs with the most negative weights should be positioned closely in the latent space, while those with the most positive weights should be positioned farther apart. As shown in Figure 5a the top three pairs with the lowest weights (0DRI7_dri) dr17 = -0.068, ODR7 close together in the latent space, while the top three pairs with the highest weights (ØDRIs_ de7 = 0.044, IDRI = 0.041, IDRIO dr11 = 0.040) are far apart for LIT-LVM. Unlike LIT-LVM, PCA-derived HLA embeddings in Figure 5b fail to place all compatible pairs closely for instance, DR7 dr15, the secoud most compatible pair, are widely separated. Furthermore, the PCA embeddings also place many nodes extremely close together near the origin, which does not provide an informative visualization compared to the LIT-LVM embeddings. Latent space representations for HLA-A and HLA-B are shown in Section C.4.2
0r15_ dr15 = -0.555, ODRIOS dr1 = 0.050) are
Our findings show that our LIT-IVM approach with latent distance model effectively captures the HLA compatibilities that influence kidney graft survival through distances between the latent positions of the HLAs, in addition to achieving prediction accuracy competitive with and superior than more complex ensemble methods such as random survival forest.
6 Conclusion
Our underlying hypothesis in this paper was that the coefficients for interaction terms in linear predictors possess an approximate low-dimensional structure. We imposed this approximate low-dimensional structure on the interaction coefficient matrix by representing each feature using a latent vector and penalizing deviations of the estimated interaction coefficients from their expected values modeled by the latent structure. After a thorough investigation on wide variety of simulated and real datascts, we found that our proposod LIT-LVM approach, which can be viewed as a form of structured regularization specifically for interaction coefficients, generally outperforms elastic net regularization for linear predictors, well factorization
13

---

Published in Transactions on Machine Learning Research (09/2025)
machines. The improvement seems to increase with the ratio p² /n; thus, LIT-LVM seems particularly well suited for moderate and high-dimensional settings.
Limitations In this paper, we limited our investigation to second-order interaction terms. Some dalasets may exhibit higher-order interactions; our proposed approach could potentially be extended to incorporate these, with the interaction coefficient matrix replacod by a higher-order tensor. The approximate low- dimensional structure could then be enforced using tensor factorization approaches (Kuleshov et al. 2015 Our LIT-IVM approach also excludes terms of the form æz., which can be viewed as self interactions and are usually included in polynomial features. These self interactions could be added into LIT-LVM as terms on the diagonal of @ and by choosing a different type of low-dimensional structure that incorporates these terms. Our proposed approach also only scales to thousands of features, it requires estimation of the explicit p x p interaction matrix e Problems with even higher values of p are sometimes called strrahig-dimmnsional problems and require methods such as factorization machines or the penalized interaction estimation (PIE) technique proposed by Wang et al. 22021).
Acknowledgments
Research reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Number R01LM013311 as part of the NSF/NLM Generalizable Data Science Methods for Biomedical Resoarch Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This material is also based upon work supported by the National Science Foundation grant 131-211871..
The data reported here have been supplied by the Hennepin Healthcare Rescarch Institute (HHRT) as the contractor for the Scientifie Registry of Transplant Recipients (SRTR). The interpretation and reporting of these data are the responsibility of the author(s) and in way should be seen as official policy of interpretation by the SRTR the U.S. Government.
References
Kyohei Atarashi, Satoshi Oyama, and Masahito Kurihara. Factorization machines with regularization for sparse feature interactions. Journal of Machine Learning Research, 22(153):1-50, 2021.
Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. The Annals of Statistics, 13311111111,11, 2013.
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. Higher-order factorization machines. In Advances in Neural Information Processing Systems, volume 29, 2016.
Emmanuel Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM, 58(3):1 1-73 2011.
Nam Hee Choi, William Li, and Ji Zhu. Variable selection with the strong heredity constraint and its oracle property. Journal of the American Statistical Association, 105(489):354 364, 2010.
David Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Method- ological), 34(2):187 202, 1972.
Abhimanyu Dubey, Filip Radenovic, and Dhruv Mahajan. Scalable interpretability via polynomials. In Advances in neurol information processing systems, volume 35, pp. 36748-36766 2022.
BJ Foster, M Dahhou, X Zhang, RW Platt, JM Smith, and JA Hanley. Impact of HLA mismatch at first kidncy transplant on lifetime with graft function in young recipients. American Journal of Transplantation, 14(4):876-88) 885, 2014.
Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, 18(17-18):2529 254445555 1999.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Frank E Harrell, Robert M Califf, David B Pryor, Kerry L Lee, and Robert A Rosati. Evaluating the yield of medical tests. JAMA, 247(18))2543-22 -24454466666, 1982.
Peter Hoff, Adrian Raftery, and Mark Handcock. Latent space approaches to social network analysis. Journal of the American Statistical Association, 97(460):1090-0995 2002.
Zhipeng Huang and Kevin S Xu. latent space model for HLA compatibility networks in kidney trans- plantation. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine, pp. 1020-1027, 2022.
Zhipeng Huang, Hadeel Soliman, Subhadoep Paul, and Kevin S Xu. A mutually exciting latent space Hawkes process model for continuous-lime networks. In Proceedings of the 38th Conference on Uncerlainly in Artificial Intelligence, pp. 863-873. 2022.
Hemant Ishwaran, Udaya B Kogalur, Eugene H Blackstone, and Michael S Lauer. Random survival forests. The Annals of Applied Statistics, pp. 841-860, 2008.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations, 2015.
Volodymyr Kuleshov, Arun Chaganty, and Percy Liang. Tensor factorization via matrix factorization. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, pp. 507 516, 2015.
Danicl D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, (01(6755) 7918,7911111 1999.
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.
2013.
Nikolaos Nakis, Abdulkadir Celikkanat, Louis Boucherie, Christian Djurhuus, Felix Burmester, Daniel Mathias Holmelund, Monika Frolcová, and Morten Mørup. Characterizing polarization in social networks using the signed relational latent distance model. In Proceedings of the 26th International Conference on Artificial Intelligence and Statislics, pp. 11489-115000 2023.
Mohammadreza Nemati, Haonan Zhang, Michael Sloma, Dulat Bekbolsynov, Hong Wang, Stanislaw Step- kowski, and Kevin S Xu. Predicting kidney transplant survival using multiple feature representations for hlas. Artificial Intelligence in Medicine, 145:102675, 2023.
G Opelz, T Wujeiak, Döhler, Scherer, and J Mytilincos. HLA compatibility and organ transplant survival. Collaborative transplant study. Reviews in Immunogenetics,, 1(3):334 342, 1999.
Zhen Pan, Enhong Chen, Qi Liu, Tong Xu, Haiping Ma, and Hongjic Lin. Sparse factorization machines for click-through rate prediction. In Proceedings of the 16th IEEE International Conference on Data Mining, pp. 400-409, 2016.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in Optimázation, 1(3): 127-239, 2014.
Peter Radchenko and Gareth M. James. Variable selection using adaptive nonlinear interaction structures high dimensions. Journal of the American Statistical Association, 105(492):1541 1553, 2010.
Steffen Rendle. Factorization machines. In Proceedings of the IEEE International Conference on Data Mining, pp. 995-1000, 2010.
Steffen Rendle. Factorization machines with libFM. ACM Transactions on Intelligent Systems and Technology, 3(3):1-22, 2012.
Steffen Rendle. Scaling factorization machines to relational data. Proceedings of the VLDB Endowment, 6(5): 337-348, 2013.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Emile Richard, Pierre-Andre Savalle, and Nicolas Vayatis. Estimation of simultaneously sparse and low rank
matrices. In Proceedingg of the 29th International Conference on Machine Learning, pp. 51-58, 2012.
David Rügamer. Scalable higher-order tensor product spline models. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, pp. 1-9, 2024.
David Rügamer, Andreas Bender, Simon Wiegrebe, Daniel Racek, Bernd Bischl, Christian L Müller, and Clemens Stachl. Factorized structured regression for large-scale varying cocfficient models. In Proceedings of the Joinl European Conference on Machine Learning and Knowledge Discovery in Dalabases, pp. 20-35, 35, 2022.
Andrew J Steele, Spiros C Denaxas, Anoop D Shah, Harry Hemingway, and Nicholas M Tuscombe. Machine learning models in electronic health records can outperform conventional survival models for predicting palient mortality in coronary artery disease. PLOS ONE, 1300):4020 2018.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Sociely; Series B (Methodological), 58(1):267-255 1996.
Joaquin Vanschoren, Jan N van Riju, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. SIGKDD Explorations, 15(2):49-60. 2013.
Cheng Wang, Binyan Jiang, and Liping Zhu. Penalized interaction estimation for ultrahigh dimensional quadratic regression. Statistica Sinica, (1):ppp. 1549-1570, 2021.
Jianpeng Xu, Kaixiang Lin, Pang-Ning Tan, and Jiayu Zhou. Synergies that matter: Efficient interaction selection via sparse factorization machine. In Proceedings of the SIAM International Conference on Data Mining, pp. 108-116, 2016.
Ming Yuan, V. Roshan Joseph, and Hui Zou, Structured variable selection and estimation. The Annals of Applied Statistics, 3(4):1738-175) 2009.
Peng Zhao, Guilherme Rocha, and Bin Yu. The composite absolute penalties family for grouped and hierarchical variable selection. The Annals of Statistics, 36(66):3448-334) -4988, 2009.
Ke Zhou, Hongyuan Zha, and Le Song. Learning social infectivity in sparse low-rank networks using multi- dimensional Hawkes processes. In Proceedings of the 16th International Conference Artificial Intelligence and Statistics, pp. 641-649, 2013.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301 320, 2005.
A Additional Model Information
A.1 Targeted Interactions
Inclusion of the interaction terms between all the features might not be well suited given the conditions and the domain pertinent to a specific application. Instead, we may be interested in incorporating the interaction between a first set of features X C with dimension of and sccond set of features X" S X with dimension of g. We refer to these as largeled interactions, which may come from domain knowledge or specific interest in set of interactions. With targeted interactions, the interaction matrix O changes from a square p x p to possibly rectangular x g matrix representing the coefficients for the interaction terms between the features of x'x"". We denote the latent representation matrices of the two sets of features x" as Z Znxd respectively.
To account for targeted interactions, it crucial to incorporate a method that selectively prevents updates the model components representing these non-existent interactions. This selective update implemented using a mask matrix M, which comprises elements valued at or 1. The value of 0 is assigned to positions
16

---

Published in Transactions on Machine Learning Research (09/2025)
corresponding to non-interacting feature pairs, and is assigned elsewhere. The use of this mask effectively ensures that updates during the training process are only applied to the relevant elements of the interaction weights and their corresponding latent representations.
Implementing a mask matrix provides a controlled mechanism for training a model while preserving the integrity of the domain knowledge within the model's structure, Tt helps in maintaining the sparsity of the interaction matrix O and the latent representation matrices Z and Z", thus preventing the model from fitting to spurious interactions that not of interest. This is particularly crucial when the dimensionality of the feature space is high, where there is a risk of overfitting spurious interactions. Accordingly, the loss functions for the low-rank and latent distance models are modified to incorporate the mask matrix, ensuring that only the targeted interactions contribute to the loss calculation.
A.2 Model Complexity
Our LIT-LVM model is parameterized by the (p+ 1)-dimensional coefficient vector 3, the x p interaction matrix ④ and the x matrix of latent positions Z. Since the latent dimension d < p, there are a total of O(p²) parameters, which the same number of paramcters that a usual lincar predictor with interaction terms would have. On the other hand, FMs store only factorized representation of using ZZ", so they have only a total of O(pd) parameters, resulting in a smaller trained model.
The time complexity of our estimation procedure for LIT-LVM per epoch is ((np²=pp'0 = ((pp²) (mpp) np² dominates pdd. This is the same O(np²) time complexity usual linear predictor with interaction terms, there is also no asymptotic increase in computation. FMs can again achieve lower ((ppd) time complexity due to their lower number of model parameters.
B Additional Details and Results on Simulation Experiments
B.1 Linear Regression with Low Rank Model
We repeat the experiment from Section for true latent dimensions of and 10, with results shown in Figure 6 The same trends as in Figure 2a can be observed, with the phase transition around p 50.
Next, compare results for different values of d, with results shown in Figure 7 For each value of drrne the lowest RMSE is achieved for the correctly specified d = dhrues one would expect. When dune 5, choosing d = 2, which is too small, takes away most of the gain achieved compared to just using elastic net, while choosing d = 10, which is too high, still results in an tremendous improvement compared to elastic net, although less than for d = derue 5. This suggests that, when in doubt, it is better to choose a higher rather than lower dimensional latent representation.
B.2 Logistic Regression with Latent Distance Model
The results when we repeat the experiment from Section 4.2 across different values of durue are shown in Figure g Again, we see superior prediction accuracy for LIT-LVM, although the improvement is lesser higher values of dirue-
Next, compare our results for different values of d, with results shown in Figure For each value of deeues the highest AUC is achieved for the correctly specified d = derue. as one would expect. When durue 5, There is still an improverent compared to elastic net even with the misspecified d = 2 and d d= 10. Again, choosing d too high rather than too low results higher accuracy. The same trends can be observed for derue = 2 and dirue 10.
Finally, we explore what happens when we let p grow an extremely high value so that (2) > n. The results up to p = 500, for which (g) 124,750 with n 1,000 are shown in Figure 10 Notice that LIT-IVM continues to outperform elastic net until the point where all of the methods are barely better than random guess at p 500.
17

---

Published in Transactions on Machine Learning Research (09/2025)
6:10 elastic
elactic
10a
Vean PASE Dky triee
Moan (Ma stale)
10"
10a
10
10
p:40
p:50
p:60
p:70
p:40
p:50
p:60
p.70
(a) dirue 5
(b) derue 10
Figure 6: Results of simulation experiments on linear regression for different values of the true latent dimension dirue. Lower RMSE denotes higher prediction accuracy. Results for duree = 2 are shown in Figure 2a Error bars denote one standard error.
d:10
0:10
10A
Nown ROSS thg Caler
Non IYGE log scll
10a
10-1
P:40
p:50
p:60
p:70
p:40
p:50
p:60
P:70
dreue
(b) dirua
10P
d:5
d:10
net
10a
10
P:40
p:50
(c) derue
Figure Results of simulation experiments on linear regression for different values of the chosen latent dimension d and true latent dimension darue- Lower RMSE denotes higher prediction accuracy. Error bars denote one standard error.
18

---

Published in Transactions on Machine Learning Research (09/2025)
clastic net
d:10
dastic
Net AUC
Non ALC
0.5
0.5
p:20
p:30
p:40
p:50
p:60
p:20
p:30
p:40
p:50
p:60
dirue
5
(b) derue 10
Figure 8: Results of simulation experiments logistic regression for different values of the true latent dimension dr Higher AUC denotes higher prediction accuracy. Results for derue 2 are shown in Figure Error bars denote one standard error.
2b
1.0
elastic
1.0
0.9
Mac.JC
Vtanh
p:30
dirue
(b) derue
0:2
d:10
1.0
elsttic
0.9
0.8
0.7
0.6
0.5
p.20
p:30
p:40
p:50
p:60
(c) derue 10
Figure 9: Results of simulation experiments on logistic regression for different values of the chosen latent dimension d and true latent dimension drrue- Higher AUC denotes higher prediction accuracy. Error bars denote one standard error.
19

---

Published in Transactions on Machine Learning Research (09/202))
1.0
d:2
0.9
d:5
d:10
Mean AUC
0.8
elastic net
0.6
0.5
0.4
p:100
p:200
p:500
Figure 10: Performance comparison of LIT-INM vs. elastic net on logistic regression simulation in an extremely high dimension sctting. Error bars denote one standard error. LIT-LVM consistently outperforms elastic net up to 500.
1.00
0.98
Mean AUC
0.96
0.94
d:2
0.92
d:5
d:10
0.90
p:20, FM
LIT_LVM LVM
p:30, FM
p:30. 033,, LIT LVT LVM
p:40, LIT LVM
p:50, FM
Pe90 0500, LIT LVM I
p:60, LIT LIT_LVM LVM
p:20, LIT
p:40,
p:60,
Figure 11: Mean AUC comparison between LIT-LVM and FMs in classification simulation experiments. Error bars denote one standard error. The feature matrix and weights are generated as described in Section 4.2
B.3 LIT-LVM vs. Sparse Factorization Machines (FMs)
In this simulation study, we evaluate the performance of sparse FMs compared to LIT-LVM through series of classification experiments under varied conditions such increased noise and sparsity injected in the interaction matrix O These conditions potentially affect the predictive accuracy of models like FMs that rely exact. low-rank structures. By introducing greater noise variance and higher sparsity levels in the interaction weights during the simulations, we aim to demonstrate how each model handles deviations from the exact low-rank assumption, thereby providing insights into their practical use scenarios where this assumption does not hold. Since we are interested in the accuracy of the FM and the computation time, we formulate FMs as a special case of LIT-LVM with extremely large 100, 000. (The model recovered LIT-LVM is equivalent to that of FM in the limit X
critical for assessing the performance of the models, they
discussed in Section 3.3
Figure 11 displays the mean AUC results for both models under bascline noise and sparsity conditions, while Figures 12 and 13 show the results when the models are subjected to increased noise levels and greater sparsity. Figure 12 shows the effect of increasing the noise variance to 4 on the performance of the models, highlighting significant drop in the accuracy of FMs due to their reliance on an exact low-rank structure of the interaction weight matrix. Similarly, Figure [3 demonstrates how increasing the sparsity level in the interaction weights impacts the performance of FMs compared to LIT-LVM. These experiments underscore
20

---

Published in Transactions on Machine Learning Research (09/2025)
1.00
d:2
6:10
0.90
Mean AUC
0.85
0.75
0.70
0.65
p:20, FM
p:20, LIT LVM
p:30, FM
Why LI7 voey pad
p:40, FM
p:40, LIT LVM
p:30, FM
p:50, LIT LVM
p:60, FM
p:60, LIT LVM
Figure 12: Mean AUC comparison between LIT-LVM and FMs in classification simulation experiments Error bars denote one standard error. The feature matrix is generated as described in Section 4.2 with increasod noise variance (set to 4) addod to the true interaction coefficients O Due this added noise, the performance of FMs, which assume an exact low-rank structure for the interaction coefficient matrix, drops significantly compared to their performance in Figure 11. where the noise was lower.
1.000
d:2
d:10
0.975
0.950
0.925
Mean AUC
0.900
0.875
0.050
0.625
0.800
p:20, FM
p:20, LIT-LVM
9:30. FM
p:30. LIF-VMM
p:40, FM
p:40, LIT-LVM
p:50, FM
p:50, LIT-LVM
p:60, FM
p:60, LIT-LVM
Figure 13: Mean AUC comparison between LIT-LVM and FMs in classification simulation experiments. Error bars denote one standard error. The feature matrix is generated as described in Section 422 with increased sparsity level imposed on interaction coefficients O. Due to this addod sparsity, the performance of FMs, which assume an exact low-rank structure for the interaction coelficient matrix, drops significantly compared to their performance in Figure 11 where the sparsity level was lower.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Regression datasets from OpenML used for real data experiments, comprising a wide range of number of samples n and features p. Datasets with slash in the number of features p have categorical variables, so the entry denotes before before/after after one-hot encoding. The ratio p² /n is computed using the number of features after onc-hot encoding. The OpenML dataset ID column includes link to the dataset.
Datasct Name
# of Samples n
# of Features p
p²/n
OpenML Datasct ID
boston bike leatriingddmadd
506
14/22
0.957
531
1,737
6
0.0207
44142
diamonds fri _0__4_500__1_
53,940
10
0.00185
44140
500
10
0.200
604
pedical_chargee
163,065
12
0.000883
441 46
ayc-taxi-green-dec
581,835
9
0.000139
441 43
pol
15,000
26
0.0451
44134
rmftsa emftsa_ladata ladata
508
10
0.196
666
socmob
1,156
6/39
1.32
541
space_ga
3,106
7
0.158
507
superconduct
21,263
79
0.294
44148
wind
6,574
15
0.0342
503
Table 5: Classification dalasets from OpenMI, used for real data experiments, comprising a wide range of number of samples n and features The OpenML dataset ID column includes a link to the dataset.
Dataset Name
of Samples n
# of Features p
p²/n
OpenML Dataset ID
Bioresponse
3,434
420
51.4
45019
clean1
476
169
60.0
40665
clean2
6598
169
4.33
40666
aye_movements
7,698
24
0.0748
44157
Fri__c__000_10 100
500
101
20.4
742
[___c4_1000_110
1,000
101
10.2
718
hill-valley
1,212
101
8.41
1479
jannis
57,580
55
0.0525
43977
jasmine
2,984
145
7.05
45057
tecator
240
125
65.1
851
the robustness of LIT-LVM under more challenging conditions where assumptions of the exact low-rank structures in FMs do not hold.
C
Additional Details and Results on Real Data Experiments
C.1 OpenML Datasets
In our experiments, we utilize a diverse set of benchmark datasets to evaluate the performance of the proposed model. These datasets are chosen to ensure comprehensive coverage of different types data distributions and characteristics. By using these datasets, we aim to thoroughly validate the performance and generalization ability of our model across both classification and regression tasks.
C.1.1 Data Description
We employ a range of datasets as detailed in Table | for regression and Table 5 for classification. These datascts are selected to test the model's ability to handle different scenarios. The diversity in the number of samples and features ensures that our model is evaluated on wide spectrum of regression and classification challenges. Links to these datasets also provided for of access.
22

---

Published in Transactions on Machine Learning Research (09/202))
0.75
Model Type
LIT-VMM
0.65
0.60
0.50
10
30
50
Low-rank Dimensian (d)
Figure 14: Performance comparison of LIT-LVM vs. FM fri c4_ 10011000_110 100 classification dataset with different values for the dimension d of the latent representation. LIT-LVM outperforms FM for all values of d.
C.1.2 Hyperparameter Tuning
We ensure that our proposed LIT-LVM approach, clastic net, and FMs are initialized with identical weights, pro- viding a fair comparison. The regularization parameter À for the structured regularization in LIT-IVM is cho- sen from [0.01, 0.1, 1, 10, 100], and the elastic net parameters x and 22 are chosen from 0.00, 0.1, 1, 10, 100]. For LIT-LVM, we choose fixed dimension d unless otherwise specified. For FMs, we choose d using grid search over [2, 10, 25, 50], as its assumption of exact rather than approximate low rank structure may require higher d.
We use the hierNet Rt package implementing hierarchical lasso (Bien et al. 2013 We tune its regularization parameter over the same grid [0.01, 0.1, 1, 100] as for the other methods while leaving the & penalty parameter fixed at 10-8,, as the authors recommend not to tune this parameter. To match the other methods, we consider only interaction terms and exclude the "self-interactions" (entries on the diagonal of the interaction coefficient matrix).
C.1.3 Comparison of LIT-LVM with FMs
The comparison between FM and LIT-LVM models in Figure EogureI4sshrn shows that LIT-LVM consistently outperforms FM across all values of the dimension d of the latent representation, with largest performance gap observed al d= and d d= 5. This suggests that LIT-LVM's structured regularization is more effective in estimating interactions, particularly in lower-dimensional latent spaces. FM demonstrates gradual improvement with increasing dimensionality, achieving its best performance at d 50. This suggests that FMs better approximate the full interaction matrix when the d is larger. This is not surprising because FMs assume an exact factorization of the interaction matrix θ which is likely not exactly low rank in real data sets such as 100. On the other hand, LIT-LVM's assumption of approximate low rank allows to better represent the interaction matrix using lower dimensions such as 3 or = 5.
C.2 Convergence of Optimization Procedure
To complement our empirical evaluations, we analyze the training dynamics of our model across various datasets. Specifically, we plot the training and validation loss curves over epochs in Figure 15 showing consistent and stable convergence of our optimization procedure.
23

---

Published in Transactions on Machine Learning Research (09/2025)
gm
04-
50000
Figure 15: Training and validation loss curves over epochs across different OpenML classification datasets. Each plot shows steady convergence, providing empirical evidence of stable optimization.
96.0
Mean
95.6
95.2
95.0
10a
109
107
10F
Figure 16: Prediction AUC on the clean2 dataset as X increases beyond 10a The curve is nearly flat, indicating that LIT-LVM behaves like a sparse FM at large At-
24

---

Published in Transactions on Machine Learning Research (09/2025)
C.3 Fitting Sparse Factorization Machine Models using LIT-LVM
To further validate that LIT-LVM with large A approximales the behavior of sparse lactorization machine (FM), we conduct an additional experiment on the clean2 dataset. Specifically, we increase A values beyond the 10 value that we use in carlier experiments and mcasure its effect on prediction accuracy. As shown in Figure 16 the AUC remains nearly constant beyond this threshold, confirming that the LIT-LVM model with d = 10 is sufficiently large to represent a sparse FM model.
C.4 Survival Prediction and Compatibility Estimation for Kidney Transplantation
C.4.1 Data Preprocessing
For this study, represent each HLA a categorical feature which we call an HLA type. We define HLA pairs to represent the interactions between donor and recipient HLA types. The HLA type and pair features are encoded using the approach of Nemati et al. 2023) which we summarize in the following.
HLA types are encoded using a one-hot-like scheme, producing binary variables such as DON_A1, DON DON_A2, [EC_A1, and REC_ _2c__2. The binary variable set to one the donor or recipient possesses the corresponding HLA type. We refer to the encoding as one-hot-like because some HLA types are splits of a broad HLA type. For example, A23 is a split of A9, so that each occurrence of A23 in a donor is encoded with a one in both columns DON A23 and DON_A9.
HLA pairs are similarly represented with binary one-hot-like variables such as DON DON_A1 etc., again accounting for broads and splits. A value of one in a particular column indicates that both the donor and recipient possess the specified HLA types, and the donor-recipient HLA pair is biologically relevant. Some HLA pairs are not biologically relevant due to the asymmetry between donors and recipients if a donor possesses an HLA type that is not in the recipient, then the recipient's immune system may reject the transplant, so the HLA pair is biologically relevant for evaluating donor-recipicnt compatibility. On the other hand, if recipient possesses HLA type that is not present in the donor, it does not create a problem for the recipient, so the HLA pair is not biologically relevant, and we do not place a one in the column for that HLA pair. We refer interested readers to Nemati et al. (2023) for further details and illustrative examples of the encodings.
A1, JON_A1_ MA_RECCAA _2,
C.4.2 Additional Results
A plot of Brier score over time is shown in Figure 17 Within a short period of time just under 1 year, RSF already shows worse Brier scores than the other approaches and has worse calibration over the entire time horizon. The Cox PH variants except for LIT-IVM all have roughly equal Brier scores. Beyond about 5 years, LIT-LVM shows noticeable improvement compared to the other Cox PH models, and this improvement persists over the entire time horizon, resulting in the lower integrated Bricr score for LIT-IVM shown in Table 3
HLA Latent Representations The latent representations for HLA-A and HLA-B are shown in Figure 18. We make similar observations to those for HLA-DR in Figure 5 First, notice that the PCA embeddings for HLA-A and HLA-B tend to place almost all of the HLAs together around the origin, even more so than for HLA-DR. This decreases the interpretability of the embedding, especially compared to the LIT-LVM embedding, where the HLAS are more spaced apart. Furthermore, the distances between low compatibility and high compatibility are also not well preserved the PCA embedding compared the LIT-IVM embedding. For HLA-A, the most compatible pairs are (0AGs_a2 -0.053, 0 -0.052, 0 022_a68 -0.050), which are placed in the vicinity of each other, while the least compatible pairs are (0A32_a72 0.050, 0AI_03 0.044, A a3 0.040), which are placed far from each other. Similarly, for HLA-B, the most compatible pairs are (OBs 660 -0.010, 0358 bs = -0.006, lBss bas -0.005), which are placed in the vicinity of each other, while the least compatible pairs are (Oneo_Lac has = 0.007, Ops5_ 639 = 0.007, O358_b72 672 0.005), which are placed far from each other.
25

---

Published in Transactions on Machine Learning Research (09/2025)
LIT-LVM
Cox PH
PCA-Cox
Factorization Machine
RSF
Brier Score
0.00
Years
Figure 17: Comparison of Brier scores 30 time points. LIT-IVM achieves the best calibration, indicated by the lowest Brier score. The improvement is greatest at high survival times, beyond the median survival time of about 10 years.
26

---

Published in Transactions on Machine Learning Research (09/2025)
0.88
0.10
Denor Reciplent
Reciplent
0.6
0.38
0.06
04
A3
0.04
0.02
12
0.02
13%
0.00
A23422
26
A28
A30
MP
A2d
-0.06
-0.015
(a) HLA-A latent space represenlation using LIT-LVM
(b) HLA-A latent space representation using PCA
1393%
0.08
Doner Recipient
2530l
0.04
104
0.02
154
0.02
14
52g
02
k5%
60
-0.02
012
02/
1392g
15g
000
15.P
Doner Decipent
0.36
-0.04
0.004
0.00
0.04
0.06
(c) HLA-B latent space representation using LIT-LVM
(d) HLA-B latent space representation using PCA
Figure 18: HI.A-A and HLAAB latent positions from LIT-IVM and PCA, highlighting the top three most compatible and least compatible HLA pairs according to the elastic net penalized Cox PH model's coefficients. Green dashed lines denote the most compatible pairs, and magenta dashed lines denote least compatible pairs. In LIT-LVM, the most and least compatible pairs are placed close together and far apart, respectively, while the PCA latent spaces fail preserve this property.
27


Reviews:
[{"review_id": "k262483iHM", "reviewer": "Reviewer_bGaA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed latent space representations are more flexible than those considered in the literature, and seem to perform well on a wide variety of datasets.\n\n**Weaknesses**\n\nMy major concerns are about the interpretation of the latent distance model, tuning of hyperparameters, and choice of methods for comparison:\n\n1) While the low-rank model (4) is relatively straightforward to interpret, the latent distance model (5) is more difficult to understand. Why does the distance have a negative effect on $\\theta_{jk}$? If two features are far apart in the latent space, the relationship implies that they should have a large negative interaction effect. Why is the negative rather than the positive direction preferred? How should one choose the value of $\\alpha_0$? The paper should develop some intuition and give examples of applications where this model seems plausible.\n\n2) The proposed optimization problem involves many hyperparameters, including the regularization parameters ($\\lambda_1$, $\\lambda_2$, and $\\lambda_l$) and the latent dimension $d$. Since a fully data-driven method for hyperparameter tuning would be prohibitively expensive, the paper should provide more guidance and heuristics for selecting the hyperparameters.\n\n3) The paper cited only Lou et al. (2013) for sparse modeling of interaction terms. In fact, there are a much larger literature on sparse interaction models in regression. Some early contributions include Zhao et al. (2009), Yuan et al. (2009), Choi et al. (2010), and Radchenko and James (2010). More recent methods allow faster implementations that scale up to 10,000 features (e.g., Wang et al., 2021). These methods should be carefully reviewed and included for numerical comparisons, in addition to the elastic net which does not explicitly model the interaction structures.\n\n**References**\n\nPeng Zhao, Guilherme Rocha, and Bin Yu. The composite absolute penalties family for grouped and hierarchical variable selection. *The Annals of Statistics*, 37(6A):3468-3497, 2009.\n\nMing Yuan, V. Roshan Joseph, and Hui Zou. Structured variable selection and estimation. *The Annals of Applied Statistics*, 3(4):1738-1757, 2009.\n\nNam Hee Choi, William Li, and Ji Zhu. Variable selection with the strong heredity constraint and its oracle property. *Journal of the American Statistical Association*, 105(489):354-364, 2010.\n\nPeter Radchenko and Gareth M. James. Variable selection using adaptive nonlinear interaction structures in high dimensions. *Journal of the American Statistical Association*, 105(492):1541-1553, 2010.\n\nCheng Wang, Binyan Jiang, and Liping Zhu. Penalized interaction estimation for ultrahigh dimensional quadratic regression. *Statistica Sinica*, 31(3):1549-1570, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a structural regularization method for simultaneously estimating the main effects and interaction effects in regression. The idea is to assume that the interaction coefficient matrix has an approximate low-rank structure, which can be captured by a low-rank model or a latent distance model. The paper formulated the optimization problem and conducted experiments on simulated and real data to validate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See comments above.\n\nMinor issue:\n\nBeginning of Introduction, “Linear predictors such as linear regression and logistic regression …”: predictors are variables, not models; please rephrase.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0hi32bFQac", "reviewer": "Reviewer_zEr4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- **Effective Overfitting Mitigation in High-Dimensional Settings**: LIT-LVM is designed to further mitigate overfitting in high-dimensional environments, especially when the number of interaction terms (on the order of $p^2$) is large compared to the number of samples ($n$). This structured regularization is particularly beneficial as the ratio $p^2/n$ increases.\n- **Broad Applicability to Various Linear Predictors**: The LIT-LVM framework is applicable to many types of linear predictors, including linear regression, logistic regression, and the Cox Proportional Hazards (Cox PH) model, and can be used in conjunction with standard regularization techniques like elastic net.\n- **Greater Modeling Flexibility due to Approximate Structure**: Unlike Factorization Machines (FMs), which assume an exact low-rank structure for interaction coefficients, LIT-LVM hypothesizes an approximate low-dimensional structure, penalizing deviations from it.\n- **Superior Prediction Accuracy**: The approach consistently demonstrates superior prediction accuracy compared to traditional methods like elastic net and factorization machines (FMs) across a wide range of simulated and real datasets for both regression and classification tasks. This improvement is notable, especially as feature dimensions increase.\n- **Provides Interpretable Low-Dimensional Latent Representations**: LIT-LVM offers a significant advantage by providing low-dimensional latent representations for features. These representations are useful for visualizing and analyzing relationships between features. For example, in the kidney transplantation study, LIT-LVM's latent space effectively positioned compatible HLA pairs closer and incompatible pairs farther apart, offering valuable interpretability that PCA failed to achieve.\n\n### Weaknesses\n\n**[Low-Dimensional Assumption]**\n\n- The assumption that “this matrix has an approximate low-dimensional structure” (Section 1) would benefit from clarification regarding how it corresponds to specific real-world or hypothetical scenarios.\n\n**[Non-Convexity]**\n\n- Methods that involve latent variable models and approximate low-dimensional structures, such as the LIT-LVM approach described, often lead to non-convex optimization problems. Please clarify whether the proposed method falls into this category.\n- If it is non-convex, I have concerns about the following point. The paper mentions that the optimization is performed using Adam. While this is one of the standard optimization algorithms for such problems, the manuscript does not discuss the implications of this non-convexity, nor does it provide any theoretical guarantees or experimental analyses regarding the convergence of the optimization procedure. Since It is unclear how stable and optimizable the proposed method is, this is a significant omission for a method involving a non-convex objective.\n\n**[Flexibility vs. Sparse FMs]**\n\n- There is an inconsistent argument about flexibility. Initially, LIT-LVM is framed as preventing overfitting via structured regularization to counteract the high flexibility of interaction terms. However, in Section 3.3 and the comparison with Sparse FM, it is claimed that LIT-LVM outperforms due to its greater modeling flexibility. Please reconcile these two perspectives.\n- In addition to the above, the proposed method and its clear competitor, Sparse FMs, are not compared in numerical experiments.\n- In Section 5.2, there is a description of a comparison with Sparse FMs, but the label in Table 3 is Cox PH (FM).\n- In any case, providing a consistent and comprehensive comparison against Sparse FMs across all experiments would strengthen the persuasiveness of your claims.\n\n**[Robustness]**\n\n- The numerical experiments in Section 4 and Appendix B evaluate the proposed method using a discrete set of latent dimensions (specifically,  $d=2,5,10$). If $d$ is continuously varied over the natural numbers, does the proposed method still identify the optimal dimension and yield the best results\n- Can the proposed method maintain high performance compared to the comparative methods even in situations of model misspecification (e.g., when the interaction term is not $x_jx_k$)?\n\n**[Reproducibility]**\n\n- I attempted to reproduce the numerical experiments using the provided GitHub repository, but there were no usage instructions.\n- Could you add the necessary information for execution, such as `README.md` and `requirements.txt`? (Personally, I would appreciate it if you could add it with Pipenv.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel regularization approach (LIT-LVM) designed to accurately estimate coefficients for interaction terms in linear predictors. LIT-LVM is founded on the hypothesis that the coefficients for different interaction terms possess an approximate low-dimensional structure, and it models this by representing each feature with a latent vector in a low-dimensional space. By incorporating a latent variable model term into its total loss function, which penalizes deviations from this hypothesized latent structure, LIT-LVM achieves superior prediction accuracy compared to traditional elastic net regularization and Factorization Machines (FMs). Furthermore, it provides interpretable low-dimensional latent representations of features, which are valuable for visualizing and analyzing relationships, as demonstrated in applications such as modeling donor-recipient compatibility in kidney transplantation. The method's robustness to noisy or non-exactly low-rank interaction structures, in contrast to the exact low-rank assumption of FMs, contributes to its enhanced performance on real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on TMLR's acceptance criteria, I have an overall favorable opinion of this paper. However, as noted in the Weaknesses section, I also have several concerns that need to be addressed. To ensure eventual acceptance, I would like to ask your views on these concerns, including appropriate revisions to the paper.\n\n[Minor Comments]\n\n- “such that so that” in the first paragraph of Section 2.2.3;\n- $\\theta_{\\theta}^2$ in the caption of Figure 3;\n- “0.0.004” at (‘space_ga’, ‘FM’) in Table 1;\n- $\\mathrm{logistic}(\\beta^{\\top}x+\\theta_{\\mathrm{flat}}^{\\top}x_{\\mathrm{int}}) + \\eta)$ in the first paragraph of Section 4.2.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Etuvc7Paws", "reviewer": "Reviewer_QNQs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(1)The proposed model directly tackles the p^2>n problem in linear interaction model by softly shrinking the interaction matrix toward a low-rank or latent-distance form rather than enforcing an exact factorisation.\n\n(2)It works with linear, logistic and survival objectives and uses an Adam-plus-proximal optimiser that avoids any SVDs while keeping per-epoch cost at O(n*p^2), which does not seem to be too expensive for big p.\n\n(3)The learned latent vectors create interpretable maps (e.g., HLA compatibility clusters), and the authors provide code and hyper-parameter grids to aid reproducibility.\n\nWeaknesses\n(1) Model quality hinges on four hyper-parameters (\\lambda_1,\\lambda_2,\\lambda_\\ell,d), yet the paper offers minimal tuning guidance.\n\n(2) If the true interaction matrix is near full-rank (like network-type covariate), the latent penalty may bias estimates, and robustness is demonstrated only on synthetic data. I do not see how to avoid this. In this scenario, the framework handles pairwise interactions only, leaving higher-order effects unexplored (which is pretty common in ANOVA type models).\n\n(3) See requested changes, for the detailed explanation for the weakness in experimental part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors look at linear models enhanced with pair-wise interaction features (product terms like x_i*x_j).\nThose interactions quickly explode in the number of features p give p(p-1)/2 extra coefficients—so even strong sparsity penalties such as the lasso or elastic-net start to overfit when p^2\\geq n.\n\nTheir key observation is that the matrix of interaction weights x_i*x_j for  (which is a p*p upper-triangular matrix), is rarely arbitrary: many of the pair-wise effects are correlated because the underlying features share common “roles” or “types”. Formally, they hypothesise an approximate low-dimensional structure in the matrix and capture it with a latent-variable model (LVM)\n\nThe main approach is to assign every feature a latent (but lower dimensional d<p) vector z_j, and then the targeted model requires each interaction weights to sit near a simple function of  a dot-product  or  pair-wise distance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the main issue for the current paper is its lack of discussion to related researches, to name a few:\n\nProjection-pursuit chases non-linear ridge functions of projections; LIT-LVM stays linear but says: “if you must keep all x_i*x_j terms, at least tie their coefficients together through a low-dimensional latent map so they don’t overfit.” I am not sure I understand this, how is the model fundamentally different from projection pursuit, or it is a revised version with sparsity penalty?\nc.f. Friedman, Jerome H., and Werner Stuetzle. \"Projection pursuit regression.\" Journal of the American statistical Association 76.376 (1981): 817-823.\n\nDeep-kernel methods use a neural network as an unknown feature extractor before a Gaussian-process kernel; they can model very rich surfaces but give up the transparency of explicit \\beta and \\Theta. LIT-LVM keeps every coefficient on the table and simply nudges them toward a coherent latent pattern, are these two methods related or .\nc.f., Noack, Marcus M., Hengrui Luo, and Mark D. Risser. \"A unifying perspective on non-stationary kernels for deeper Gaussian processes.\" APL Machine Learning 2.1 (2024).\nWilson, Andrew Gordon, et al. \"Deep kernel learning.\" Artificial intelligence and statistics. PMLR, 2016.\n\nI think that the proposed model fills a niche between classical linear-interaction models (too many parameters), low-rank factorization (too rigid), and fully non-parametric surfaces (hard to interpret), but this is not clear in the current draft; specifically if the authors do not provide theoretic guarantees (e.g., consistency, risk bounds), then the experimental evidence for comparing its performance against related methods above, and state-of-the-art method is insufficient in the current shape. \n\n(1)Requested changes: can the authors add discussion to related literature and methods, then append experimental evidence to illustrate their method? \n\n(2)Requested changes 2: One of the benefit of LIT model is illustrated in \"HLA Latent Representations\" application, it only compares its outcome with PCA; Admittedly, it is not too popular to perform dimension reduction on latent variable space, yet there are certain line of research called supervised dimension reduction (e.g., SIR), are these relevant and worth compared to? In addition, can we use other unsupervised dimension reduction methods to have a more comprehensive comparison?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)