On the regularization landscape for the linear recommendation models

1 INTRODUCTION . Research progress on algorithms for recommendation has escalated in recent years , partially fueled by the adoption of deep learning ( DL ) techniques . However , recent studies have found that many new deep learning recommendation models have shown sub-par performance against simpler linear recommendation models ( Dacrema et al. , 2019 ; Rendle et al. , 2019 ) . Although some studies are available to analyze linear vs non-linear models ( Dacrema et al. , 2019 ) , it remains puzzling why these seemingly different techniques all result in models with similar performance or even similar cost functions . In the latest study , Jin et al . ( 2021 ) examine the relationship between the widely used matrix factorization ( MF ) , such as ALS ( Hu et al. , 2008 ) , and the linear autoencoder ( LAE ) which encompasses the recent performance leaders , such as EASE ( Steck , 2019 ) and EDLAE ( Steck , 2020 ) . They considered two basic regularization forms ( see Eqs . ( 1 ) and ( 6 ) ) and found that the optimal ( closed-form ) solutions of both models recover the directions of principal components , while shrinking the corresponding singular values differently . They suggest that the difference may enable LAE to utilize a larger number of latent dimensions to improve recommendation accuracy . This finding highlights the similarity and difference between LAE and MF . In this paper , we go much beyond the two basic models studied in ( Jin et al. , 2021 ) , to analyze a large number of recent performance leaders of ( linear ) recommendation algorithms , and aim to provide an in-depth understanding of various regularized objective functions and towards unifying them through their closed-form solutions . In return , the closed-forms serve as the barebones engine to help reveal what drives the performance improvement for the recent recommendation algorithms . We examine three open and closely related questions : ( i ) Characterization of models : Recent recommendation models are built upon a diverse set of techniques , including dropouts ( Srivastava et al. , 2014 ) , variational methods ( Kingma & Welling , 2014 ) , and matrix denoising ( Tipping & Bishop , 1999b ) . We aim to identify a unified regularization framework to ( re ) -interpret different models and understand how they are related . We are specifically interested in how models based on variational autoencoder ( VAE , a natural generalization of recent approaches ) relate to other models ( Liang et al. , 2018 ) . ( ii ) Weighted generalization of regularizers : A key idea in a recent performance leader EDLAE ( Steck , 2020 ) is to utilize of weighted/non-uniform regularizers on the parameters ’ Frobenius norm . Applying dropouts ( Cavazza et al. , 2018 ) is shown to be equivalent to re-weighting the exponents in the regularizers , such as designing the weighted sum of regularizers based on other norms or tuning the exponent weights . We are also interested in determining what circumstance closed-form solutions still exist . ( iii ) Low-rank closed-form solutions for EDLAE . While most linear recommendation models are shown to have closed-form solutions , the low-rank version of recent performance leader EDLAE ( Steck , 2020 ) remains an exception . Fitting this model requires using ADMM ( Steck , 2020 ) . Although its performance may not be better than its full-rank counterpart , low-rank solutions are easier to interpret , use less storage , and can be more scalable with respect to the number of items . In addition , a closed-form solution disentangles key performance drivers from nuances ( e.g. , need to tune learning rate or deal with local optimal ) , and can help reveal the key driven factors comparing with other closed-form solutions . More importantly , such solutions are significantly easier to implement and be tested using a generic matrix computation platform ( without specialized recommendation library ) . Can we approximate low-rank EDLAE with closed-form solutions ? Our investigation led to the following discovery and resolves the above questions : Regularizer dichotomy ( Section 2 ) : We found that all of the leading ( linear ) recommendation models can be categorized into those that implement possibly weighted nuclear-norm regularizers , or those that implement Frobenius-norm regularizers . Specifically , we characterize the Variational Linear Autoencoders ( VLAE ) as a form of the weighted nuclear-norm regularization problem , in which the weights possess a specific combinatorial structure . We observe that it is not matrix factorization or LAE that determines the shrinkage structure ( as Jin et al . ( 2021 ) suggested ) , but instead it is the form of regularization . Thus , our paper provides a more complete and accurate characterization of a linear recommendation model ’ s performance under different regularizations . Rigidity of nuclear-norm regularizers ( Section 3 ) : With the dichotomy result , we next aim to understand whether the weighted sum idea for Frobenius-norm regularizers ( Steck , 2020 ; Jin et al. , 2021 ) can be generalized to nuclear-norm regularizers , and whether tuning exponent weights can be beneficial . First , while VLAE is equivalent to nuclear-norm regularization and is easier to optimize ( weighted nuclear norms are non-smooth but VLAE ’ s objectives are smooth ) , its closedform solution possesses a ( surprisingly ) rigid structure , i.e. , the weighted regularization will lead to an auto-sorting singular value reduction and the larger single values tend to receive smaller reduction ( non-ascending reduction ) . Second , it has been shown that the solution structure for a model with a squared nuclear-norm regularizer ∥W∥2∗ ( i.e. , dropout ’ s equivalence ) is strikingly similar to that for using ∥W∥∗-regularizer ( Gu et al. , 2014 ) . We generalize the result to show that the solution structures for ∥W∥p∗ are highly similar for all p ≥ 1 ( Regularization invariant/rigidity with respect to p ) . But when p = 1 , 2 , the solution and hyperparameters possess favorable properties which make hyperparameter searches easier . This also partially explains why only p = 1 , 2 have been extensively considered . These rigidity properties severely limit the search space and explains why models that use only nuclear-norm regularizers share the same performance ceiling even when hyperparameters are extensively searched . Closed-form solution for low-rank EDLAE ( Section 4 ) : The ( weighted ) Frobenius-norm regularizers ∥ΛW∥2F ( Λ is the hyperparameter diagonal matrix ) are implemented in EASE ( Steck , 2019 ) and EDLAE ( Steck , 2020 ) . These models produce closed-form full-rank estimators ; and if the zero-diagonal constraint on W is enforced , their singular vectors will no longer coincide with those of the data , and will deliver ( slightly ) better performance . However , no closed-form solution for the low-rank estimator is known and existing approaches rely on ADMM or Alternating Least Square ( Steck , 2020 ) . In this paper , we propose two low-rank , closed-form estimators that deliver comparable results to the full-rank models ( EASE and full-rank EDLAE ) as well as the ADMM-based solutions ( Steck , 2020 ) , and thus resolve the aforementioned third open question . 2 BACKGROUND AND REGULARIZER DICHOTOMY . This section first explains the background , and then give an overview of the dichotomy results . Some key theorems are deferred to Section 3 and Appendices . Background . Recommendation algorithms can be categorized into explicit ones that aim to predict unseen ratings between a user and an item and implicit ones that aim to predict actions , such as user click or add-cart ( Steck , 2019 ; Dacrema et al. , 2019 ; Zhang et al. , 2019 ) . We focus on the implicit Table 1 : Investigating the closed/analytic solutions of linear models . dMat ( · ) denotes a diagonal matrix , diag ( X ) is the vector on the diagonal of X . Λ is the ( hyperparameter ) diagonal matrix as a coefficient of the regularization term . W ∗ ( or P ∗ , Q∗ , etc ) is the optimal solution for corresponding case , except for cases 9-12 , where Ŵ is the low-rank closed-form solution . Model Regularization Solution Nuclear norm 1 . Regularized PCA ( Zheng et al. , 2018 ) min P , Q ||X − PQ||2F + λ · ( ||P ||2F + ||Q||2F ) X SVD = UΣV T Ω = √ ( σi − λ ) + P ∗ = Uk Q ∗ = ΩV Tk 2 . MF dropout ( Cavazza et al. , 2018 ) min P , Q , d ||X − PQ||2F + d 1− p p · d∑ k=1 ||Pk||22 · ||QTk ||22 min Y ||X − Y ||2F + 1− p p ||Y ||2∗ X SVD = UΣV T Y ∗ = P ∗ ·Q∗ = U · Sµ ( Σ ) · V T 3 . WLAE ( Bao et al. , 2020 ) min W1 , W2 ∥X −XW1W2∥2F + ∥W1Λ 1 2 ∥2F + ∥Λ 1 2W2∥2F , W ∗1 = V ( I − ΛS−2 ) 1 2PT W ∗2 = P ( I − ΛS−2 ) 1 2V T 4 . VLAE ( this paper ) min P , Q ||X − PQ||2F + ||Λ1/2Q||2F + ||PΛ1/2||2F min A , B ||X −XAB||2F + ||ΛB||2F + ||XA||2F min rank ( W ) ≤k ||X −W ||2F + 2||W ||w , ∗ X SVD = UΣV T P ∗ = Uk · diag ( √ σ1 − λ ( k ) , . . . , √ σk − λ ( 1 ) ) · Ω Q∗ = ΩT · diag ( √ σ1 − λ ( k ) , . . . , √ σk − λ ( 1 ) ) · V Tk A∗ = X†P ∗Λ 1 2 B∗ = Λ− 1 2Q∗ Frobenius norm 5 . EASE ( full rank ) ( Steck , 2019 ) min W ||X −XW ||2F + λ · ||W ||2F s.t . diag ( W ) = 0 C = ( XTX + λI ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) 6 . DLAE ( full rank ) ( Steck , 2020 ) min W ||X −XW ||2F + ||Λ1/2 ·W ||2F Λ = p 1− p dMat ( diag ( XTX ) ) W ∗ = ( XTX + Λ ) −1XTX 7 . EDLAE ( full rank ) ( Steck , 2020 ) min W ||X −XW ||2F + ||Λ1/2 ·W ||2F Λ = p 1− p dMat ( diag ( XTX ) ) s.t . diag ( W ) = 0 C = ( XTX + Λ ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) 8 . LRR ( Jin et al. , 2021 ) min rank ( W ) ≤k ||X −XW ||2F + ||ΓW ||2F Y ∗ = XW ∗ SVD = UΣV Ŵ = ( XTX + ΓTΓ ) −1XTX ( VkV T k ) 9 . EDLAE-ADMM ( Steck , 2020 ) min A , B ||X −X ( AB − dMat ( diag ( AB ) ) ) ||2F +||Λ1/2 · ( AB − dMat ( diag ( AB ) ) ||2F ADMM update A , B 10 . LR-DLAE ( this paper ) min rank ( W ) ≤k ||X −XW ||2F + ||Λ1/2 ·W ||2F Λ = p 1− p dMat ( diag ( XTX ) ) W ∗ = ( XTX + Λ ) −1XTX Y ∗ = XW ∗ SVD = UΣV T Ŵ = W ∗ ( VkV T k ) 11 . LR-EDLAE-1 ( this paper ) min rank ( W ′ ) ≤k ||X −XW ||2F + ||Λ1/2 ·W ||2F W = W ′ − dMat ( diag ( W ′ ) ) Λ = p 1− p dMat ( diag ( XTX ) ) C = ( XTX + Λ ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) Y ∗ = XW ∗ SVD = UΣV T Ŵ = W ∗ ( VkV T k ) 12 . LR-EDLAE-2 ( this paper ) min rank ( W ′ ) ≤k ||X −XW ||2F + ||Λ1/2 ·W ||2F W = W ′ − dMat ( diag ( W ′ ) ) Λ = p 1− p dMat ( diag ( XTX ) ) C = ( XTX + Λ ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) W ∗ SVD = UΣV T Ŵ = UkΣkV T k problem because it is more economically relevant . Here , let n be the number of items and m be the number of users . We are given a binary matrix X ∈ { 0 , 1 } m×n that represents the interaction between users and items so far , i.e. , Xi , j = 1 iff user i has purchased or made a rating on item j . Our goal is to produce a real-valued matrix X̂ , which we evaluate against future interactions using information retrieval ( Top-k ) metrics such as Recall or nDCG . The problem is closely related to matrix completion ( MC ) because Xi , j = 0 can be viewed as “ missing a data point ” . But MC ’ s evaluation metric is mean-squared error and is different from ours ( Candès & Tao , 2010 ) . The connection between two problems results in models with similar objectives ( Zheng et al. , 2018 ) . A technique developed for one problem often finds its counterpart for the other . Nevertheless , because of the difference in evaluation , efficacy of an algorithm for one problem does not imply its performance guarantee for the other . Thus , the non-overlapping component between two problems remains substantial . We also remark that our structural results on weighted nuclear-norm is new and applicable to MC .

Summary: This paper categorizes various linear models based on the applied regularization: nuclear norm vs Frobenius norm regularization. It is claimed that this categorization is key to understanding the prediction accuracies of the various linear model-classes. The paper also makes connections between nuclear norm regularization and Frobenius-norm regularization, and also discusses L_p norm regularization of the singular values. The paper also describes 2 methods that yield a closed-form solution of a low-rank model with Frobenius norm regularization: while it is not motivated/explained why these methods should work from a conceptual/theoretical perspective, the experiments show that they indeed work surprisingly well.