On the regularization landscape for the linear recommendation models

1 INTRODUCTION . Research progress on algorithms for recommendation has escalated in recent years , partially fueled by the adoption of deep learning ( DL ) techniques . However , recent studies have found that many new deep learning recommendation models have shown sub-par performance against simpler linear recommendation models ( Dacrema et al. , 2019 ; Rendle et al. , 2019 ) . Although some studies are available to analyze linear vs non-linear models ( Dacrema et al. , 2019 ) , it remains puzzling why these seemingly different techniques all result in models with similar performance or even similar cost functions . In the latest study , Jin et al . ( 2021 ) examine the relationship between the widely used matrix factorization ( MF ) , such as ALS ( Hu et al. , 2008 ) , and the linear autoencoder ( LAE ) which encompasses the recent performance leaders , such as EASE ( Steck , 2019 ) and EDLAE ( Steck , 2020 ) . They considered two basic regularization forms ( see Eqs . ( 1 ) and ( 6 ) ) and found that the optimal ( closed-form ) solutions of both models recover the directions of principal components , while shrinking the corresponding singular values differently . They suggest that the difference may enable LAE to utilize a larger number of latent dimensions to improve recommendation accuracy . This finding highlights the similarity and difference between LAE and MF . In this paper , we go much beyond the two basic models studied in ( Jin et al. , 2021 ) , to analyze a large number of recent performance leaders of ( linear ) recommendation algorithms , and aim to provide an in-depth understanding of various regularized objective functions and towards unifying them through their closed-form solutions . In return , the closed-forms serve as the barebones engine to help reveal what drives the performance improvement for the recent recommendation algorithms . We examine three open and closely related questions : ( i ) Characterization of models : Recent recommendation models are built upon a diverse set of techniques , including dropouts ( Srivastava et al. , 2014 ) , variational methods ( Kingma & Welling , 2014 ) , and matrix denoising ( Tipping & Bishop , 1999b ) . We aim to identify a unified regularization framework to ( re ) -interpret different models and understand how they are related . We are specifically interested in how models based on variational autoencoder ( VAE , a natural generalization of recent approaches ) relate to other models ( Liang et al. , 2018 ) . ( ii ) Weighted generalization of regularizers : A key idea in a recent performance leader EDLAE ( Steck , 2020 ) is to utilize of weighted/non-uniform regularizers on the parameters ’ Frobenius norm . Applying dropouts ( Cavazza et al. , 2018 ) is shown to be equivalent to re-weighting the exponents in the regularizers , such as designing the weighted sum of regularizers based on other norms or tuning the exponent weights . We are also interested in determining what circumstance closed-form solutions still exist . ( iii ) Low-rank closed-form solutions for EDLAE . While most linear recommendation models are shown to have closed-form solutions , the low-rank version of recent performance leader EDLAE ( Steck , 2020 ) remains an exception . Fitting this model requires using ADMM ( Steck , 2020 ) . Although its performance may not be better than its full-rank counterpart , low-rank solutions are easier to interpret , use less storage , and can be more scalable with respect to the number of items . In addition , a closed-form solution disentangles key performance drivers from nuances ( e.g. , need to tune learning rate or deal with local optimal ) , and can help reveal the key driven factors comparing with other closed-form solutions . More importantly , such solutions are significantly easier to implement and be tested using a generic matrix computation platform ( without specialized recommendation library ) . Can we approximate low-rank EDLAE with closed-form solutions ? Our investigation led to the following discovery and resolves the above questions : Regularizer dichotomy ( Section 2 ) : We found that all of the leading ( linear ) recommendation models can be categorized into those that implement possibly weighted nuclear-norm regularizers , or those that implement Frobenius-norm regularizers . Specifically , we characterize the Variational Linear Autoencoders ( VLAE ) as a form of the weighted nuclear-norm regularization problem , in which the weights possess a specific combinatorial structure . We observe that it is not matrix factorization or LAE that determines the shrinkage structure ( as Jin et al . ( 2021 ) suggested ) , but instead it is the form of regularization . Thus , our paper provides a more complete and accurate characterization of a linear recommendation model ’ s performance under different regularizations . Rigidity of nuclear-norm regularizers ( Section 3 ) : With the dichotomy result , we next aim to understand whether the weighted sum idea for Frobenius-norm regularizers ( Steck , 2020 ; Jin et al. , 2021 ) can be generalized to nuclear-norm regularizers , and whether tuning exponent weights can be beneficial . First , while VLAE is equivalent to nuclear-norm regularization and is easier to optimize ( weighted nuclear norms are non-smooth but VLAE ’ s objectives are smooth ) , its closedform solution possesses a ( surprisingly ) rigid structure , i.e. , the weighted regularization will lead to an auto-sorting singular value reduction and the larger single values tend to receive smaller reduction ( non-ascending reduction ) . Second , it has been shown that the solution structure for a model with a squared nuclear-norm regularizer ∥W∥2∗ ( i.e. , dropout ’ s equivalence ) is strikingly similar to that for using ∥W∥∗-regularizer ( Gu et al. , 2014 ) . We generalize the result to show that the solution structures for ∥W∥p∗ are highly similar for all p ≥ 1 ( Regularization invariant/rigidity with respect to p ) . But when p = 1 , 2 , the solution and hyperparameters possess favorable properties which make hyperparameter searches easier . This also partially explains why only p = 1 , 2 have been extensively considered . These rigidity properties severely limit the search space and explains why models that use only nuclear-norm regularizers share the same performance ceiling even when hyperparameters are extensively searched . Closed-form solution for low-rank EDLAE ( Section 4 ) : The ( weighted ) Frobenius-norm regularizers ∥ΛW∥2F ( Λ is the hyperparameter diagonal matrix ) are implemented in EASE ( Steck , 2019 ) and EDLAE ( Steck , 2020 ) . These models produce closed-form full-rank estimators ; and if the zero-diagonal constraint on W is enforced , their singular vectors will no longer coincide with those of the data , and will deliver ( slightly ) better performance . However , no closed-form solution for the low-rank estimator is known and existing approaches rely on ADMM or Alternating Least Square ( Steck , 2020 ) . In this paper , we propose two low-rank , closed-form estimators that deliver comparable results to the full-rank models ( EASE and full-rank EDLAE ) as well as the ADMM-based solutions ( Steck , 2020 ) , and thus resolve the aforementioned third open question . 2 BACKGROUND AND REGULARIZER DICHOTOMY . This section first explains the background , and then give an overview of the dichotomy results . Some key theorems are deferred to Section 3 and Appendices . Background . Recommendation algorithms can be categorized into explicit ones that aim to predict unseen ratings between a user and an item and implicit ones that aim to predict actions , such as user click or add-cart ( Steck , 2019 ; Dacrema et al. , 2019 ; Zhang et al. , 2019 ) . We focus on the implicit Table 1 : Investigating the closed/analytic solutions of linear models . dMat ( · ) denotes a diagonal matrix , diag ( X ) is the vector on the diagonal of X . Λ is the ( hyperparameter ) diagonal matrix as a coefficient of the regularization term . W ∗ ( or P ∗ , Q∗ , etc ) is the optimal solution for corresponding case , except for cases 9-12 , where Ŵ is the low-rank closed-form solution . Model Regularization Solution Nuclear norm 1 . Regularized PCA ( Zheng et al. , 2018 ) min P , Q ||X − PQ||2F + λ · ( ||P ||2F + ||Q||2F ) X SVD = UΣV T Ω = √ ( σi − λ ) + P ∗ = Uk Q ∗ = ΩV Tk 2 . MF dropout ( Cavazza et al. , 2018 ) min P , Q , d ||X − PQ||2F + d 1− p p · d∑ k=1 ||Pk||22 · ||QTk ||22 min Y ||X − Y ||2F + 1− p p ||Y ||2∗ X SVD = UΣV T Y ∗ = P ∗ ·Q∗ = U · Sµ ( Σ ) · V T 3 . WLAE ( Bao et al. , 2020 ) min W1 , W2 ∥X −XW1W2∥2F + ∥W1Λ 1 2 ∥2F + ∥Λ 1 2W2∥2F , W ∗1 = V ( I − ΛS−2 ) 1 2PT W ∗2 = P ( I − ΛS−2 ) 1 2V T 4 . VLAE ( this paper ) min P , Q ||X − PQ||2F + ||Λ1/2Q||2F + ||PΛ1/2||2F min A , B ||X −XAB||2F + ||ΛB||2F + ||XA||2F min rank ( W ) ≤k ||X −W ||2F + 2||W ||w , ∗ X SVD = UΣV T P ∗ = Uk · diag ( √ σ1 − λ ( k ) , . . . , √ σk − λ ( 1 ) ) · Ω Q∗ = ΩT · diag ( √ σ1 − λ ( k ) , . . . , √ σk − λ ( 1 ) ) · V Tk A∗ = X†P ∗Λ 1 2 B∗ = Λ− 1 2Q∗ Frobenius norm 5 . EASE ( full rank ) ( Steck , 2019 ) min W ||X −XW ||2F + λ · ||W ||2F s.t . diag ( W ) = 0 C = ( XTX + λI ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) 6 . DLAE ( full rank ) ( Steck , 2020 ) min W ||X −XW ||2F + ||Λ1/2 ·W ||2F Λ = p 1− p dMat ( diag ( XTX ) ) W ∗ = ( XTX + Λ ) −1XTX 7 . EDLAE ( full rank ) ( Steck , 2020 ) min W ||X −XW ||2F + ||Λ1/2 ·W ||2F Λ = p 1− p dMat ( diag ( XTX ) ) s.t . diag ( W ) = 0 C = ( XTX + Λ ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) 8 . LRR ( Jin et al. , 2021 ) min rank ( W ) ≤k ||X −XW ||2F + ||ΓW ||2F Y ∗ = XW ∗ SVD = UΣV Ŵ = ( XTX + ΓTΓ ) −1XTX ( VkV T k ) 9 . EDLAE-ADMM ( Steck , 2020 ) min A , B ||X −X ( AB − dMat ( diag ( AB ) ) ) ||2F +||Λ1/2 · ( AB − dMat ( diag ( AB ) ) ||2F ADMM update A , B 10 . LR-DLAE ( this paper ) min rank ( W ) ≤k ||X −XW ||2F + ||Λ1/2 ·W ||2F Λ = p 1− p dMat ( diag ( XTX ) ) W ∗ = ( XTX + Λ ) −1XTX Y ∗ = XW ∗ SVD = UΣV T Ŵ = W ∗ ( VkV T k ) 11 . LR-EDLAE-1 ( this paper ) min rank ( W ′ ) ≤k ||X −XW ||2F + ||Λ1/2 ·W ||2F W = W ′ − dMat ( diag ( W ′ ) ) Λ = p 1− p dMat ( diag ( XTX ) ) C = ( XTX + Λ ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) Y ∗ = XW ∗ SVD = UΣV T Ŵ = W ∗ ( VkV T k ) 12 . LR-EDLAE-2 ( this paper ) min rank ( W ′ ) ≤k ||X −XW ||2F + ||Λ1/2 ·W ||2F W = W ′ − dMat ( diag ( W ′ ) ) Λ = p 1− p dMat ( diag ( XTX ) ) C = ( XTX + Λ ) −1 W ∗ = I − C · dMat ( diag ( 1⊘ C ) ) W ∗ SVD = UΣV T Ŵ = UkΣkV T k problem because it is more economically relevant . Here , let n be the number of items and m be the number of users . We are given a binary matrix X ∈ { 0 , 1 } m×n that represents the interaction between users and items so far , i.e. , Xi , j = 1 iff user i has purchased or made a rating on item j . Our goal is to produce a real-valued matrix X̂ , which we evaluate against future interactions using information retrieval ( Top-k ) metrics such as Recall or nDCG . The problem is closely related to matrix completion ( MC ) because Xi , j = 0 can be viewed as “ missing a data point ” . But MC ’ s evaluation metric is mean-squared error and is different from ours ( Candès & Tao , 2010 ) . The connection between two problems results in models with similar objectives ( Zheng et al. , 2018 ) . A technique developed for one problem often finds its counterpart for the other . Nevertheless , because of the difference in evaluation , efficacy of an algorithm for one problem does not imply its performance guarantee for the other . Thus , the non-overlapping component between two problems remains substantial . We also remark that our structural results on weighted nuclear-norm is new and applicable to MC .

Summary: This paper analyzed the linear models for recommendation from their regularization terms, more specifically, via nuclear norm and Frobenius norm. Based on the analyzes, this paper proposed two alternatives for linear recommendation models with closed form solutions. Experiments on three large scale datasets showed that the proposed alternatives are comparable to the state-of-the-art baseline methods.