Fine-grained Generalization Analysis of Inductive Matrix Completion

? r logpdq , where d is the dimension of the side information and r is the rank . ( 2 ) We introduce the ( smoothed ) adjusted trace-norm minimization strategy , an inductive analogue of the weighted trace norm , for which we show guarantees of the order Opdr logpdqq under arbitrary sampling . In the inductive case , a similar rate was previously achieved only under uniform sampling and for exact recovery . Both our results align with the state of the art in the particular case of standard ( non-inductive ) matrix completion , where they are known to be tight up to log terms . Experiments further confirm that our strategy outperforms standard inductive matrix completion on various synthetic datasets and real problems , justifying its place as an important tool in the arsenal of methods for matrix completion using side information . 1 Introduction . Matrix completion ( MC ) is the machine learning problem of recovering the missing entries of a partially observed matrix . It is the go-to approach in various application domains such as recommender systems [ 1 , 2 ] and social network analysis [ 3 , 4 , 5 ] . The SoftImpute algorithm [ 6 , 7 ] is among the most popular MC methods . It solves the following convex problem encouraging low-rank solutions : min ZPRmˆn 1 2 } PΩpZ ´ Gq } 2Fr ` λ } Z } ˚ , ( 1 ) where PΩ denotes the projection on the set Ω of observed entries , G is the ground truth matrix , and } . } ˚ denotes the nuclear norm ( i.e. , the sum of the matrix ’ s singular values ) . Besides the incomplete matrix , additional information may be available in applications such as movie recommendation or drug interaction prediction [ 8 , 9 , 10 , 11 ] . For instance in movie recommendation , one may have access to the movies ’ genres , their synopsis , the gender and occupation of the users , or a friendship network between the users . Inductive matrix completion ( IMC ) [ 11 , 12 , 13 , 14 ] exploits such side information . It assumes that the side information is summarized in matrices X P Rmˆd1 and Y P Rnˆd2 , with the row vectors representing the users and items , respectively . IMC then optimizes the following objective function min MPRd1ˆd2 1 2 } PΩpXMY J ´ Gq } 2Fr ` λ } M } ˚ . ( 2 ) This model has been used in many domains also beyond movie recommendation [ 8 , 10 , 15 ] . In this paper , we contribute to a better theoretical understanding of IMC and related methods in the approximate recovery case . In this setting we obtain guarantees in terms of a bound on the expected 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . loss which decreases with the number of samples . Our best results concern the distribution-free case , meaning that our bounds are valid for any sampling distribution . This is in sharp contrast to the vast areas of literature where one assumes the distribution is uniform [ 16 , 17 , 18 ] . Our analysis leads to substantial gains compared to the state of the art results [ 19 , 20 , 21 ] , as we obtain near optimal bounds in situations where the state of the art bounds are vacuous , as is explained below . Although for uniform sampling , near-tight exact recovery bounds of Oprd logpdq logpnqq exist1 for IMC [ 16 , 17 ] , the approximate recovery case ( especially in a distribution-free setting ) is far less understood . The state-of-the-art distribution-free results for IMC were proved in [ 19 , 20 ] ( and in [ 21 ] for a kernel formulation of IMC ) and , expressed in terms of generalisation error bounds , scale as O ´ xyM a 1 { N ¯ , ( 3 ) where x : “ } XJ } 2,8 “ maxu } X . , u } 2 is the maximum norm of a left side information vector ( row of X ) , N is the number of available samples , and y : “ } Y J } 2,8 “ maxv } Y . , v } 2 is the maximum norm of a right side information vector ( row of Y ) . This implies that reaching a given loss threshold ϵ requires Opx2y2M2 { ϵ2q entries , where M is a bound on the nuclear norm of M . In this case , we say that the ’ sample complexity ’ is Opx2y2M2q . To understand how those bounds scale with the matrix dimensions , consider the simple case where X and Y are made up of blocks of identity matrices . In that case , we have x “ y “ 1 , yielding a sample complexity of OpM2q . Since } M } 2˚ „ d2r , this yields a bound of order rd2 . Such bounds have a remarkable property : they do not depend on the size n of the matrix and instead depend only on the size d of the side information . This means that they capture the fact that valuable information can be extracted even for users and items for which no ratings are observed . On the other hand , these bounds have a strong dependence on the size d of the side information . As an illustration , consider that they are vacuous when X “ I and Y “ I , since the required number of entries Oprd2q “ Oprn2q then grows faster than the total number of entries n2 . This is despite the fact that in that situation , distribution-free bounds for standard matrix completion yield a sample complexity of Opn3 { 2 ? rq for the standard regulariser [ 22 ] and Opnr logpnqq for a modified regulariser ( the smoothed weighted trace norm from [ 23 ] ) . Thus , these existing distribution-free IMC bounds are very far from tight . In fact , they are only meaningful when the size of the side information is negligible compared to the general scale of the problem , which is a significant limitation in terms of the elegance of the theory ( mismatch with MC bounds , separate proof techniques for separate regimes ) and in practice ( real-life side information could be very high-dimensional , especially if it is extracted from a neural network [ 24 ] or from a wide variety of different sources ) . To reinforce that point , note that any side information with a strong cluster structure2 would exhibit similar failings to the above mentioned identity side information case . In this work , we bridge the gap between the state-of-the art in matrix completion and inductive matrix completion with the trace norm by providing distribution-free bounds for IMC which combine both of the following advantages : ( 1 ) a lack of dependence in the size of the original matrices , and ( 2 ) a more refined dependence on the size of the side information : the dependence on d in our bounds is almost the same as the dependence on n ( the size of the matrix ) for the state-of-the-art MC results . More precisely , our first contribution is to provide a bound of order Opd3 { 2 ? r logpdqq for the standard regulariser ( 2 ) . The proof builds on techniques from [ 22 , 25 ] , but is substantially more involved due to the complicated dependence structure generated by the side information . As our second contribution , we construct analogues of the ideas of [ 23 , 26 ] for the IMC setting : we begin by showing a bound of order Oprd logpdqq for a class of distributions with certain uniformity assumptions ( our `` uniform inductive marginals '' ) , and then design a new `` adjusted trace norm regulariser '' for the problem ( 2 ) with similar properties to the weighted trace norm [ 26 , 23 ] in MC . Instead of simply renormalising rows and columns of M as in previous work , our method requires rescaling the core matrix M along data-dependent orientations that capture interplay between the sampling distribution and the side information matrices X , Y . Our contributions are summarised as follows . 1with some orthogonality assumptions on the side information 2where the users and items are approximately split into ’ communities ’ , see also Appendix A 1 . We provide distribution-free generalisation bounds for the inductive matrix completion model ( 2 ) ( assuming a fixed upper bound on the nuclear norm ) which scale like Opd3 { 2 ? r logpdqq where r is a soft relaxation of the rank . 2 . In the case of uniform or approximately uniform sampling , we provide a bound of order Oprd logpdqq for approximate recovery . 3 . We introduce a modified version of the IMC objective ( 2 ) , which we refer to as adjusted trace norm regularsation ( ATR ) . An empirical version E-ATR is also introduced and both achieve bounds of order Oprd logpdqq in the distribution-free setting . 4 . We experimentally demonstrate on synthetic data that our adjusted regulariser outperforms the standard IMC objective ( 2 ) in many situations . 5 . We incorporate our method into a model involving a non-inductive term and evaluate it on real-life datasets , demonstrating substantially improved performance . This paper is organized as follows . In Section 2 we review some related work . In Section 3 we introduce our main results . Finally , in Section 4 we present our experimental results . 2 Related work . In both MC and IMC , the existing literature consists of several main branches differing in their main assumptions : exact recovery versus approximate recovery and uniform sampling versus distributionfree bounds . In exact recovery , the matrix is assumed deterministic , and we want to recover its missing entries exactly [ 17 , 16 , 27 , 28 ] . In approximate recovery , the matrix is assumed noisy , and we want to recover its missing entries only approximately , within some interval around their expectation [ 19 , 20 , 21 , 18 , 29 ] . Approximate recovery theory is typically expressed in terms of uniform generalisation bounds over a function class using a matrix-norm constraint . Assuming that the entries are sampled from a specific distribution ( e.g. , uniform ) , one typically can achieve much faster rates than distribution-free theory regardless of the distribution . The typical sample complexity of standard MC under uniform sampling is Opnr log2pnqq for exact recovery ( proved in the series of breakthrough papers [ 27 , 28 , 30 ] ) and Opnr logpnqq for approximate recovery [ 23 ] . In [ 31 , 32 ] , an improved rate of nr logpnq logprq ( for exact recovery ) was shown . The most closely related papers to ours are [ 22 ] and [ 23 ] , which both work only on standard matrix completion without side information . In [ 22 ] , a bound of order Opn3 { 2 ? rq was obtained in the distribution-free setting for matrix completion with the trace norm , whilst in [ 23 ] , rates of Oprn logpnqq are shown for sampling with uniform marginals and for a smoothed version of the weighted trace norm regulariser in the distribution-free case . We almost perfectly extend most of the results from both papers to the inductive case , which requires many technical modifications . Within the IMC framework the closest works are those which also deal with approximate recovery in the non uniform sampling case : [ 21 , 33 , 19 , 20 ] . Their bounds , presented in many different contexts , translate to sample complexities of type Oprd2q . Other celebrated works in the theoretical study of IMC include : [ 16 ] and [ 17 ] , which showed rates of order d2r3 logpdq and rd logpdq logpnq respectively for exact recovery with uniform sampling , together with other important contributions ( see appendix ) . In the case of exact recovery , the rate of rd logpdq logpnq was obtained only under the assumption that the side information matrices have orthonormal columns . Some bounds use a completely different regulariser ( such as the max norm ) to achieve better rates [ 34 , 35 ] etc . These works also do not involve side information . In Figures 1 and 2 , we summarize state-of-the-art ( s.o.t.a . ) results in both MC and IMC . Note the problem of exact recovery in the distribution-free case is ill-defined ( hence the N/As in our table ) . In approximate recovery bounds , we omit a factor of 1 { ϵ2 , where ϵ is the tolerance threshold in terms of expected loss ) , as this factor is present in all approximate recovery bounds 3 . In exact recovery bounds , the rate is the order of magnitude of the threshold past which exact recovery occurs with high probability . 3To our best knowledge , all results show a decline in population expected loss of the order of a 1 { N where N is the sample size Other related works include ( IMCNF ) [ 19 , 20 ] , which proposed the following model : min 1 2 ÿ pi , jqPΩ |Gi , j ´ pXMY J ` Zqi , j |2 ` λ1 } M } ˚ ` λ2 } Z } ˚ , ( 4 ) where λ1 , λ2 are regularisation parameters , Gi , j denotes the observed entries and the predictors take the form pXMY J ` Zq . This model relies on the cross-validated hyperparameters λ1 , λ2 to balance the importance of the side information . The authors also showed results based on a combination of a bound for the inductive term XMY J and a bound for the non inductive term Z . The non inductive terms in the bounds are similar to [ 22 ] , whilst the bounds for the inductive term are proved from scratch and have also later appeared in a different form in [ 21 , 33 ] together with a kernel formulation of IMC . In Subsection 4.2 we combine our framework with this strategy to reach competitive results . In [ 36 ] , the authors introduce a model consisting of a sum of mutually orthogonal IMC terms together with an explicit optimization strategy in the specific case where the available side information consists in partitions of the users and items into communities . In [ 37 ] , the authors further extend the model to learn the community membership functions together with the ground truth matrix , based only on the sampled entries . The case of a single IMC term where the side information is in the form of a community partition is useful to develop intuition into the equivalent roles of d1 , d2 in our bounds versus m , n in MC bounds . Whilst generalization bounds were proved in [ 36 ] with a similar scaling as our bound from Thm 3.1 ( and in particular are better than the state-of-the-art IMC bounds if applied to this situation ) , they only apply to the specific case of community side information . In this work ( Theorem 3.1 ) we achieve the first IMC bounds which cover the whole range of possible side information matrices X , Y , whilst providing the correct scaling ( up to log terms ) in the case of identity or community side information . Community side information has also been studied in other discrete contexts where individual behaviour is assumed to be a noisy realisation of community side information [ 38 , 39 ] . Another work is [ 18 ] which introduces a joint model that imposes a nuclear norm-based constraint on both M and XMY J through a modification of the objective . The authors prove bounds for their method which match the state of the art in IMC [ 17 , 19 ] and MC [ 22 ] when the side information is perfect and useless respectively . The dependence on the side information is better in our case . Further discussion of that paper is included in the appendix . Of course , there are also many other works which propose modified optimization problems for the Recommender Systems task through other rank-sparsity inducing regularisers [ 35 , 34 , 40 ] and even exploiting other ground truth structure besides the low-rank property [ 41 , 42 ] .

Summary: This paper studies generalisation bounds of Inductive Matrix Completion (IMC), a variant of the classic matrix completion (MC) problem. The main contribution of the paper is improved bounds in the distribution-free setting. Moreover, the authors introduce the (smoothed) adjusted trace-norm minimization strategy, for which they show non-trivial guarantees under arbitrary sampling. These two results generalize similar results previously proved for standard matrix completion to IMC.