Factorization Machines : A New Tool for Sparse Data

Factorization machines are a new type of model that is well suited to very high-cardinality, sparsely observed transactional data. This paper presents the new FACTMAC procedure, which implements factorization machines in SAS® Visual Data Mining and Machine Learning. This powerful and flexible model can be thought of as a low-rank approximation of a matrix or a tensor, and it can be efficiently estimated when most of the elements of that matrix or tensor are unknown. Thanks to a highly parallel stochastic gradient descent optimization solver, PROC FACTMAC can quickly handle data sets that contain tens of millions of rows. The paper includes examples that show you how to use PROC FACTMAC to recommend movies to users based on tens of millions of past ratings, predict whether fine food will be highly rated by connoisseurs, restore heavily damaged high-resolution images, and discover shot styles that best fit individual basketball players. INTRODUCTION Factorization models, which include factorization machines as a special case, are a broad class of models popular in statistics and machine learning. For example, principal component analysis is a wellknown type of factorization model that has long been a staple of dimensionality reduction. For another example, matrix factorization has been widely used in text analysis and recommender systems. More recently, Rendle (2010, 2012) has proposed factorization machines for recommender systems and clickthrough rate prediction. Factorization machines are a powerful model that significantly extends matrix factorization. Factorization machines are included in SAS Visual Data Mining and Machine Learning. The initial release supported matrix factorization with biases, and the latest implementation supports pairwise-interaction tensor factorization and nonnegative factorization. A macro is provided in the Appendix so that you can still perform pairwise-interaction tensor factorization even if you have PROC FACTMAC from the first release of SAS Visual Data Mining and Machine Learning. This paper begins by briefly explaining the most relevant technical details of factorization machines for data scientists. Then it focuses on applications of factorization machines to solve real-world business problems. The application sections, which can be read by non-experts, include usage tips as well as code. The following application examples are presented:  recommending movies to users based on tens of millions of past ratings  predicting whether a fine food item will be highly rated by connoisseurs  restoring heavily damaged high-resolution images  discovering shot styles that best fit individual basketball players From these examples, you will learn how to spot which types of problems are good candidates for factorization machines, how to prepare data for PROC FACTMAC, how to score new data by using score code or PROC ASTORE, and what strategies to use for choosing parameters and training the best factorization machine models. THE FACTORIZATION MACHINE MODEL This section begins with a brief mathematical description of factorization machines. Assuming a training set D = {(xi , yi)}, with i = 1, . , n, where xi refers to the ith observation and yi refers to the ith target value, the factorization machine model of order 2 is written as