Title: STRUCTURED DEEP FACTORIZATION MACHINE: TOWARDS GENERAL-PURPOSE ARCHITECTURES

Abstract: In spite of their great success, traditional factorization algorithms typically do not support features (e.g., Matrix Factorization), or their complexity scales quadratically with the number of features (e.g, Factorization Machine). On the other hand, neural methods allow large feature sets, but are often designed for a specific application. We propose novel deep factorization methods that allow efficient and flexible feature representation. For example, we enable describing items with natural language with complexity linear to the vocabulary size-this enables prediction for unseen items and avoids the cold start problem. We show that our architecture can generalize some previously published single-purpose neural architectures. Our experiments suggest improved training times and accuracy compared to shallow methods.

1. INTRODUCTION:

In recent years, predictive tasks that traditionally have been solved with factorization are now being studied within the context of neural networks. These solutions often work as black boxes, and many times they are designed specifically for a single task with an arbitrary network that may not have much justification. We propose Deep Structured Factorization Machine, a family of general-purpose factorization techniques that can be used stand-alone or as a "design pattern" within a larger neural network. Our work provides some insight into how to enable general-purpose factorization within neural architectures without losing interpretability and a principled design.

Previous factorization methods do not scale to large feature sets and make strong assumptions about their latent structure. Our main contribution is that we enable a general-purpose framework that enables efficient factorization of datasets with complex feature sets. For example, applications of factorization in natural language scale quadratically in the number of words in the vocabulary. Our solution allows inference with linear runtime complexity on the vocabulary size. Previous work has explored how to improve factorization's accuracy (see § 3.3) with its current limitations withstanding; alternatively, some have proposed how to make it tractable for a particular domain-for example, text (Zheng et al., 2017) . We believe that we are the first ones to propose an efficient general-purpose method. Interestingly, our experiments indicate that Structured Deep Factorization has large improvements in predictive accuracy and runtime compared to some recent ad-hoc models.

2. PRELIMINARIES:

Factorization Machine (Rendle, 2010) is one of the most succesful methods for general purpose factorization. Rendle (2010) formulated it as an extension to polynomial regression. Consider a degree-2 polynomial (quadratic) regression, where we want to predict a target variable y from a feature vector x ∈ R n : ŷ(x; b, w) = ω b 0 + λ 1 (x, b) + λ p (x, w)

(1)

Here, λ 1 and λ p are one-way and pairwise interactions:

EQUATION

In words, n is the total number of features, the term b 0 is an intercept, b i is the strength of the i-th feature, and w i,j is the interaction coefficient between the i-th and j-th feature. The function ω is an activation. Choices for ω include a linear link (ω(x) = x) for continuous outputs, or logistic link (ω(x) = log exp(x) exp( x+1) ) for binary outputs.

Factorization Machine replaces the two-way individual pairwise parameters with shared parameters β i . This is a rank-r vector of factors-embeddings in the neural literature-that encode the interaction between features:

λ FM (x, β) n i=1 n j=i+1 x i β i • x j β j (4)

Intuitively, the dot product (•) returns a scalar that measures the (dis)similarity between the two factors. Polynomial regression has n 2 interaction parameters, and Factorization Machine has n × r. While using r n makes the model less expressive, this is often desirable because factorization is typically used when features have some shared latent structure. Factorization Machine may dramatically reduce the number of parameters to estimate; however, inference has runtime that scales quadratically with the number of features-it requires O(n 2 ) dot products. Rendle (2010) shows that when the feature vector x consists only of two categorical features in one-hot encoding,Factorization Machine is equivalent to the popular Matrix Factorization algorithm (Koren et al., 2009) .

3. STRUCTURED FACTORIZATION MACHINE:

Factorization Machine has the following working-assumptions and limitations:

1. Strong parameter sharing. Factorization Machine assumes that all of the pairwise interactions factorize into a common shared space. This may not be the case when using large and complex feature sets. To the extent of our knowledge, softening this assumption has not been studied in the literature before. We study how to do so in § 3.1. 2. Intractable for large feature sets. Doing inference on Factorization Machine has a runtime complexity that scales quadratically with the number of features. Thus, it is intractable for complex feature sets such as text. We address this in § 3.2.

We propose Structured Factorization Machine as a building block that we will extend with neural methods. It is a simple yet effective way to model structure between groups of features. In Factorization Machine, all the features interact with each other. In the structured model, the features only interact with features from a different group. We do this by defining groups of features κ. We define Structured Factorization Machine as:

EQUATION

The interactions occur from features in different groups:

λ s (x, β, I, J) i∈I j∈J x i β i • x j β j (6)

For example, consider a model with four features (n = 4). If we define κ = {{1, 2}, {3, 4}}, feature x 1 would only interact with x 3 and x 4 . Without loss of generality, we could define a model that is equivalent to a shallow Factorization Machine by allowing each feature to be in a singleton group: κ = {{1}, {2}, {3}, {4}}.

Figure 1 compares existing factorization methods with our novel models. In the rest of this section we review them. 

. . . x k . . . x j x i x k x j x i x j x i ... . x k x j x i (a) Shallow . . . x k . . . x j x i x k x j x i x j x i ... . x k x j x i (b) Deep-Out . . . x k . . . x j x i x k x j x i x j x i ... . x k x j x i (c) Deep-In . . . x k . . . x j x i x k x j x i x j x i ... . x k x j x i (d) "Neural"

3.1. STRUCTURED DEEP-OUT FACTORIZATION MACHINE:

We now contrast Polynomial Regression (Equation 1) and Factorization Machine. Polynomial regression is convex, and thus can be optimized globally. Because Factorization Machine is nonconvex, optimization can only guarantee to find local optima.

The factorization assumption is more desirable than Polynomial Regression when there is shared structure between the features. The larger number of parameters of Polynomial Regression makes it likelier to overfit. However, we consider two properties of independent interactions that are desirable. First, it is unclear if all interactions strongly factorize when using very large feature sets. For example, some interactions may have different importance-perhaps because some features are irrelevant and should be discarded. Second, it may be easier to optimize a model with individual parameters than a model with factorized parameters. Polynomial regression converges quickly because the likelihood function is convex-a unique solution exists.

We aim to combine the advantage of both models. We propose a hybrid factorized-independent method that we call Structured Deep-Out Factorization Machine, because it extends Structured Factorization Machine outside the dot product (see Figure 1b ):

ω b 0 + n i=1 b i x i + ψ |κ| i=1 |κ| j=i w i,j λ s (x, β, κ i , κ j ) (7)

Here, the the embeddings β are factorized and shared across the pairwise interactions. The parameters w are independent to each interaction group defined by κ.

The function ψ is any activation function. If we were interested in interpreting the parameters, we would constrain w to be non-negative and choose ψ to be linear. Alternatively, the Rectified Linear Unit (ReLU) activation may be desirable because it is easy to optimize (Krizhevsky et al., 2012) :

ψ(x) = max(0, x).

We leave for future work the consideration of deeper networks.

3.2. STRUCTURED DEEP-IN FACTORIZATION MACHINE:

Structured Deep-In Factorization Machine allows treating a group of features as a single one; it extracts features from each feature group, and builds latent factors from them. Consider Figure 1c : the first group only has a single feature which is projected to an embedding (just like a regular Factorization Machine); the second group has multiple features, which are together projected to a single embedding. More formally:

EQUATION

In this notation, x κi is a subvector that contains all of the features that belong to the group κ i . Thus,

x κi = [x j |j ∈ κ i ].

The intuition is that by grouping (sub-)features as a single entity, we can can reason on a higher level of abstraction. Instead of individual sub-features interacting among each other, the entities interact with other entities. Here, φ i is a feature extraction feature that inputs the i-th feature group of the instance, and returns an embedding. The simplest implementation for φ i is a linear fully-connected layer, where the output of the r-th entry is:

φ i x i ; β r = di a=1 β ra x ia

Within a group, a feature extraction function may allow for subfeatures to interact with each other. Across groups, entities interact with each other via the output of φ only.

Leveraging item descriptions to circumvent the cold-start problem We now describe how we can use Structured Deep-In Factorization Machine for large feature sets-such as natural language.

In this scenario Factorization Machine is not tractable: if we use each word in the vocabulary as independent feature, we have to consider the interaction of each word with every word in the vocabulary and the other features, which means the number of interaction terms would depend quadratically on the size of the vocabulary. A traditional work-around is to ignore the features, and use matrix factorization by using a unique one-hot index for each item (e.g, each image/text), which avoids the quadratic dependence on the vocabulary size, but in this case the model suffers from the cold-start issue: inference is not possible for a new item not seen during training, even if it shares many words with existing items.

We can use the Deep-In model to both use large feature sets and overcome the cold-start problem. This is only possible when there is an alternative description of the item available (for example an image or a text). In Figure 2 , we show how we address this problem by treating the words as indexed features, but placed within a structured feature group κ w . A feature extraction function φ acts on the features in κ w , and the other features interact with the words only via the output of φ. Notice that this implies we can precompute and store the latent factors of the labels seen during training, and predictions during inference can be sped-up. For example if we have two feature groups (e.g, a label and an item), first we compute the feature extraction function to the unseen items and their embeddings, and then we simply apply a dot product over the stored vectors of the labels. Here, y(x) is the true target value for x obtained from training data, and ŷ(x) is the one estimated by the model; the hyperparameter γ controls the amount of regularization. For the labeling and classification tasks, we optimize the binary cross-entropy for y ∈ {0, 1}:

EQUATION

For the regression tasks where the target value is continuous, we optimize the mean squared error (MSE):

EQUATION

Neural models are typically learned using mini-batch updates, where the incremental descent is performed with respect to several instances at a time. For the implementation of this paper, we built our models using the Keras programming toolkit (Chollet et al., 2015) , that is now part of Tensorflow (Abadi et al., 2015) . It enables automatic differentiation, and is bundled with a generalpurpose optimization algorithm called ADAM (Kingma & Ba, 2014) that needs no tuning of gradient step sizes.

For the multi-label classification tasks, we use Structured Factorization Machine with a binary output. In this case we would have least two feature groups-one of the feature groups is the label that we want to predict, and the other group(s) is the input from which we want to make the prediction. The output indicates whether the label is associated with the input (y = +1), or not (y = 0). The datasets we use for our labeling experiments only contains positive labels, thus for each training example we sample a set of negative labels equal to the number of positive labels. We choose one of the following sampling strategies according to the best validation error, in each case excluding the actual positive labels for each training example -(i) uniformly from all possible labels, or (ii) from the empirical distributions of positive labels. Other sampling strategies have been proposed (Rendle et al., 2009; Rendle & Freudenthaler, 2014; Anonymous, In submission) .

4. EMPIRICAL RESULTS:

We now address our working hypotheses. For all our experiments we define a development set and a single test set which is 10% of the dataset, and a part of the development set is used for early stopping or validating hyper-parameters. Since these datasets are large and require significant time to train on an Nvidia K80 GPU cluster, we report results on only a single training-test split. For the labeling and classification tasks we use evaluation metric called Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC). Since we only observe positive labels, for such tasks in the test set we sample a labels according to the label frequency. This ensures that if a model merely predicts the labels according to their popularity, it would have an AUC of 0.5. A caveat of our evaluation strategy is that we could be underestimating the performance of our models-there is a small probability that the sampled negatives labels are false negatives. However, since we apply the evaluation strategy consistently across our methods and baselines, the relative difference of the AUC is meaningful. We choose the AUC as a metric because it is popular for both classification and ranking problems. For the regression tasks, we use MSE as the evaluation metric. In preliminary experiments we noticed that regularization slows down convergence with no gains in prediction accuracy, so we avoid overfitting only by using early stopping. We share most of the code for the experiments online1 for reproducibility.

4.1. IS THE FACTORIZATION HYPOTHESIS TOO STRONG FOR LARGE FEATURE SETS?:

Factorization algorithms are desirable for smaller feature sets (e.g, users and items) that have shared latent structure, because a more parsimonious model is less prone to overfitting. We now investigate if relaxing the factorization constraints for models with larger and more heterogeneous features is useful. We do this by comparing deep-out and a shallow structured models under the same conditions. For reproducibility details on our grid-search see Appendix A.1. We test this on classification, regression, and collaborative filtering (with implicit feedback prediction) tasks, respectively:

• Subscription prediction We predict the binary outcome of whether a marketing campaign is successful using the UCI Bank Marketing Data Set (Lichman, 2013; Moro et al., 2014) . This is a small dataset with only 45,211 observations. We use 17 categorical and real-valued features.

• Airline delays We predict the real-valued delay of american flights using the 2008 RITA dataset2 . using only 8 categorical features. This dataset is 7 million rows.

• Course prediction On this collaborative filtering task with implicit feedback, we use partial study plans collected from a company that has an educational-technology presence.

We only consider data from students with three or more course enrollments, and courses that have had at least enrollments from 10 students. The resulting dataset contains approximately 36 million enrollment instances from roughly 4.7 million students and 2,578 courses (out of 2,930 that exist on the ontology). For sampling implicit feedback, we sample the course and subject variables together. The features we use are all categorical: student id, course id, subject level 1, subject level 2, subject level 3, university id, student year (discretized), and data source types.

For both shallow and deep models, we define the groups the same way: we group each continuous feature by itself; we group the multiple dimensions of the one-hot encoding of categorical variables together. This way, for example, the 12 dimensions that are used to encode a "month" feature don't interact with each other. Table 1 summarizes our results. Adding a handful of deep-out parameters is very effective for improving forecasting quality or training time. For example, shallow factorization does not do much better than random chance in the subscription prediction, but the deep-out approach improves the AUC by roughly 35%. We hypothesize that this dataset has features with little shared structured. On the courses dataset, the deep method is almost twice as fast as the shallow method, with some modest improvement on performance. 

In this section, we focus on natural language processing tasks. Table 2 provides descriptive statistics of the datasets that we consider. To choose the feature extraction function φ for text, we use a Convolutional Neural Network (CNN) that has been shown to be effective for natural language tasks (Kalchbrenner et al., 2014; Weston et al., 2014) . In Appendix A.2 we describe this network and its hyper-parameters. Instead of tuning the hyper-parameters, we follow previously published guidelines (Zhang & Wallace, 2015) .

4.2.1. CAN DEEP-IN FM BE USED TO ESTIMATE ITEM EMBEDDINGS FROM TEXT?:

We experiment on two datasets to predict a label from text, obtained from an educational technology company: 1. Concept labeling. This dataset contains passages of texts related to topics taught at university courses. Subject matter experts annotated them with concepts. Our task is to design a model that can infer concepts from text.

2. Skill labeling. This dataset contains job posting texts annotated with the skills that are deemed to be useful by hiring managers. Thus, we try to predict the skills required for a job, particularly those that are not explicitly mentioned in the job posting or description.

For each task, we build two feature groups: the labels and the document text. The document texts are represented as a sequence of one-hot encoded words. As we describe in § 3.2, it is reasonable to use Matrix Factorization and ignore the item descriptions. Table 3 shows that Structured Deep-In Factorization Machine outperforms this strategy. Unfortunately, we cannot compare to Factorization Machine because it is intractable with a large vocabulary, as it would lead to interactions that scale quadratically in the vocabulary size (100,000 in our case).

4.2.2. IS DEEP-IN FM EFFECTIVE FOR COLD-START PREDICTIONS?:

Matrix factorization cannot generalize to unseen items. Instead, we compare to Collaborative Topic Regression (CTR-Wang & Blei ( 2011)), a method with an open-source Python implementation3 that leverages text features.

Because the implementation we found is relatively slow, we cannot test them on the concept and skill datasets. Instead, we use the CiteULike dataset which consists of pairs of scientific articles and the users who have added them to their personal libraries. We use it to predict users who may have added a given article to their library. We compare the performance of Structured Deep-In Factorization Machine with CTR using pre-defined cross-validation splits4 . We use 1% of the training set for early stopping.

For CTR we use the hyper-parameters reported by the authors as best, except for r which we found had a significant impact on training time . We only consider r ∈ {5, 10, 15} and choose the value which gives the best performance for CTR (details in Appendix 6). On the warm-start condition, CTR has an AUC of 0.9356; however, it shows significant degradation in performance for unseen documents an it only performs slightly better than random chance with an AUC of 0.5047. On the other hand, Deep-In FM achieves AUC of 0.9401 on the warm-start condition, and it only degrades to 0.9124 on unseen documents. For completeness, we also tested Deep-In FM on cold-start prediction for the Concepts and Skills dataset, on 1% of documents unseen during training. We obtained AUC of 0.92 and 0.67, respectively, which are within 3% of the warm-start results. Deep-In FM can also be trained over ten times faster, since it can leverage GPUs. 5 We also note that we have not tuned the architecture or hyper-parameters of the feature extraction function φ for each dataset and greater improvements are possible by optimizing them.

4.2.3. COMPARISON WITH ALTERNATIVE CNN-BASED TEXT FACTORIZATION:

We now compare with a method called DeepCoNN, a deep network specifically designed for incorporating text into matrix factorization (Zheng et al., 2017) -which reportedly, is the state of the 

5. CONCLUSION:

We present a general purpose method for factorizing large feature sets; we demonstrate it in several applications, such as using text to enable prediction for unseen items and circumvent the cold-start problem. Future work may soften our requirement of domain knowledge-in general, our methods require feature groups and feature extraction functions defined by experts. We did not pursue an exhaustive comparison with previously published methods; for example, there are other algorithms that rely on Bayesian optimization (De Campos et al., 2010) to infer the item embeddings from text which we did not benchmark. Although we apply our methods on six datasets altogether, further experimentation may be able to situate under which conditions our methods are effective.

Our methods generalize previously published single-purpose neural networks. For example, TagSpace (Weston et al., 2014) is a very successful method, but it is limited to a single textual feature. With the correct feature extraction function, Structured Deep-In Factorization Machine can be used to implement a TagSpace model.

Compared to previous general-purpose approaches, our work makes less assumptions about the training data and allows more flexibility. We provide evidence that the factorization hypothesis may be too restrictive-when relaxed we see higher predictive accuracy with a dramatic improvement of training speed. We show experimental results outperforming an algorithm specifically designed for text-even when using the same feature extraction CNN. This suggests that the need for ad-hoc networks should be situated in relationship to the improvements over a general-purpose method.

To the extent of our knowledge, our work is the first to propose a general purpose factorization algorithm that enables efficient inference on arbitrary feature sets.

A APPENDIX: REPRODUCIBILITY DETAILS A.1 HYPER-PARAMETER SETUP FOR STRUCTURED DEEP-OUT FACTORIZATION MACHINE EXPERIMENTS For the courses dataset, our experimentation strategy killed experiments that took too long. In practice, that means that some shallow experiments with batch size of 5,000 observations were not executed. Here we describe the details of the feature extraction function φ used in our experiments for labelling in §4.2.1 and §4.2.2. An overview of the network is given in Fig. 3 . We choose the most popular words of each dataset to build a vocabulary of size n, and convert the words of each document to a sequence of length t of one-hot encodings of the input words. If the input text is shorter than t, then we pad it with zeros; if the text is longer, we truncate it by discarding the trailing words. Therefore, for a vocabulary size n, the input has dimensions t × n, and this matrix is then passed through the following layers:

1. We use an embedding layer to assign a d-dimensional vector to each word in the input passage of text. This is done through a d × n-dimensional lookup table, which results in an t × d matrix.

2. We extract features from the embeddings with functions called convolutional filters (LeCun et al., 1998) (also called feature maps). A convolutional filter is simply a matrix learned from an input. We learn f filters that are applied on groups of m adjacent word embeddings, thus each of our filters is a d × m matrix of learned parameters. Filters are applied by computing the element-wise dot product of the filter along a sliding window of the entire input. The resulting output for each filter is a vector of length t -m + 1. We also apply a ReLU activation to the output of each filter. 3. Consider the case of inputs of different lengths. For very short texts, the output of the filters will be mostly zero since the input is zero-padded. To enforce learning from the features of the text, and not just its length we apply a function called 1-max pooling to the output of the filters: from the t -m + 1 output vector of each filter, we select the maximum value. This yields a vector of length F , a representation of the passage which is independent of its length.

:

https://goo.gl/zEQBiA

http://stat-computing.org/dataexpo/2009/the-data.html

https://github.com/arongdari/python-topic-model

For warm-start we use https://www.cs.cmu.edu/ ˜chongw/data/citeulike/folds/ cf-train-1-items.dat and for cold-start predictions, we use the file ofm-train-1-items.dat and the corresponding test sets for each

Deep-In FM and MF were trained on an Nvidia K80 GPU, while CTR was trained on a Xeon E5-2666 v3 CPU.

https://www.yelp.com/dataset/challenge

Figures and Tables:

Figure 1: Network architectures for factorization models. The white clouds () represent one or more factorized deep layers, and the dark clouds ( ) softens the dot product constraints.

Figure 2: Comparison of how factorization may use item descriptions features.

Figure 3: Feature extraction network used for labelling tasks. We use f=1000 convolutional filters each of width 3 (words)

Results of softening the factorization assumption. The reported time is the number of minutes it takes to train all of the hyper-parameter combinations (except for the course dataset that we only report the best model because the grid-search is adaptive)

Table content: <table><tr><td>Dataset</td><td colspan="4">Shallow Performance Time (mins) Performance Time (mins) Deep Out</td></tr><tr><td colspan="2">Subscription (AUC) .54</td><td>1.2</td><td>.72</td><td>1.8</td></tr><tr><td>Airline (RMSE)</td><td>1,349.47</td><td>177.2</td><td>1,337.0</td><td>38.7</td></tr><tr><td>Courses (AUC)</td><td>.79</td><td>98*</td><td>.80</td><td>54.9*</td></tr></table>

Details of labelling task datasets

Table content: <table><tr><td/><td>Number of</td><td>Total number</td><td>Mean labels</td><td>Mean document</td><td>Negative label</td></tr><tr><td/><td>documents</td><td>of labels</td><td>per document</td><td>length (words)</td><td>sampling strategy</td></tr><tr><td>Concepts</td><td>42,710</td><td>186,215</td><td>117</td><td>2,117</td><td>Uniform</td></tr><tr><td>Skills</td><td>544,388</td><td>25,299</td><td>72</td><td>276</td><td>Empirical</td></tr><tr><td colspan="2">CiteULike users 16,980</td><td>5,551</td><td>12</td><td>195</td><td>Empirical</td></tr></table>

Labeling results for warm-start documents customer ratings when textual reviews are available. For Deep-In FM we use the same feature extraction function (see Appendix A.3 for details) used by DeepCoNN. We evaluate on the Yelp dataset 6 , which consists of 4.7 million reviews of restaurants. For each user-item pair, DeepCoNN concatenates the text from all reviews for that item and all reviews by that user.The concatenated text is fed into a feature extraction function followed by a factorization machine. In contrast, for Structured Factorization, we build 3 feature groups: item identifiers (in this case, restaurants), users and review text.Table 4 compares our methods to DeepConn's published results because a public implementation is not available. We see that Structured Deep-In Factorization Machine provides a large performance increase when comparing the reported improvement, over Matrix Factorization, of the mean squared error. Our approach is more general, and we claim that it is also more efficient. Since DeepCoNN concatenates text, when the average reviews per user is nu and reviews per item is ni , each text is duplicated on average ni × nu times per training epoch. In contrast, for Deep-In FM each review is seen only once per epoch. Thus it can be 1-2 orders of magnitude more efficient for datasets where ni × nu is large.

Table content: <table><tr><td/><td/><td/><td colspan="2">Table 4: Yelp rating prediction</td></tr><tr><td>Model</td><td colspan="2">Concepts AUC Train mins. AUC Train mins. Skills</td><td>MSE</td><td>Gain over MF</td></tr><tr><td colspan="2">Deep-In F.M. 0.92 1,876 Matrix Fact. 0.88 64</td><td>0.65 3,256 0.61 8</td><td colspan="2">Matrix Factorization 1.561 -Deep-In FM 0.480 69.2 % DeepCoNN 1.441 19.6 %</td></tr><tr><td colspan="2">art for predicting</td><td/><td/></tr></table>

Grid search hyper-parameters

Table content: <table><tr><td>Dataset</td><td>Mini-batch size</td><td colspan="2">Embedding dimension Deep-Out Activation</td></tr><tr><td colspan="2">Subscription 20,000; 40,000</td><td>5; 10; 20</td><td>ReLU</td></tr><tr><td>Airline</td><td>1,000; 10,000</td><td>2; 5; 10; 20</td><td>linear</td></tr><tr><td>Courses</td><td colspan="2">5,000; 10,000; 30,000; 40,000 30;50;70</td><td>linear</td></tr><tr><td colspan="3">A.2 FEATURE EXTRACTION NETWORK FOR NATURAL LANGUAGE</td><td/></tr></table>

Peer review (iclr2018):
 This paper presents a method for matrix factorization using DNNs. The suggestion is to make the factorization machine (eqn 1) deep, by grouping the features meaningfully (eqn 5), extracting nonlinear features from original inputs (deep-in, eqn 8), and adding additional nonlinearity after computing pairwise interactions (deep-out, eqn 7). From the methodology point of view, such extensions are relatively straightforward. As an example, from the experimental results, it seems the grouping of features is done mostly with domain knowledge (e.g., months of year) and not learned automatically. The authors claim the proposed method can circumvent the cold-start problem, and presented some experimental results on recommendation systems with text features. While the application problems look quite interesting, in my opinion, the paper needs to make the context and contribution clearer. In particular, there is a huge literature in collaborative filtering, and I believe there is by now sufficient work on collaborative filtering with input features (and possibly dealing with the cold-start problem). I think this paper does not connect very well with that literature. When reading it, at times I felt the main purpose of this paper is to solve the application problems presented in experimental results, instead of proposing a general framework. I suggest the authors to demonstrate their method on some well-known datasets (e.g., MovieLens, Netflix), to give the readers an idea if the proposed method is indeed advantageous over more classical methods, or if the success of this paper is mostly due to clever processing of text features using DNNs. Some detailed comments: 1. eqn 4 does not indicate any rank-r factors. 2. some statements do not seem straightforward/justified to me: -- the paper uses the word "inference" several times without definition -- "if we were interested in interpreting the parameters, we could constrain w to be non-negative ... ". Is this easy to do, and can the authors demonstrate this in their experiments and show interpretable examples? -- "Note that if the dot product is replaced with a neural function, fast inference for cold-start ...". 3. the experimental setup seems quite unusual to me: "since we only observe positive labels, for such tasks in the test set we sample a labels according to the label frequency". This seems very problematic if most of the entries are not observed. Why cannot you use the typical evaluation procedure for collaborative filtering, where you hide some known entries during model training, and evaluate on these entries during test?