Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The problem of utilizing all available information (across modalities) about a product to learn a meaningful "joint" embedding is an interesting one, and certainly seems like it a promising direction for improving recommender systems, especially in the "cold start" scenario. I'm unaware of approaches combining as many modalities as proposed in this paper, so an effective solution could indeed be significant. However, there are many aspects of the proposed architecture that seem sub-optimal to me:

1. A major benefit of neural-network based systems is that the entire system can be trained end-to-end, jointly. The proposed approach sticks together largely pre-trained modules for different modalities... this can be justifiable when there is very little training data available on which to train jointly. With 10M product pairs, however, this doesn't seem to be the case for the Amazon dataset (although I haven't worked with this dataset myself so perhaps I'm missing something... either way it's not discussed at all in the paper). I consider the lack of a jointly fine-tuned model a major shortcoming of the proposed approach.

2. The discussion of "pairwise residual units" is confusing and not well-motivated. The residual formulation (if I understand it correctly) applies a ReLU layer to the concatenation of the modality specific embeddings, giving a new similarity (after dot products) that can be added to the similarity obtained from the concatenation directly. Why not just have an additional fully-connected layer that mixes the modality specific embeddings to form a final embedding (perhaps of lower dimensionality)? This should at least be presented as a baseline, if the pairwise residual unit is claimed as a contribution... I don't find the provided explanation convincing (in what way does the residual approach reduce parameter count?).

3. More minor: The choice of TextCNN for the text embedding vectors seems fine (although I wonder how an LSTM-based approach would perform)... However the details surrounding how it is used are obscured in the paper. In response to a question, the authors mention that it runs on the concatenation of the first 10 words of the title and product description. Especially for the description, this seems insufficiently long to contain a lot of information to me.

More care could be given to motivating the choices made in the paper. Finally, I'm not familiar with state of the art on this dataset... do the comparisons accurately reflect it? It seems only one competing technique is presented, with none on the more challenging cold-start scenarios.

Minor detail: In the second paragraph of page 3, there is a reference that just says (cite Julian).
review_title: Interesting problem and good motivation, unconvincing solution architecture

final_score: rating: null
review_content: Earlier work on content-based recommendation focused on not just performance but also on the interpretability of the resulting models. Can you say anything about whether effective interpretations of the models latent dimensions can be gleaned when learning in this way?

Is Prod2Vec realistic in cold-start scenarios? Presumably co-purchase information wouldn't be available for cold-start products and thus an embedding couldn't be estimated.

I didn't understand why different sets of baselines seem to appear across the three tables, but maybe I missed some detail here.

review_title: pre-review questions

final_score: rating: 3: Clear rejection
review_content: This paper proposes combining different modalities of product content (e.g. review text, images, co-purchase info ...etc) in order to learn one unified product representation for recommender systems. While the idea of combining multiple sources of information is indeed an effective approach for handling data sparsity in recommender systems, I have some reservations on the approach proposed in this paper:

1) Some modalities are not necessarily relevant for the recommendation task or item similarity. For example, cover images of books or movies (which are product types in the experiments of this paper) do not tell us much about their content. The paper should clearly motivate and show how different modalities contribute to the final task.

2) The connection between the proposed joint product embedding and residual networks is a bit awkward. The original residual layers are composed of adding the original input vector to the output of an MLP, i.e. several affine transformations followed by non-linearities. These layers allow training very deep neural networks (up to 1000 layers) as a result of easier gradient flow. In contrast, the pairwise residual unit of this paper adds the dot product of two item vectors to the dot product of the same vectors but after applying a simple non-linearity. The motivation of this architecture is not very obvious, and is not well motivated in the paper.

3) While it is a minor point, but the choice of the term embedding for the dot product of two items is not usual. Embeddings usually refer to vectors in R^n, and for specific entities. Here it refers to the final output, and renders the output layer in Figure 2 pointless.

Finally, I believe the paper can be improved by focusing more on motivating architectural choices, and being more concise in your description. The paper is currently very long (11 pages) and I strongly encourage you to shorten it.

review_title: null

final_score: rating: null
review_content: 1. There are many modules in the system that are trained separately. Furthemore, fine-tuning for e.g. the image model appears to only take place on the FC7 layer. Jointly training all the components of the model would presumably improve performance, perhaps considerably. Did you try this? If not, why not?
2. For the joint product embedding, it was stated that a fully connected hidden layer that mixes the individual embeddings wasn't used in order to have a smaller number of parameters and increase comprehensibility. But presumably it could also perform better than the chosen approach. Did you try it?
3. Could you provide some more details on the way TextCNN was used? Was it used on full product descriptions, even when they consist of multiple sentences? Were titles and descriptions embedded separately?
4. In the table of results at the end, I didn't see any results for a model trained on both the books and movies data and tested on both in the cold start scenario. Did you evaluate this case?
review_title: Questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a method to combine arbitrary content into recommender systems, such as images, text, etc. These various features have been previously used to improve recommender systems, though what's novel here is the contribution of a general-purpose framework to combine arbitrary feature types.

Positively, the idea of combining many heterogeneous feature types into RS is ambitious and fairly novel. Previous works have certainly sought to include various feature types to improve RSs, though combining different features types successfully is difficult.

Negatively, there are a few aspects of the paper that are a bit ad-hoc. In particular:
-- There are a lot of pieces here being "glued together" to build the system. Different parts are trained separately and then combined together using another learning stage. There's nothing wrong with doing things in this way (and indeed it's the most straightforward and likely to work approach), but it pushes the contribution more toward the "system building" direction as opposed to the "end-to-end learning" direction which is more the focus of this conference.
-- Further to the above, this makes it hard to say how easily the model would generalize to arbitrary feature types, say e.g. if I had audio or video features describing the item. To incorporate such features into the system would require a lot of implementation work, as opposed to being a system where I can just throw more features in and expect it to work.

The pre-review comments address some of these issues. Some of the responses aren't entirely convincing, e.g. it'd be better to have the same baselines across tables, rather than dropping some because "the case had already been made elsewhere".

Other than that, I like the effort to combine several different feature types in real recommender systems datasets. I'm not entirely sure how strong the baselines are, they seem more like ablation-style experiments rather than comparison against any state-of-the-art RS.

review_title: null

Meta-review: The idea of combining many modalities for product recommendation is a good one and well worth exploring. However, the approach presented in this paper is unsatisfying, as it involves combining several pre-trained models, in a somewhat ad hoc manner. Overall a nice problem, but the formulation and results are not presented clearly enough.
Decision: Reject