Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: This paper strikes me as similar to the recent trend of "unsupervised" learning, but the paper is not really motivated in that way. The similar work of Wang and Gupta isn't mentioned in the intro, nor are many other recent works using weak supervision signals to train deep networks. It would be nice to hear more about the relationship of this work to those works. If object persistence hasn't been used as a supervision signal before then great. Although Wang and Gupta (and maybe others in this space) are explicitly motivated by object persistence -- their narrative is that it is useful for a network to try and learn that views of a 3d object as it rotated are still the same object.

Many of these works are robotics applications, e.g. Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge, Zeng et al. Section 6. 

Anyway, it would be nice to clarify how this work relates to these "supervision from multiple views of the same objects" works.
review_title: Similarity to "unsupervised" feature learning

final_score: rating: 7: Good paper, accept
review_content: On one hand this paper is fairly standard in that it uses deep metric learning with a Siamese architecture. On the other, the connections to human perception involving persistence is quite interesting. I'm not an expert in human vision, but the comparison in general and the induced hierarchical groupings in particular seem like something that should interest people in this community. The experimental suite is ok but I was disappointed that it is 100% synthetic. The authors could have used a minimally viable real dataset such as ALOI http://aloi.science.uva.nl .

In summary, the mechanics of the proposed approach are not new, but the findings about the transfer of similarity judgement to novel object classes are interesting.
review_title: interesting connections to human perception

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a model to learn across different views of objects.  The key insight is to use a triplet loss that encourages two different views of the same object to be closer than an image of a different object.  The approach is evaluated on object instance and category retrieval and compared against baseline CNNs (untrained AlexNet and AlexNet fine-tuned for category classification) using fc7 features with cosine distance.  Furthermore, a comparison against human perception on the "Tenenbaum objects” is shown.

Positives: Leveraging a triplet loss for this problem may have some novelty (although it may be somewhat limited given some concurrent work; see below).  The paper is reasonably written.

Negatives: The paper is missing relevant references of related work in this space and should compare against an existing approach.

More details:

The “image purification” paper is very related to this work:

[A] Joint Embeddings of Shapes and Images via CNN Image Purification. Hao Su*, Yangyan Li*, Charles Qi, Noa Fish, Daniel Cohen-Or, Leonidas Guibas. SIGGRAPH Asia 2015.

There they learn to map CNN features to (hand-designed) light field descriptors of 3D shapes for view-invariant object retrieval.  If possible, it would be good to compare directly against this approach (e.g., the cross-view retrieval experiment in Table 1 of [A]).  It appears that code and data is available online (http://shapenet.github.io/JointEmbedding/).

Somewhat related to the proposed method is recent work on multi-view 3D object retrieval:

[B] Multi-View 3D Object Retrieval With Deep Embedding Network. Haiyun Guo, Jinqiao Wang, Yue Gao, Jianqiang Li, and Hanqing Lu. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 12, DECEMBER 2016.

There they developed a triplet loss as well, but for multi-view retrieval (given multiple images of the same object).  Given the similarity of the developed approach, it somewhat limits the novelty of the proposed approach in my view.

Also related are approaches that predict a volumetric representation of an input 2D image (going from image to canonical orientation of 3D shape):

[C] R. Girdhar, D. Fouhey, M. Rodriguez, A. Gupta. Learning a Predictable and Generative Vector Representation for Objects. ECCV 2016.

[D] Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. Jiajun Wu*, Chengkai Zhang*, Tianfan Xue, William T. Freeman, and Joshua B. Tenenbaum. NIPS 2016.

For the experiments, I would like to see a comparison using different feature layers (e.g., conv4, conv5, pool4, pool5) and feature comparison (dot product, Eucllidean).  It has been shown that different layers and feature comparisons perform differently for a given task, e.g.,

[E] Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views. Francisco Massa, Bryan C. Russell, Mathieu Aubry. Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[F] Understanding Deep Features with Computer-Generated Imagery. Mathieu Aubry and Bryan C. Russell. IEEE International Conference on Computer Vision (ICCV), 2015.

review_title: Potentially interesting idea, but important references and baseline comparisons are missing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: I think learning a deep feature representation that is supervised to group dissimilar views of the same object is interesting. The paper isn't technically especially novel but that doesn't bother me at all. It does a good job exploring a new form of supervision with a new dataset. I'm also not bothered that the dataset is synthetic, but it would be good to have more experiments with real data, as well. 

I think the paper goes too far in linking itself to human vision. I would prefer the intro not have as much cognitive science or neuroscience. The second to fourth paragraphs of the intro in particular feels like it over-stating the contribution of this paper as somehow revealing some truth about human vision. Really, the narrative is much simpler -- "we often want deep feature representations that are viewpoint invariant. We supervise a deep network accordingly. Humans also have some capability to be viewpoint invariant which has been widely studied [citations]". I am skeptical of any claimed connections bigger than that.

I think 3.1 should not be based on tree-to-tree distance comparisons but instead based on the entire matrix of instance-to-instance similarity assessments. Why do the lossy conversion to trees first? I don't think "Remarkably" is justified in the statement "Remarkably, we found that OPnets similarity judgement matches a set of data on human similarity judgement, significantly better than AlexNet"

I'm not an expert on human vision, but from browsing online and from what I've learned before it seems that "object persistence" frequently relates to the concept of occlusion. Occlusion is never mentioned in this paper. I feel like the use of human vision terms might be misleading or overclaiming.


review_title: Nice form of supervision to explore

final_score: rating: null
review_content: Van der Maaten had a followup work to t-SNE (with Weinberger) called t-STE (t-Distributed Stochastic Triplet Embedding). It seems like it could be useful here, at least for visualization. Were you aware of it, and did you consider using it?
review_title: applicability of t-STE

Meta-review: The paper proposes a model for multi-view learning that uses a triplet loss to encourage different views of the same object to be closer together then the views of two different objects. The technical novelty of the model is somewhat limited, and the reviewers are concerned that experimental evaluations are done exclusively on synthetic data. The connections to human perception appear interesting. Earlier issues with missing references to prior work and comparisons with baseline models appear to have been substantially addressed in revisions of the paper. We strongly encourage the authors to further revise their paper to address the remaining outstanding issues mentioned above.
Decision: Accept (Poster)