Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the problem of learning a descriptor for 3D shape. The two main contributions of the paper are related to using a mechanism for detecting hard examples for training the descriptor, while also doing it without requiring labels for this procedure. This is done with a combination from techniques inspired by adversarial learning and contrastive learning. The authors compare with a number of methods on the ModelNet40 benchmark.
main_review: - I find the main method of the paper interesting and practical. I particularly like the idea of identifying and rendering challenging viewpoints on the fly, compared to the more static setting of using a pre-rendered and pre-specified set of views adopted by many other works. The combination with contrastive learning is useful to avoid the extra requirement of class labels.

- Regarding rendering new views on the fly, I was wondering if there is a significant computational overhead when it comes to rendering these views, since this happens online during training? Although I like the idea, I can see how this can make training slow.

- In fact, how does this approach compare with just performing extreme data augmentations without mining hard negatives? I think this could be an interesting setting to check for the ablation study. In general, I think, that there are more choices that could be ablated in the main ablation study, instead of just the number of views (e.g., potentially a different hard negative mining method?).

- I like the presentation of the paper overall (e.g., the paper is clearly written, Figure 2 gives intuition about the type of hard examples that the method discovers), but there are quite a few typos (I list some of these below), so I would encourage the authors to do a more thorough proofread to eliminate those.
pg 5 : some wrong pointers leading to ??
pg 5 : "we can reformulate Equation" -> equation number missing
pg 6 : "a PyTorch3D" -> delete a
pg 6 : one-the-fly -> on-the-fly
pg 7 : we random rotate -> we randomly rotate
pg 7 : mash -> mesh
pg 7 : feces -> faces
pg 7 : Our MVCNN -> Our shouldn't be capitalized
pg 8 : "In this experiment, test the rotation" -> In this experiment, we test the rotation

- Evaluation happens on relatively saturated benchmarks, but quantitative performance is solid.

- The citations for the methods in Table 1 are missing. It would be much easier for the readers if the citations were also included in the Table, instead of only the names of the methods.
summary_of_the_review: All in all, I think this is an interesting paper, and the method could be applied in other settings as well. In fact, I would like to hear from the authors if they have other ideas for future extensions, because ModelNet is more of a simpler toy setting. I'm currently giving a Weak Accept rating, but I'm very interested in reading the response of the authors and the other reviews.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses a method to learning a 3D shape descriptor. They start with meshes of objects, which they render using a differentiable renderer. They pass the rendered images to a feature extraction network $f_{\theta}$ and obtain a feature embedding. This embedding is used in the contrastive learning procedure. Together with the "real" images, they use adversarially perturbed images. The perturbation parameters include rotation angles and random non-linear transformations. At test time they use many images of the object to get object embedding. 

The paper is evaluated on one dataset in the main paper and on another one in the supplement. Two problems are considered: object classification and object retrieval. According to the tables the description can bump the accuracy by a couple of percent. 
main_review: The area in which the present paper contributes is important and interesting. The paper proposes an unsupervised descriptor which seems to be efficient. 

However, I believe in the current form the paper is not ready for a publication. 

**1. The presentation is lacking clarity and details.** For example, in section 3.1. Equation 2 uses $t(\cdot)$ which is not defined until the self-supervised adversarial view generation paragraph. Then $f_{\theta}$ is introduced prior to eq 2 but not used before eq 3, differential renderer is introduced twice before eq 1. There are missing pointers to sections and equations, notation is not strict $Iadv$ vs $I^{adv}$. All these issues undermine the quality of the paper. Then, in the first sentence of section 4, they say they evaluate on 3 dataset, while in fact they do so on two. Table 1 misses references to baselines, what is VoxNet?. The reader can find VoxNet in references and in supplement, but the name VoxNet is not used in the paper apart from the table 1. Furthermore, important implementation details are missing. It's not clear how they integrate into GVCNN and MVCNN, I guess there can be many ways. I believe, this single concern is sufficient to recommend rejection and encourage resubmission. 

**2. Unsupervisedness of the approach.** I cannot really name the current approach unsupervised for object classification at least. MVCNN and GVCNN use labels and it's not clear how it's the descriptors are integrated, whether they're finetuned or anything.

**3. Ablation of adversarial views.** It's not totally clear from the paper if the method is better than just using random augmentations and random rotations, or somehow uniformly distributed.
summary_of_the_review: Overall, I believe the paper is below the bar
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel method that combines contrastive learning and an adversarial method to improve multi-view  approaches for shape classification and retrieval in a self-supervised way. The authors introduce adversarial views that enable to explicitly train the network on challenging views of a shape. The method is trained in a self-supervised way and does not require any label.
main_review: ### Pros
The method is based on two powerful and novel ideas: 
- using contrastive learning for learning efficient descriptors in a self-supervised way for this task. The self-supervision alone performs impressively well. 
- further improving the method by generating novel adversarial views.  

Remarkably, the method outperforms relevant baselines that are trained in a supervised way.

The idea of using adversarial views is particularly interesting: in addition to being efficient, adversarial views are conveniently interpretable contrary to imperceptible noise values in more common adversarial methods. They could be useful for other applications in 3D vision due to their interpretability.

The paper provides comprehensive experiments on multiple datasets while comparing to relevant baselines. Moreover, ablation studies are run extensively on the different parameters and losses. 

Up to the results section, the paper is well written and clear. The illustrations are clear and nicely describe the methods and results.

### Cons

The rule for bold numbers in the tables should be consistent. In table 1, the results in bold are the ones from the presented method (“ours”) (3D2SeqViews/Retrieval is not in bold in table 1). In table 2 the best results are in bold.

Starting from section 4.1 the writing feels rushed and is unclear. It would be good to perform a careful pass over this section. Some sentences are unclear and there are many typos.

A list of a few problems from the text:

Section 3.1:
- “Section ?? and Section ??” the section numbers are missing
- “we can reformulate Equation and make” the equation number is missing

Section 4.1:
- water eight 3D shapes in the mesh format -> watertight (?) 3D shapes in the mesh format 
- we random rotate -> we randomly rotate
- random scaling the mash object -> randomly scale the mesh
- “Connotations of all feces remain the same as origin”  this sentence is unclear and there is a typo
- water-tight propriety -> watertight property

summary_of_the_review: The authors propose powerful and novel ideas. They are clearly supported by the performance shown in the extensive experiments and ablation studies. I strongly suggest the authors to work on the writing, especially in the results section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors train a 3D object descriptor using existing multi view CNN architectures, but improve the training procedure with:
- augmentations in the form of 3D rotations,
- adversarial view sampling for forging hard examples during training,
- an unsupervised (contrastive) loss.

When trained with this procedure, CNNs are more performant on downstream tasks such as object retrieval or classification. 
main_review: Strengths:
- Exploiting a differentiable renderer to optimize for the viewpoints that fool the network the most seems to be effective, as displayed on Fig 2.
- The method is applied to 2 different backbone networks (MVCNN and GVCNN) and is thus generic.
- Metrics are indeed in favour of the proposed method compared to a supervised training, with no augmentation or adversarial samples.


Weaknesses:
- The idea of using gradient descent for optimising for the pose of objects that fool a CNN is not novel, and references should be added. For example, see *Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects* (Alcorn et al., CVPR 2019).
- The benefit of 3D augmentation and adversarial view sampling should clearly be investigated separately. The very last table of the supplementary (Tab. 6) is an attempt at ablating the adversarial view generation but should be made clearer (why is the caption mentioning a “supervised version”?) and moved to the main paper. The impact on final performance on the number of back propagation steps for the rotation matrix would also contribute to prove the method is effective. Similarly, Table 5 in the supplementary is an attempt at quantifying the effect of augmentations, but seems to be carried on already trained networks. What would be interesting is to train networks with different sets of augmentation and describe which augmentations are the most effective.
- Figure 1 is unclear, and might benefit from being split into 2. Currently it is trying to describe both contrastive learning and adversarial view generation, but both are mixed and therefore not clearly displayed. For example:
    - Why is a differentiable renderer needed for the augmented instance and the input 3D instance? Having a differentiable renderer is only useful for the adversarial instance.
    - The caption for “adversarial view generation” introduces the augmented 3D shape $t(x)$, while the adversarial view is not related to this augmentation.
    - Similarly, a “rotation matrix” is mentioned, but not introduced previously
- By contrast to the above points, some experiments are displayed in the main paper that are of less relevance, such as “4.4.1 effect of the number of views”: this does not help to quantify the benefits of the proposed method.
- In Fig. 2, it would be interesting to show how L is increasing during hard sample mining.

- The conclusion mentions a “novel multi-view pipeline”, but the pipeline/network is not novel. What is novel is a way of augmenting data during training, and using a contrastive loss on it.

Small missing technical details:
- What is the number of training epochs?
- What is the size of a descriptor?
- How is the task of object retrieval carried on?

Many typos, and quite verbose language that does not help having a smooth read:
- The abstract+intro should state more clearly what the contributions are. The “selection of shape-instance-dependent views towards the learning of more informative 3D shape representation” is not really informative.
- Fig. 1 “prospective”?
- 2.2 “required to generatING adversarial…”
- 3.1 “To this end, the direct adaptation of the adversarial sample generation method prevents us dynamically generating adversarial views due to field viewpoints” is unclear.
- In the very next sentence the differentiable renderer is introduced twice, and a task specify predictor is introduced but was never mentioned before.
- Equation (2) refers to t and x’ which were not introduced previously.
- “Self-supervised Adversarial Views Generation” refers to Sections ?? and ??, and mentions “proprieties” instead of “property”. “We can reformulate Equation ???”
- 3.2 : “generated from Equation 1, 3, 7” : pick one? Similarly in Algorithm 1: pick one equation and make it self contained?
- Algorithm 1 refers to L_final, which was not introduced previously.
- 4.1 Dataset “datasets that WERE acquired” / “WATERTIGHT 3D shapes” 
- 4.1 Implementation and training details: “We adopt a pytorch 3d” -> “We use pytorch3d” . “One-the-fly” -> on-the-fly
- 4.2 “Random mash” -> mesh. “Connoations of all feces” -> sorry what? “Propriety” -> property.
- End of Experiment results “ To this end, a significant advantage of our proposed method raises compared” : unclear.
- …
- The supplementary has broken references to the main paper (“Section ??”)
summary_of_the_review: The method seems to be working well, but is not very novel and lacks ablation studies to disentangle which components contribute to the performance improvements and how.
review_title: null

Meta-review: This submission received a diverging set of the final ratings: 6, 3, 6, 5. On the positive side, reviewers appreciated practicality of the approach and supporting empirical results. At the same time, all of them expressed concerns with the presentation (typos, unfinished sentences, inconsistent notations). Additional requests for clarifications and ablation studies have been mainly addressed in the rebuttal. The most skeptical reviewer did not participate in the post-rebuttal discussion, thus the final decision took that into account..

The AC has read the paper and verified that the minor technical issues pointed out by the reviewers have been fixed in the updated version (there are still a couple of typos remaining). This submission was further discussed between AC and SAC, as well as in the PC calibration meeting. Both AC and SAC agreed with the comment of Reviewer aAcK who pointed out that generating adversarial samples for mining hard examples has been explored in more general but related contexts before, which limits the novelty of this work to an application of a known idea to a particular domain (3D). At the same time, performance gains on the ModelNet40 dataset are marginal compared to the point cloud based baselines, while the proposed method still uses point clouds for generating adversarial views. In combination with other minor issues pointed out by the reviewers, and given that none of the reviewers was championing the paper, AC and SAC believe that the weaknesses of this paper at the end outweigh its strengths and do not recommend acceptance at this stage.
Decision: Reject