Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presents a semi-supervised approach to learn the rotation of objects in an image. The primary motivation is that for rotation estimation datasets may not always be fully labeled, so learning partially from labeled and partially for unlabeled is important. The approach is to use a CVAE with a supervised loss and an unsupervised loss and to jointly train the network. Limited experiments that show performance are presented.

First, the paper solves a very interesting problem with potentially wide applications. The paper is reasonably well-written.

Unfortunately, I don't believe that the contributions of the paper meet the standards of ICLR. I justify my opinion below. The experiments are also very weak.

- While the high level goal of "pose estimation" is clear. Even after reading the paper multiple times, I did not understand the setting well. It appears like the paper looks at the problem of 2D orientation estimation of objects in images. However, this setting is restrictive and not very practical in reality. We mostly care about 3D pose estimation. It would have been good to see results on 3D rotations at the very least.

- Contribution: It is unclear to me what the primary contribution(s) of the paper is. The entire section on CVAE's and losses are quite standard in literature. The interesting part is in combining the supervised and unsupervised parts of the method for the task for pose estimation. But in the end this is a simple weighted loss function (equation 5). So I wonder what is the novelty? What are the new capabilities enabled by this approach?

- Related Work:

Implicit 3D Orientation Learning for 6D Object Detection from RGB Images, ECCV 18


- I would have loved to see a description of the differences in the loss functions (1) and (2). Perhaps this can help elevate the contribution more?

- I also missed justification of why the particular design choice is suitable for this problem? Would direct regression using a simple CNN work better?

- In equation (4), how are the two losses balanced?

- The dataset generation part is just confusing. ModelNet40 is rendered but only 2D rotation is predicted? What does 2D rotation mean for a 3D object?

- Could this method be tested on a dataset like dSprites (https://github.com/deepmind/dsprites-dataset) which has 3D rotations?

- Regarding experiments: I was disappointed to see no comparisons with other approaches or even a simple baseline. A CNN that directly regresses orientation could help put the tables and plots in perspective.

Overall, the problem is important (if lifted to 3D) with important applications. However, the paper does not say anything new about how to solve the problem and the experiments are weak. In its current state, I am unable to recommend acceptance.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes to employ conditional variational autoencoder (CVAE) to estimate the geometry of 2D rotations of objects given images partially labeled. Here, the label represents the geometry of the 2D rotation. The proposed method introduces two latent representation. z is the ordinal latent variable and r a latent representation for the rotations where the latent variable is defined in the 1-dimensional circle in R^2 so that it can naturally represent a hyperspherical latent space. 
The construction of the proposed CVAE is straightforward. For labeled images, the (evidence lower bound of the) loglikelihood of the image-rotation pairs is maximized. For labeled images. For labeled images, the (evidence lower bound of the) loglikelihood of the images is maximized. 

The decision of the reviewer of this paper is weak reject. The major reason is the lack of technical originality. The construction itself would be novel while each component (e.g., CVAE,  latent representation for the rotations, and semi-supervised construction of CVAE) have been already known.

Experimental results are not surprising but show that the presented method is useful to some extent. In a sense, it is a bit disappointing that we need 50+% images needed to be labeled to achieve < 20-degree error. One interesting observation of this paper is that more labeled images give better results than giving greater number of renders. Expansion to 3D rotations would be a good challenge. 


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper tackles the task of rotation estimation in a setting where both labelled and unlabelled examples are available for training. It proposes to learn a generative model of images (a VAE), where the ‘code’ is factored into a latent vector z and the object rotation r.  As in training a VAE, an image encoder that predicts the distribution over (z, r) and the generator are jointly trained, but with additional supervision on the distribution over r for the labelled examples.

I think the overall idea of learning a disentangled generative model in a semi-supervised setting is simple and elegant, and could in principle help leverage unlabelled data. However,  I do have some concerns regarding the specific contributions of this work, and several reservations about the experiments reported, and would overall argue for rejection.

Concerns:
1) The central empirical result stated is that using this approach allows one to reduce amount of labelled data by 10-20 %. First, even if valid, this is not a very convincing reduction in the amount of supervision. However, I feel this claim is not well-established by the experiments:

1a)  The paper should report a baseline with only using the loss in eqn 3 and only training the encoder (using various fractions of training data) to predict the rotation i.e. purely discriminative training without training a generative model. The current plots of performance vs fraction of labelled data don't mean much until compared to a similar plot for this baseline. The current results don't really highlight the importance of training the generative model or using the unlabelled data.

1b) I think there are some inconsistencies in performances reported in Fig 2. I assume the test set is same despite different training data, because the paper states "All of the trained models are evaluated with respect to the complete test set". In this regard, I am puzzled why using 100% labelled data with 16 renders is significantly better than using 50% labelled data with 32 renders -- these should imply similar number of labelled examples, and more unlabelled ones in the former.

2) While the discussion points to this, the paper would really benefit from having results in a real setting, in particular as pose estimation is a field with a lot of prior methods that have been shown to work in these settings. The current results are all in a setup with synthetic, unoccluded data, without background variation, equidistant camera uniformly sampled along a circle. The central idea of using a generative model would be much more difficult to operationalize in a realistic setting where these simplifying assumptions are not made, and I'd only be convinced about the applicability of the approach by results in that setting. As a possible setup, one case use many imagenet images in conjunction with labelled examples in PASCAL3D+ to try this approach.

3) The overall approach maybe novel in context of pose estimation, but this idea of learning a disentangled generative model is not, and there are several papers which do so with varying amount of supervision e.g. see [1] below for similar ideas, and pointers. While some details here may vary, in context of these prior works, I'd view this paper as mostly applying well-established ideas to a new task.

--
In addition to the above, I have a question regarding the training/testing data:
Q: The dataset description only states data was randomly divided - was this random division at an image level, or model level i.e. could different renderings of the same model be in train and test set?
--

[1] Learning Disentangled Representations with Semi-Supervised Deep Generative Models, NIPS 2017. Siddharth et. al.
review_title: Official Blind Review #2

Meta-review: This paper addresses the problem of rotation estimation in 2D images. The method attempted to reduce the labeling need by learning in a semi-supervised fashion. The approach learns a VAE where the latent code is be factored into the latent vector and the object rotation. 

All reviewers agreed that this paper is not ready for acceptance. The reviewers did express promise in the direction of this work. However, there were a few main concerns. First, the focus on 2D instead of 3D orientation. The general consensus was that 3D would be more pertinent use case and that extension of the proposed approach from 2D to 3D is likely non-trivial. The second issue is that minimal technical novelty. The reviewers argue that the proposed solution is a combination of existing techniques to a new problem area. 

Since the work does not have sufficient technical novelty to compare against other disentanglement works and is being applied to a less relevant experimental setting, the AC does not recommend acceptance. 

Decision: Reject