Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes an algorithm for supervising networks for image classification and reconstruction with the object's hierarchical categories in mind. The claimed benefits are the improved generalizability and interpretability. The paper reports per-category-level analysis on the semantic image reconstruction task and retrieval on seen and unseen object categories.

I am currently leaning towards weak accept because I find the paper's claim and technical details generally convincing and simultaneously extracting low-level and high-level features trained using the hierarchical levels of categories is novel. Generalization to unseen categories tends to be a good proxy for real world performance and directly learning the high level categories is a useful idea for doing so.

Although I am leaning towards weak accept, I think this paper is close to borderline because the findings do not seem experimentally well validated. It would be more interesting to see Table 3 on multiple unseen categories instead of one special case per dataset. Another idea for experiments is doing cross-dataset evaluations where different datasets may have different leaf categories but shared high level ones. I think it may also be interesting to compare with a non-hierarchical retrieval model and then obtain their high-level prediction accuracy using the corresponding parent level categories.

The paper generally needs polishing. Minor typos I found:

Page 5: classificaiton, classifers
Page 6: intuitional->intuitive
Page 7: an significant
Page 8: leraning
Page 1: human
Figure 3: arbitary
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposed the hierarchical disentangle network (HDN) that leverages hierarchical characteristics of object categories to learn disentangled representation in multiple levels. Their coarse-to-fine manner approach allows each level to focus on learning specific representations in its granularity. This is achieved through supervised learning on each level where they train classifiers to distinguish each particular category from its ‘sibling’ categories which are close to each other. Experiments are conducted on four datasets to validate the method.  

Exploiting object hierarchy to learn disentangled representation is a promising direction but I lean towards rejecting this submission due to
1. No results on commonly used disentanglement metrics (e.g. see [1])
2. No comparison with existing supervised/unsupervised methods on disentangled representations (e.g. [2][3])
3. The needs for full supervision on each level and manually designed fixed hierarchy require labels for the full hierarchy and make it not applicable to many existing data. This probably is why the proposed approach did not work well for more complex datasets like ImageNet.


These also should be addressed:
1. The choice of adaptive instance normalization should be discussed. AdaIN could be used to account for small changes like color or local changes, but can it be used for larger and more global change (for example from animal category to human). If not, is it a limitation of this method?
2. Justification for several choices made in the method, for example in the form of qualitative/quantitative ablation studies - usage of local ‘brother’ categories, image/feature reconstruction losses
3. Can the metric in Table 1 prove disentanglement is achieved? What if E and G learned some way to fool the classifiers
4. Authors use conditional generative adversarial networks but it seems that there is no noise.
5. Discussion of failure cases. For example, the authors mentioned that the proposed approach did not work well for ImageNet. Why is this the case?


Minor comments:
- some citations are not properly formatted

[1] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, Locatello et al.
[2] Disentangled Sequential Autoencoder, Li and Mandt
[3] Exploring Disentangled Feature Representation Beyond Face Identification, Liu et al.


review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper studies the problem of learning disentangled representation in a hierarchical manner. It proposed a hierarchical disentangle network (HDN) which tackles the disentangling process in a coarse-to-fine manner. Specifically, common representations are captured at root level and unique representations are learned at lower hierarchical level. The HDN is trained in a generative adversarial network (GAN) manner, with additional hierarchical classification loss enforcing the disentanglement. Experiments are conducted on CelebA (attributes), Fashion-MNIST (category), and CAD Cars (category & pose).

Overall, this paper’s contribution seems quite outdated and presentation is not very clear.

(1) Learning hierarchical representation using GAN has been explored in Kaneko et al. 2018 but not even mentioned in the paper. 

Generative Adversarial Image Synthesis with Decision Tree Latent Controller. Kaneko et al. In CVPR 2018.

As far as reviewer understands, the bottomline for publication at ICLR is to demonstrate significant improvement/technical novelty compared to prior art. 

This paper should also compare against GANs or other state-of-the-art generative models with flat representation (especially on face generation) in terms of SSIM, inception score, and FID score. Without such comparisons, it is unclear what is the value of hierarchical representation proposed here.

-- Glow: Generative Flow with Invertible 1x1 Convolutions. Kingma and Dhariwal. In NeurIPS 2018.
-- Progressive Growing of GANs for Improved Quality, Stability and Variation. Karras et al. In ICLR 2018.
-- A Style-based Generator Architecture for Generative Adversarial Networks. Karras et al. In CVPR 2019.

(2) The interpolation results (see Figure 5) look a bit strange as the transition between last columns are not very smooth. Also, please provide details about this experiment: are you applying linear interpolation? What’s the interpolation parameter for each of the column?

(3) For image retrieval experiment, it is not clear if the proposed method is better than any state-of-the-art generative models with flat representations. One strong baseline is to use the latent representation of a pre-trained GAN model as comparison.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper proposes a method for solving the following problem: given an Image I from a labelled dataset with a label hierarchy as a tree of depth L, produce a set of vector representations {F_1, F_2 ... F_L}, such that a) the set can be used to reconstruct I as well as possible and b) each representation in the set only contains information about the label at the corresponding level in the label tree.

While I am not extremely familiar with work in disentangled representation learning, the authors claim is true to my knowledge that most work on disentangling factors does not explicitly take into account hierarchical structure as this work does. Therefore, this work appears novel and interesting to me. I will leave the assessment of the degree of novelty to other reviewers/AC who may be more familiar with the literature.

The approach is also effective, and the authors demonstrate through visualizations and experiments that the proposed model can be trained and accomplishes its objectives reasonably well. My overall decision is to accept this paper, but there are some improvements I'd like to see since I found it difficult to understand in some places. 

- There is repeated use of the term "granularity" in the abstract and Sec. 1 which is undefined. What, according to the authors, is the difference between having a hierarchical structure and multi-granularity? I suggest clarifying what is meant by this, or avoid using the term (used hierarchy instead).

- In Sec. 3.2, it appears that what is meant by R_l is actually the set {R_1, ..., R_L}. This would imply that the R's from different levels are randomly combined, and the number of representations combined is always L. Is this correct? In either case, what happens here should be made much clearer. It took me several readings to arrive at this interpretation.

- It took me a while to infer how the results in Figure 3 are generated. There is a sudden switch in Sec. 4.1 from model training details to its use for semantic translation, which was not explained.

Minor suggestions:

- Please use parenthetical citations throughout the paper where appropriate (use \citep{}) to avoid breaking the flow of reading.
- Pg. 1, last line: "us human" -> "humans"
- Pg. 2, line 1: "with others" -> "to others"
- Pg. 2, line 2: "hierarchy structure" -> "hierarchical structure"
- Pg. 2: "multi-granularity nature" -> "multi-granular nature" or "hierarchical nature"
- Pg. 7, line 2: "an significant" -> "a significant"
- Pg. 7, line 4: "At the last" -> "At the end"
- Pg. 7, "it can be reached" -> "it can be seen that"
- Pg. 7: "Table.4.1 gives the evaluation results" -? "Table 1 gives ..."
- Pg. 7, Please revise: "which is deserved to make more efforts"
- Pg. 7: "to applied" -> "to be applied"
- Pg. 8, line 1: Remove "quite"; typo in "leraning"
review_title: Official Blind Review #2

Meta-review: The authors propose a new method for learning hierarchically disentangled representations. One reviewer is positive, one is between weak accept and borderline and two reviewers recommend rejection, and keep their assessment after rebuttal and a discussion. The main criticism is the lack of disentanglement metrics and comparisons. After reading the paper and the discussion, the AC tends to agree with the negative reviewers. Authors are encouraged to strengthen their work and resubmit to a future venue.
Decision: Reject