Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: To advance the investigation on hierarchy and compositionality in representation learning methods, the authors proposed the Hangul Fonts Dataset, which is of known hierarchy, compositional structure and naturalistic variability. Several typical representation learning methods are evaluated on the dataset. Analysis on both unsupervised and supervised methods are given.

---

**Updated after the authors‘ response:**

The authors addressed my main concerns in the rebuttal thoroughly. Though there are still open issues, I believe publishing this work would encourage the community to work on these. Therefore, I raise my rating from 5 to 6.
strengths: 1. The explicit hierarchy and compositionality of the proposed dataset is beneficial for detailed investigations on representation learning.
2. The metric used for hierarchy and composition evaluation is quite interesting to me. 
3. The experiments conducted are thorough and clear, supporting the conclusions well.
4. The dataset could be easily extended with different fonts. And the authors provide corresponding codes.
weaknesses: 1. Section 2 is a little harsh to follow for those with little knowledge of Hangul. 

2. In my opinion, it would be better if the authors could provide statistical analysis for the claims in L104-107. For example, the KMeans clustering accuracy with `Initial Label` as representation and `Medial Label` as the generative variable. 

3. In Fig. 4C/D, there exists normalized accuracy lower than zero. To me it is unexpected since this implicates that the accuracy is lower than chance accuracy. Could the authors give some explanations on how did this happen?

4. I expect the KMeans clustering accuracy between different generative variables. I think the result could demonstrate the upperbound of the hierarchical structure information that could be learned by supervised methods, which could justify why the hierarchical structure information should be acquired by the representation learned by supervised methods.

5. It seems except the analysis involving 'all geometry', few analysis on representation and generative variables from different levels is conducted. I would love to see the Bag-of-Atoms features to be contained in the analysis in Fig. 4.
correctness: I believe the claims are correct. The dataset is constructed soundly. The experiment is well-designed and appropriate.
clarity: Though Section 2 is harsh to follow for those with little knowledge of Hangul, the other parts are generally clear.
relation to prior work: The relation to prior works has been clearly discussed.
additional feedback: 1. In Line 95, it seems that a period is missed before ``` 'There'.
review_title: Review on Submission 87

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors proposed a new Hangul Fonts Dataset in order to address the problem of lacking hierarchy and compositionality in common datasets, such as ImageNet. With the dataset, research can investigate hierarchy and compositionality in representation learning methods. Meanwhile,  the authors provide several empirical investigations with respect to the hierarchical and compositional structure of the dataset, showing it is beneficial to the areas.
strengths: 1. the new dataset is helpful to analyze the hierarchy and compositionality in representation learning.

2. The experiments are comprehensive and the results are reasonable.

3. The dataset is synthetic, which means that it can be extended easily .
weaknesses: 1. The part of sec 2 may be hard to read if someone is not familiar with Hangul.

2. The Hangul fonts dataset is not general or practical like ImageNet. There may be some limitations. For example, the conclusions with respect to hierarchy and compositionally based on this dataset may be not suitable for downstream works.
correctness:  The dataset is constructed in a sound way. The experiment is comprehensive and correct
clarity: The paper is well-organized, but may hard to read for those without some knowledge of Hangul fonts.
relation to prior work: Yes, the relation to prior works has been discussed clearly.
additional feedback: -
review_title: Hangul fonts dataset is not general like ImageNet

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces the Hangul fonts dataset, based on the fact that Korean Hangul enjoy a compositional structure. The dataset itself is of synthetic nature, based on a generative process that creates a total of up to 11k syllables under various fonts and several variations, such as rotation or style. To highlight the potential for investigation of how learned representations in (deep) machine learning are (in-)capable of capturing hierarchy/compositionality, an empirical analysis is showcased. The latter demonstrates the persisting gap in present unsupervised and supervised methods with respect to typically desired disentanglement of representations. 

The initial review is leaning towards acceptance, *conditioned on the fact that the public availability of code/datasets and their licenses is clarified first, and the related work section is improved*. 

----------------
**Update after authors' response**

I believe the authors have made updates that clarify some of the raised concerns (description of related work, licensing, accessibility, reproducibility with code). 

Whereas some of the feedback is naturally still to be tackled in future work, I think the advantages of introducing this dataset to the community outweigh the potential remaining open points. 

I have thus raised my score from 6 to 7. 
strengths: * The paper is generally well written and easy to follow. 
* Hangul serves as a very intuitive writing system to allow a meaningful analysis of compositionality and hierarchy in ML models. Even for the reader who is unaware of the Korean language, it is easy enough to follow the underlying rules of generation. 
* In my own humble opinion, analysing a dataset such as the presented one, could provide more meaningful insights than the typically conducted “disentanglement” analysis on dataset like Celeb, where models seem somewhat cherry picked and are labelled as disentangling when some concepts like “glasses” are empirically observed to be separated, while other factors remain densely entangled. 
* Whereas I initially was lacking some crucial information (e.g. descriptions required for reproducibility), the majority of my open questions were clarified when going through the detailed appendix. The Appendix is a nice complement to the main body and provides a lot of additional information. This includes the dataset sheet. 
* There is an interesting set of initial experiments that include both a DNN perspective, but also more traditional approaches like PCA, ICA or NMF. Now one the one hand, one could come to the conclusion that many of the presented insights are well known, such as the fact that training beta-VAEs does not actually result in desired disentanglement. However, I would personally argue that this is mainly a dataset paper, and actually seeing this analysis being done is positive, even if the outcome is somewhat expected, rather than just handwaving or referencing other papers. I believe most readers will appreciate the initial and rather thorough empirical analysis (including the additional content of the appendix) 
weaknesses: In my opinion the paper has two weaknesses and one open major concern. 

1. The related work section is focused on datasets like MNIST and ImageNet, but there is no mention or discussion of actual datasets surrounding the Korean language/Hangul. To the best of my knowledge, I personally cannot think of any dataset that has a similar compositionality in terms of generation at an atomic level, but there do exist multiple very interesting Hangul datasets already. E.g. one of them would be the HangulDB (and its various publications - see https://github.com/callee2006/HangulDB) with handwritten Hangul. Or IBM/Tensorflow’s hangul recognition (https://github.com/IBM/tensorflow-hangul-recognition).  The least the authors should do is cite prior works on Hangul databases and provide a discussion of related works on the topic. See also next point. 
2. Unfortunately I am no linguist, but from what I understand, a good portion of the 11k combinations of syllables does not actually find practical use in words in the practical language. From a machine learning standpoint, the investigation of compositonality/hierarchy of the dataset of course remains interesting. However, it would be nice to have a better discussion of limitations/implications for the actual language and ML systems trained for practical tasks like recognition/translation. I believe this point would be okay, if related works (see point) were discussed in sufficient depth and differentiated first. 
3. I find it concerning that the release of the dataset /code and licensing is written in future tense, with many explicit “TODO” placeholders in the appendix.  The authors write that the release is planned and licensing is currently pending institutional review. I honestly believe that this should have been done before submission. To the best of my understanding, the rationale behind the single-blind review process of the NeurIPS benchmarks and datasets track was to allow authors to make the datasets/resources publicly available already and sort these things out prior to the review process. Obviously, the strength of a dataset/benchmark paper lies in its public accessibility. I imagine the authors are naturally intending to do a public release (like the suggested Zenodo record). There already exists a GitHub link with code, but it would have been great to clarify potential licensing issues of code and dataset beforehand to make sure it is not too prohibitive (what if the paper gets accepted and the institutional review does not concur with the suggested license? Or the code never gets released). I will currently give the benefit of doubt, but this point should be clarified before a final decision is reached. 
* As an addition to point 3: it remains unclear whether the code for the conducted investigation will be released as well? This should be clarified, as open-sourcing the code for the baseline analysis would strengthen the contribution. Lines 228 states “code for training the networks and reproducing the figures will be posted publicly”. Again, given that the review is single blind, this should have been done before, instead of promising to do this upon acceptance. 

I am willing to raise my score if the related work section with respect to Hangul is improved and the public availability is addressed. 

correctness: I believe the dataset has generally been constructed in a sound way. The experiments seem rigorous, with statistical deviations reported and a meaningful set of chosen hyper-parameters, as reported in the appendix. 
clarity: The paper is well written and easy to follow. I did not find any paragraph that was unclear. There is a sufficiently detailed introduction to Hangul, so that I believe even the reader unaware of the language will be able to follow the idea. 
relation to prior work: The paper has a rather detailed list of references with respect to models, the concept of disentanglement and the empirical analysis part. However, as mentioned under weaknesses, the reported related work with respect to Hangul datasets and their investigation in ML is lacking. 
additional feedback: As a minor remark, there’s several typos and sentences to be corrected.

Main paper: 
* lines 79-21 “We find that deep unsupervised networks to do recover the hierarchical …”  -> ? 
* line 142: availble -> available 
* line 211: probabalistic ->  probabilistic 
* line 226: are listen -> are listed 
* lines 238-241: “For the supervised deep networks, the following methods were applied to the activations of every layer both before and after the ReLU nonlinearities. For the unsupervised ….” -> what methods? 

Appendix:
* line 125: availble -> available 
* lines 106 and 125 have a “TODO”. This should be replaced with the actual content.
review_title: The paper introduces the Hangul fonts dataset and provides an interesting empirical investigation with respect to the hierarchical and compositional structure of the dataset. The contribution is solid, but discussion of related work could further strengthen the contribution. Concerns over public availability need to be clarified.

Meta-review: This paper received borderline to positive reviews. However, after considering the points raised by the reviewers and reading the paper, the AC does not think the paper is above the bar of NeurIPS. The dataset itself is artificial and not well motivated, as it does not directly serve a practical application. Given this, it is not clear why Hangul fonts are special, and why they are preferred over composing some arbitrary shapes. 

In addition, the utility of a dataset (benchmark) is partially dependent on a standardized evaluation protocol. This paper is lacking in this aspect. Although the paper has performed analysis, it is not clear what is a well-justified protocol and metric in terms of evaluation disparate methods in terms of hierarchy and composition. In general, the protocol and metric should be simple and interpretable, while kmeans on the learn representations may shed some light, it is not clear how rigorous the conclusions are. 
Decision: Reject