Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper builds on prior work on prototypical classification networks (more specifically, the work of Li et al. 2018) and additionally tries to include criteria such as orthogonality to enable applications such as fair classification. An application to hierarchical networks is also described though the details are very hard to understand. Experiments show that the resulting models are able to achieve reasonable fairness accuracy tradeoffs.
main_review: The paper attempts interesting problems but lacks on 2 major fronts: (1) It is not clear what the improvement over the existing work is, and if it is significant enough to merit acceptance at ICLR (2) the writing needs a lot of work to bring out motivation for different choices.

About the first point, the main contribution seems to lie in section 3.1, but most of the machinery here seems to be borrowed from the PCN work of Li et al. 2018. The additional contribution seems to be the concept subspace projection (if I understood correctly), whose motivation is not explained very well, and the addition of the alignment term in Eq. 1. The paper does not explain what is the additional value added by these terms over PCN.

Continuing from the previous point, the paper is very hard to understand. In the second paragraph of Section 3.1, there is some departure from PCN where some combinations of $p1$ and other prototypes are taken. The process is defined in a very handwavy manner and not clear what it is the formal mathematical operation performed here. How is this different from PCN and why was this step needed? Talking about digits and concept subspaces, do we have as many concept subspaces as the number of classes? If not, how is this number picked?

Moving on to the third paragraph of 3.1, the first few lines seem to be quite similar to PCN. However, at some point, a probability distribution is mentioned in connection with traditional softmax probabilities, but then yet another probability distribution is mentioned. It is not clear what the second distribution does. In the absence of formal equations, it is very difficult to understand what each component does. I would highly recommend describing each operation formally (in a sequential manner) and also adding a visualization like Figure 1 in the PCN paper to clearly convey the idea to the reader.

Fourth paragraph of 3.1 mentions two differences from PCN. Again, it is not clear what each of the differences achieves. Moreover, Figure 1 is neither described well in the main text nor in the caption, leaving the reader puzzled over what is happening in the figure. Instead of the regular autoencoder, a variational autoencoder is used, but again, the motivation is not clear. Other important details like the text above Equation 1, and the usage of KL diveregnce regularization term are skimmed over very quickly. The details of hierarchical classification setup in 3.4 are also glossed over quickly. The same happens in 4.2. For instance, what is meant by "adopting the conditional probability training loss introduced by Hase et al"?
summary_of_the_review: The contributions are not clear and the writing needs major work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel model — called Concept Subspace Network (CSN) — for both hierarchical and fair classification. The idea behind the network is to use sets of prototypes to define concept subspaces in the latent space defined by the neural network itself. The relationships between the subspaces can be manipulated at training time to enforce concept relationships (i.e., two concept subspaces are orthogonal if the concepts they represent are independent, while they are parallel if the concepts they represent are hierarchically organised). 
main_review: In this paper, the ideas are quite novel and mostly well presented, and the problem handled is significant. 

I have though some questions and some minor comments that I hope will be addressed in the final version: 
1. The way in which concept subspaces are defined is not clear to me. In the paper, the authors write: “Given a set of k prototypes in $R^Z$ , the prototypes define a subspace generated by starting at the first prototype, $p_1$, and adding all linear combinations of vector differences from $p_1$ to all other $p_i$; $i \in [2, k]$.” This is not clear to me, and it would be beneficial having a clearer example than the one given in the paper, in which it is not clear why we should obtain the plane $x-y$.
2. Also, in order to get a subspace, do you need the assumption that $k < Z$? 
3. In equation (2) the term $PCN(\cdot)$ is just defined as the loss introduced for PCN. Where is it defined?
4. The random baseline seems to achieve very high performance in tables 1 and 2. 
5. At page 7 the authors mention a global ordering of the nodes, how was such ordering decided? 


Minor comments: 
1. $Z$ in the figure 1 instead of $z$
2.  Add upward and downward arrows nearby the metric names to improve readability 
3. “Random” and “Rand” in Tables 1 and 2 
summary_of_the_review: The paper can be accepted if some clarifications are made
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a framework (that authors called the concept subspace network) using prototype-based representation controlling the alignment between two subspaces for the purpose of the classifier (fair or hierarch classification). 
main_review: Strength:
- The paper proposed a new prototype-based approach considering the relationship between two concepts (classification tasks) for fairness and hierarchical classification. 

Weakness
- The motivation of this paper is not clear to me. It would be helpful to understand the motivation by giving examples of major applications where both fairness and hierarchical classification should be considered. Also, is there any challenge when training a classifier using a regularization term regarding fairness in the existing hierarchy classifier training method?
- It is not clear that why the two subspaces should be orthogonal and parallel for a fair and hierarchical classifier, respectively. Specifically, in section 3.4: 
1. For fair classification, what are the two subspaces? is it correct that the two subspaces are for label classification and sensitive attributes classification (e.g., male vs. female), respectively? Then, does the orthogonal relationship between the prototypes for estimating the sensitive attribute and label prediction guarantee the independence of the actual sensitive attribute and label prediction?
2.  “In hierarchical classification, concepts are highly aligned and therefore parallel: the difference
between a toucan and a Dalmatian is similar to the difference between a generic bird and
dog.” --> In this example, why do parallel two concepts imply the difference between a toucan and a Dalmatian can be similar to the difference between a generic bird and dog? I think that the parallelism of two concepts is not related to the different relationships among prototypes. Then It is not clear that why two concepts should be parallel in the hierarchical classification. 

Questions: 
- How does one train a hierarchical classifier with 3 or more concepts? In hierarchical classification, I think there are at least 3 concepts (e.g., dog vs bird, dog species classification, bird species classification)
- In experiments, How was the parameter lambdas (in equation 2) chosen in the experiments?

Minor feedback: 
- Adding a figure describing the proposed architecture will help readers understand the framework. 

summary_of_the_review: The motivation of the paper and the intuition of the proposed approach is not clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The present paper proposes a novel architecture for prototype-based classification to support class hierarchies and fairness. In particular, hierarchies are supported by training the model for multiple classification problems jointly, each in its own subspace of the feature space, spanned by the respective prototypes. For fairness, the paper proposes to make the subspace for the classification between subgroups orthogonal to all other subspaces, such that any change in subgroup membership does not influence any other classification. In a series of experiments, the paper evaluates hierarchical classification and fairness separately as well as jointly and demonstrates equal or superior results to a state-of-the-art approach from the literature.
main_review: The paper's main strengths are:
* I found it particularly elegant to phrase both hierarchical classification and fairness in the same language, namely that of classification subspaces which are spanned by prototypes.
* The paper connects to a wide range of concepts, namely hierarchical classification, interpretability, and fairness, such that it is of potential interest to a wide range of researchers.
* The paper reports a wide range of experiments; so wide, indeed, that much experimental material had to be pushed to the appendix. I particularly appreciate the analysis of the hierarchies discovered by the prototype network and the comparison to the ground-truth hierarchy via edit distance.
* The paper is clearly written. I, for one, had no problem following along and would feel well equipped to reproduce the reported results.

The paper's main weaknesses are:
* The wide range of concepts discussed result in a certain lack of focus. Fairness, privacy, and causality are all mentioned but only discussed superficially. For fairness, this is particularly dangerous as readers may be mislead to believe that the proposed notion of orthogonality is sufficient for fairness. However, fairness has many meanings (as the paper acknowledges in the appendix) and only some of them are related to the proposed notion of orthogonality. Therefore, I would advise to revise references to fairness, privacy, and causality ind to mention explicitly that only a narrow notion of these terms is implemented by the proposed model.
* The related work fails to mention the historic roots of the prototype concept. I understand that many recent works in prototype networks make the same mistake but I would still advise to not continue it. Prototype-based classification has - to my knowledge - been pioneered by Kohonen in the late 1980s/early 1990s with his work on Learning Vector Quantization (refer to the review by Nova and Estevez, 2014; doi: [10.1007/s00521-013-1535-3](https://doi.org/10.1007/s00521-013-1535-3) ) and has since been extended in many directions, such as metric learning (Schneider et al., 2009, doi:[10.1162/neco.2009.11-08-908](https://doi.org/10.1162/neco.2009.11-08-908)), or probabilistic classification (Seo and Obermayer, 2003, doi: [10.1162/089976603321891819](https://doi.org/10.1162/089976603321891819) ). The latter extension should be of particular interest because the classification scheme is very similar to the one proposed in this paper.
* While the paper reports many different experiments, any single one seems relatively small with few data sets and (for hierarchical classification) few baselines. Further, I could not find information on the hyperparameter selection (e.g. how many prototypes and how strong the regularization strengths lambda were).

Overall, my recommendation is to accept this paper. While the paper could be more focused, make its own contribution and limitations more clearly, and experiments could be extended, I still believe that most flaws could be addressed with minor adjustments and that the core contribution of the paper is interesting enough to a wide range of scholars that publication is warranted.

Nonetheless, I would appreciate if the authors could help me to deepen my understand of the work by responding to two questions:

* I am not fully convinced that the projection into the plane spanned by the prototypes of a classification problem has any effect on the classification itself. If I understand correctly, the paper uses a softmax on the squared distances to all prototypes for classification (which is entirely reasonable). Now, let $D^2$ be the squared distance between a point $z$ and a prototype $p$, let $d^2$ be the squared distance between the projected point $\tilde z$ and the same prototype $p$, and let $h^2$ be the squared distance between $z$ and $\tilde z$. Since the distances form a right-angle triangle, we obtain $d^2 = D^2 - h^2$. This holds for any prototype in the same classification problem. Accordingly, all projected distances within one classification problem are merely the original distance minus a constant offset. This constant offset gets removed by softmax, anyways. So I would assume that the softmax probabilities are the same - no matter whether a point is projected or not.
* Why was the parity hierarchy used as ground truth for MNIST? Garnot et al. use a hierarchy based on visual similarity of the digits (e.g. 3 and 8). Wouldn't that be more natural?
summary_of_the_review: Overall, my recommendation is to accept this paper. While the paper could be more focused, make its own contribution and limitations more clearly, and experiments could be extended, I still believe that most flaws could be addressed with minor adjustments and that the core contribution of the paper is interesting enough to a wide range of scholars that publication is warranted.
review_title: null

Meta-review: This paper extends prototypical classification networks to handle class hierarchies and fairness. New neural architecture is proposed and experimental results in support of it are presented.

Unfortunately, reviewers found that paper in its current for is not sufficiently strong to be accepted at ICLR. Authors have made a significant attempt to clarify and improve the paper in their response. However, reviewers believe that contributions and motivation can be clarified further. We encourage authors to improve their work according to the specific suggestions made by the reviewers and resubmit.
Decision: Reject