Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors develop notions of comprehensibility and transparency as indicator of explainability, and develop formal notions to introduce these as constraints during model training. The authors also propose a new technique for generating saliency maps. Subsequently, they show that inclusion of these constraints improves accuracy as well as the insertion/deletion metric for quantifying the quality of saliency maps.
strength_and_weaknesses: Baking explainability into models by somehow making them inherently interpretable is a relevant problem. The approach proposed in the paper seems sensible and the results look convincing. Some more thoughts:

1. I am not sure if the notions of comprehensibility and transparency are well-established. With that in mind, the presentation of these appears somewhat ad-hoc.

2. While the development appears logical, the end result is quite simple — instead of penalizing the L_p norm of the parameters in the loss function, penalize deviation from orthogonality instead. I wouldn’t be very surprised if a little more algebra reveals some relationship to L_p norm. However, simplicity itself is not something to criticize.

3. While I wasn’t very convinced of the value of the proposal during the development, the results look quite good. Specifically, the improvement in accuracy, improvement in explainability metrics, feature composability, and generation of adversarial examples are all well documented. The only sub-section that seems unsurprising is the verification of properties — orthogonal constraints lead to orthogonality, and the inherent reconstruction constraint leads to good reconstruction.

4. I am not quite sure if I can clearly argue why the technique produces good results. The constraints were developed for one set of objectives yet they somehow improve on another (better established) set as well. A discussion of that and authors’ thoughts on the possible explanation would improve the paper.

5. Minor errors: “possesses comprehensible and transparent simultaneously”, wrong double quotes in “relation”, “only and if only”, “equivalent whether to exert”, “benefits of the two constraints benefit of transparency”, add x-axis label to Fig 3a, “assumption f enough”.
clarity,_quality,_novelty_and_reproducibility: Fairly clear, good quality, novel without being ground-breaking, and appears reproducible especially once the code is released.
summary_of_the_review: Overall, while I think the formalization does reduce to straightforward constraints, the results are (surprisingly) good, and the community could benefit from wider dissemination. I would recommend the paper to be accepted at the venue based on the performance but I would have much appreciated any deeper insights.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces two methodological components: (1) BORT, a regularization for feed-forward neural networks term that encourages the various layers to implement a bounded orthogonal projection, (2) SAT, a saliency algorithm specifically designed for BORT-regularized networks.  The empirical evaluation highlights how BORT can improve accuracy of several models on MNIST and CIFAR10, and that BORT+SAT produces "clear" saliency maps with good deletion and insertion scores (which measure sensitivity of model predictions to pixels identified as relevant by the saliency maps).  In addition, the BORT+SAT combo facilitates acquiring features with input reconstruction capabilities and generation of adversarial examples.
strength_and_weaknesses: PROS
- Proposed idea is is based on a solid intuition.
- Formalization is clean and looks correct.
- No performance drop upon applying BORT - actually, quite the contrary.
- Text is mostly well written.

CONS
- Key element (SAT) is not well described, complicating evaluation of the results.
- Other aspects of the experiments are not well explained.
- Some terminological choices are a bit confusing.
- Relationship to existing approaches is not entirely clear.
- A subset of the results rely on guided backpropagation, which is known to be flawed.
- Only two smaller image classification data sets (not a huge deal).
clarity,_quality,_novelty_and_reproducibility: **Clarity**:  The text is generally well written and well structured.  English is good, with only few grammatical issues here and there.  Pictures are also quite clear.

The formalization is relatively clean and convincing.

I think that some key terminological choices could be improved.  The terms "explainability", "comprehensibility", and "transparency" have a far more general (and fuzzily defined) meaning - in English - than the mathematical properties defined and used here.  Specifically:

 - "Explainability" has as much to do with algorithmic aspects of the explanation process as it has to do with human-centric, psychological factors such as knowledge and cognitive biases.  As such, attempting to solve it formally is (currently!) not possible.  The authors should tone down some of their claims regarding the scope of their formalization (e.g., my honest opinion is that being able to map latent features back to input space is not quite enough:  a latent point might map back to an example that has very little semantic meaning for human observer, say, a garbled image resulting from a measurement mistake or an uninformative region of a valid image.  The BORT formalization does little - and *can* do little - to prevent this from happening, see also below.)

 - What the authors called "comprehensibility" is essentially the idea that weights should recover concrete examples, i.e., prototypes.  "Prototype-based" is a better and more well-known term compared to "comprehensibility".

 - What the authors called "transparency" means that latent variables can be mapped back to input space.  This sounds more like an approximate form of "invertibility" to me.

What is confusing is that all three terms (explainability, comprehensibility, transparency) are used with different meanings in the rest of the literature.  I do encourage the authors to rename them to something more unique and easily recognizable.

Another element I did not understand is why BORT is being marketed as an optimizer.  To me it really looks like a regularization term.  Not a huge deal, just slightly confusing.


**Quality**:  The idea underlying BORT is to turn a neural network into a stack of orthogonal projections, with individual weight vectors ending up resembling a set of diverse (i.e., orthogonal) examples.  The intuition is quite clear and reasonable, and it is very much in line with prototype-based models (see below).  BORT is overall quite straightforward, but this to me is actually a plus.

One aspect I would mention in the paper is that stacks of orthogonal (linear) projections are incapable of learning, because they do preserve information whereas generalization is by its own nature lossy.  BORT works regardless because, thankfully, architectural limits (e.g., non-square weight matrices) prevent BORT-optimized neural networks from acquiring perfect orthogonal projections.

I found the empirical evaluation not entirely convincing, for several reasons:

- The experiments in Sec. 4.1 and 4.2.1 are quite clear and are positive.

- Those in Sec. 4.2.2 are very compressed and hard to parse.  It is not clear to me how guided backpropagation (a saliency technique) is used to "reconstruct the input" (Fig 4) and what the output is meant to represent: an input or a saliency map?

- Another issue with that guided backpropagation is known to be insensitive to the weights learned by the model, see:

  Adebayo J, Gilmer J, Muelly M, Goodfellow I, Hardt M, Kim B. Sanity checks for saliency maps. NeurIPS 2018.

How does this impact Fig 4?

- The **worst** issue in the paper is the description of the SAT algorithm, which is very compressed and hidden in the experiments.  SAT, despite not having been mentioned at any point in the paper, turns out to be **integral** to BORT.  In fact, using more well-known saliency techniques with BORT-trained model is not ideal (see Table 3).  Alas, I could hardly understand what SAT does.  For instance, why the result needs to be averaged over multiple images:  the way it is described, SAT sounds like a deterministic procedure so I don't see the need for averaging.  I am very confused as to why the "backprojection" SAT procedure should result in a saliency map, and what the semantics of this map are meant to be, i.e., why it can be used as an explanation.  This makes it very hard to evaluate the saliency map experiments, which are the core of the paper.

- The decomposition and synthesis experiments are also overly compressed and hard to follow.

I think SAT should be introduce much earlier in the paper, as early as the abstract, and the reason why it is needed should be clarified.  Right now it is not clear why BORT - whose main aim is to improve interpretability -  does *not* really improve interpretability unless used together with SAT.  It is very hard to evaluate BORT in isolation.


**Novelty**:  The idea of learning orthogonal transformation is not new in ML, but to the best of my knowledge this it has not been used in the context of learning deep feed-forward neural nets for the purpose of explainability.

The authors do a good job at discussing related work from the white- and black-box models, but they completely neglect the recent literature on "gray-box" (or "concept-based") models, including:

  Prototype Classification Networks (PCNs):  Li, Oscar, Hao Liu, Chaofan Chen, and Cynthia Rudin. "Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions." AAAI 2018.

  Part-prototype Networks (PPNets):  Chen C, Li O, Tao D, Barnett A, Rudin C, Su JK. This looks like that: deep learning for interpretable image recognition. NeurIPS 2019.

  Concept Whitening (CW):  Chen, Z., Bei, Y. and Rudin, C.. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2020.

  GlanceNets:  Marconato E, Passerini A, Teso S. GlanceNets: Interpretabile, Leak-proof Concept-based Models. arXiv 2022.

Most of these works are somehow related to BORT:  (i) PCNs make use of autoencoders to map latent variables to input space, adding extra constraints to ensure that the reconstruction resembles a concrete example (in a vein reminiscent of BORT), (ii) CW anticipates the idea of leveraging orthogonality to enhance interpretability, although from a slightly different angle than BORT,  (iii) PCNs and PPNets work by learning prototypes, which seems very much the objective of the "comprehensibility" criterion introduced here;  (iv) GlanceNets do come with a sound formal definition of explainability, as does the work by Hazan et al (not a gray-box model, though):

  Wolf L, Galanti T, Hazan T. A formal approach to explainability. AIES 2019.

There are several other attempts at formalizing explainability out there.

I think that the authors should at the bare minimum mention the existence of gray-box models (especially those based on prototypes and orthogonalization, which are *very* close in spirit to BORT) and their relationship to the BORT desiderata, as well as highlight that there *are* alternative attempts at formalizing explainability.


**Reproducibility**:  The code is provided in the supplement.
summary_of_the_review: Explainability-enhancing regularizer based on per-layer orthonormalization + ad-hoc saliency map algorithm.  Alas, the latter is poorly described, complicating evaluation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new optimizer called Bort to improve model explainability (comprehensibility and transparency) with Boundedness and orthogonality constraints on model parameters.
strength_and_weaknesses: Strengths
+ Explainability is a critical and challenging field in ML, and the mathematical attempt proposed in this paper is encouraging.

Weaknesses
- Boundedness constraint seems similar to cosine similarity. Any difference?
- The discussion to treat CNN and Transformer as MLP is quite sketchy, e.g. "can be unified under one meta-structure, a multi-layer perceptron (MLP for short) with optional nonparametric operations. For example, the convolutional layer and the transformer layer additionally use folding/unfolding and the self-attention operation, respectively". Will parameter sharing and attention introduce extra complexity and inconsistency to the definition and analysis of comprehensibility and transparency?
-"the row vector wij in Wi". wij is a weight, right? How can a weight "look similar to a specific data pattern"? What does "look similar" mean? How can weights look similar to an input data pattern after many layers of non-linear transformation?
- The definition of comprehensibility is confusing. Is a semantic data pattern z a vector or scalar value? How will z be proportional to a weight wij? especially in later layers?
- The definitions of comprehensibility and transparency are debatable, and it is also a question whether comprehensibility and transparency are sufficient for complete explainability.
- Both boundedness and orthogonality constraints seem to limit the expressivibility of FNNs, any insight why they actually improve accuracy? some regularization effects?
- Although  a two-layer network is a universal approximator, proposition 1 is on  a two-layer linear network.
clarity,_quality,_novelty_and_reproducibility: Quality

This paper is technically solid. However, it is over-ambitious as over-simplifying explainability to two measures: comprehensibility and transparency, whose definitions are debatable. 

Experiments are not adequate as it only compares with CAM, a 2016 paper.

Clarity

The paper is written well in general. Please fix notations as pointed out above.

Originality 

The attempt of mathematical definition and analysis of explainability is novel and plausible.
summary_of_the_review: The approach of mathematical definition and analysis of explainability proposed in this paper is novel and plausible. However, the paper is over-ambitious and experiment is not adequate.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a regularizer which tries to make a network's behavior more interpretable by encouraging orthogonality. The reviewers are generally positive about this paper, remarking that it's a simple method that seems to give good results visually and in terms of insertion/deletion metrics. Reviewers did express some concerns about why the technique works, as well as the choice of baselines to compare against. Reviewers had various suggestions of related work, which I encourage the authors to add to the paper. (To this list, I would add "Adversarial robustness as a prior", by Engstrom et al., and past work on orthogonality constraints, such as "Sorting out Lipschitz function approximation," by Anil et al.) The low accuracy of some of the models (for both the proposed method and the baselines) makes me worry a bit about how successful the technique is for modern architectures. But overall, these concerns don't seem critical, and the reviewers generally feel like there is a useful contribution here. I recommend acceptance, and encourage the authors to account for the above points when revising the paper.
justification_for_why_not_higher_score: See above.
justification_for_why_not_lower_score: I've outlined some concerns above; I raised these with the reviewers, and they still favor acceptance. I don't see a reason for me to step in and be an activist AC.
Decision: Accept: poster