Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
This paper presents a robust multiclass classification "add on" scheme for deep neural networks called TAC (total activation classifiers) in which multiple hidden layers of a network are trained to output error correcting codes (ECOCs) that are used to identify the proper class of an example and guard against spurious predictions in the case of out-of-distribution (OOD) examples and adversarial examples designed to trick the underlying model. Because ECOCs require more bits to be correctly predicted when determining the class label, TAC-augmented models are claimed to be more robust. Specifically, with TAC, the hamming (or cosine) distance to the closest code in the code book can used as a measure of model confidence, and predictions with at least a certain distance can be flagged for review or rejected.

Toy experiments on CIFAR-10 show that TAC-augmented neural networks can be trained to output ECOCs at multiple network layers, and that the distances of predicted codes is discriminative at identifying adversarial examples (though not perfectly so).

Language model experiments on 3 intent-classification tasks show that TAC improves accuracy over and above the original pretrained RoBERTA and BERT++ models. And experiments on ImageNet data with a pretrained ViT model shows that TAC better identifies missed predictions than other confidence detection measures like the max-logit-score (MLS), maximum-softmax-prediction (MPS), and a recently published approach called "DOCTOR".

Finally, on an adversarial benchmark dataset (RobustBench), TAC performs well at identifying attacks in a gray-box scenario where the attackers know the inner workings of the base neural network, but do not have access to the classification codebook. In the full white-box scenario where attackers have the code book as well, TAC is not competitive in providing protection against adversarial attacks.

strength_and_weaknesses: Strengths:
- TAC is a simple extension that allows one to add safety and robustness to new and existing neural network models.
- Experiments with TAC show promise in terms of classification accuracy and mis-prediction detection across a range of datasets and misprediction penalty ranges.
- In gray-box adversarial settings, experiments show that TAC is effective at detecting adversarial examples.

Weaknesses:
- TAC is an empirical method with intuition, but not theory, justifying the use of ECOCs output by multiple network layers for preventing against spurious predictions. The paper's experiments are suggestive, but are not extensive enough to determine whether TAC provides robust and superior decision support across a wide range of tasks, and compared to other commonly used methods like temperature calibration.
- While the value-of-classification curve (VOC) was cool and something I have not seen before, many of the experimental results regarding misclassification detection rates are presented in a manner that is rather opaque and hard to interpret in real terms (see comments on paper clarity below).
clarity,_quality,_novelty_and_reproducibility: The main themes and methods of the paper are very clear. However, the presentation of experimental results can be improved. For instance, metrics like baseline model accuracy and TAC model accuracy are reported for the intent classification tasks, but not for the vision tasks (e.g., CIFAR-10). Or if they are presented, they are presented in a different format such as the plots of figures 6 and 11.

In the mis-prediction detection experiments, the "Det. AUCROC" metric does not seem to be described, nor does the rationale for reporting the detection rate at the point where the "true positive rate (TPR) equals the false negative rate (FNR)". Why not use the point at which the TPR and FPR are equal?

Similarly, the results for the RobustBench leaderboard in table 4 are not well described and hard to make sense of. Why are all the cells for the baseline methods left blank? And what does the multicolumn column header "Individual attacks from from Auto-Attack (%)" mean?

summary_of_the_review: Overall, this paper presents a simple and promising extension for adding robustness to both new and existing deep learning models. Theoretical justification is lacking, and some of the experimental results are hard to interpret in real terms. At the same time, some of the experimental results are compelling and show real improvements above commonly used SOTA baselines for misprediction and out-of-distribution identification. Therefore, I am in favor of accepting this paper to ICLR, but not strongly so.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the robustness of deep classification models, noting that out-of-the-box models can yield arbitrarily high-confidence predictions on out-of-distribution data. As a proposed remedy, “total activation classifiers” (TAC) are introduced to capture “class-dependent patterns” which are claimed to offer a reliable way to identify test data that do not conform to such patterns. Notably, the approach does not require access to out-of-distribution data at training time.

In more detail, TAC consists of a predictor mapping data onto the unit cube. The features are extracted from the neural network by taking “slices” from all intermediate activations which are then reduced (via summation) to 2-dimensional objects (e.g., spatial dimensions in convolutional architectures for images). The goal is to “make it so that groups of high-level features fire-up more strongly depending on the class of the input.” The sequence of all such slices is called the “activation profile” whose distance is measured to the classes. The class labels are represented as distinct binary codes; the set of binary codes is chosen in this way to maximise the “discriminability of its elements.” To train TAC, the binary-cross entropy loss is used to obtain class activation profiles which minimise the coordinate-wise distance to the corresponding class; variations on this loss are explored to improve training stability.

The overall TAC procedure can be applied to both “frozen” pre-trained models to obtain a secondary confidence score as well as trained “from scratch,” at a slight drop in accuracy (A.1). Thus, the recommended approach is to apply TAC to frozen models.
strength_and_weaknesses: Overall, this paper tackles an important problem and the experimental evaluations on robustness to adversarial attacks, ability to abstain, and out-of-distribution detection seem fairly convincing overall. However, I have a few concerns about motivation for the proposed approach as well as some lacking discussion (and perhaps experiments) of related prior work.

The approach is not well-motivated. In the introduction, the question is posed, “can we enforce this property rather than hope it emerges?”, but the reader is left wondering what property is actually under discussion. It would be helpful to include further discussion of the property of interest to motivate why it’s sufficient to target this to obtain “models that know that they don’t know.” 

The use of binary codes is motivated by the need to have well-separated targets for optimization. However, in S2.4 it is stated that the codes are randomly selected with coordinates being independent binary random variables. Do the codes actually need to be binary? 
clarity,_quality,_novelty_and_reproducibility: Although there are notable differences that make the proposed approach unique and interesting, there is more than a passing similarity to https://arxiv.org/pdf/2102.12967.pdf from ICLR 2022. This prior approach also involves extracting summary statistics from all layers of the neural network, and additionally has the benefit of proposing a principled statistical hypothesis testing framework for OOD detection.
summary_of_the_review: Overall, this paper tackles an important problem and the experimental evaluations on robustness to adversarial attacks, ability to abstain, and out-of-distribution detection seem fairly convincing overall. However, I have a few concerns about motivation for the proposed approach and there is an important omission in the discussion of related work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the use of L-length binary codes (TAC) for the use of multi-class classification. Instead of having a one-hot representation, the authors' approach is to produce a binary value from each sliced-up section of the neural network's activations (Figure 1). Models can be trained to output TAC from scratch or via post-hoc add-ons to existing classifiers. 
strength_and_weaknesses: Thanks for the authors for the hard work on this paper. 

Strengths
------------
- Appealingly simple idea
- Clear writing
- Strong performance
- Seems quite generally applicable

Weaknesses
----------------
- If I understand correctly, each of the binary codes is associated with a slice of the activations. However, activations in the earlier layers are generally worse at predicting the class (and perhaps one code in a vector of them). As such, shouldn't latter (in terms of neural network topology) codes be upweighted (in terms of loss computation) during training, during the distance computation (for OOD detection), and during inference? To assess whether the premises of my question is correct, it would be great to see a plot of (a) training error and (b) OOD distribution per layer: are earlier codes harder to learn than later ones and are they worse at OOD detection? You have something like this in Figure 2, but these are per-sample: how bad is the error vs depth overall? I think this statement in the appendix supports my concern: "Moreover, at testing time, we used only the deepest of the 13 layers to compute TAC’s confidence scores since that resulted in improved performance" - what fraction of the code is this? Did you have to do this kind of trimming for any other datasets? This is something important to understand and should be lifted out of the appendix.

- Sec 2.4 - but why *not* use Hadamard matrices? Is there any advantage in using random binary codes specifically?

- How easy is your method to adapt to multi-label classification tasks? How about regression? Would be nice to have a few sentences about this, but I'm not asking for any new experiments here.

- Table 2 - L1 is used for HWU64 and CLINIC150 but cos is used for ImageNet. It's noted in the appendix that you chose cos for ImageNet, Did you choose it on a held-out validation set? Is that the procedure you generally recommend? If so, please make explicit.

- The related work discusses error-correcting output codes. Why not compare to 1-2 of these methods that can be applied to neural networks? I understand they are not as efficient, but it's important to understand how good your performance is compared to previous similar ideas and trade that off with performance gains.
clarity,_quality,_novelty_and_reproducibility: Clarity/quality - discussed above in the strengths and weaknesses.

Originality - as the authors point out, this is not too different from prior work on error-correcting output codes, but the specific way to chop up the activations seems new to me.
summary_of_the_review: This is clear and interesting work. It needs to spend more time exploring how depth is related to code quality and utility for out-of-distribution detection, and needs to compare to other ECOC work. As is, the work is a 6. If both of my concerns are addressed, it would turn into an 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: authors propose a method for calibrating classifiers so that they do not return high-confidence errors.
strength: a simple method that leads to good performance. 
Weakness: Some issues with clarity of writing and motivation.
justification_for_why_not_higher_score: While the reviewers are in favor of acceptance their level of excitement does not merit a spotlight or oral according to my current self-calibration.
justification_for_why_not_lower_score: All reviewers are in favor of acceptance.
Decision: Accept: poster