Constraining Representations Yields Models That Know What They Don't Know
Authors: Joao Monteiro
Pau Rodriguez
Pierre-Andre Noel
Issam H. Laradji
David Vazquez
Venue: ICLR 2023 poster (2023)
Keywords: Rejecting classifiers
Selective classification
Uncertainty estimation
Robust classification
Out-of-distribution detection

Abstract: A well-known failure mode of neural networks is that they may confidently return erroneous predictions. Such unsafe behaviour is particularly frequent when the use case slightly differs from the training context, and/or in the presence of an adversary. This work presents a novel direction to address these issues in a broad, general manner: imposing class-aware constraints on a model's internal activation patterns. Specifically, we assign to each class a unique, fixed, randomly-generated binary vector - hereafter called class code - and train the model so that its cross-depths activation patterns predict the appropriate class code according to the input sample's class. The resulting predictors are dubbed total activation classifiers (TAC), and TACs may either be trained from scratch, or used with negligible cost as a thin add-on on top of a frozen, pre-trained neural network. The distance between a TAC's activation pattern and the closest valid code acts as an additional confidence score, besides the default unTAC'ed prediction head's. In the add-on case, the original neural network's inference head is completely unaffected (so its accuracy remains the same) but we now have the option to use TAC's own confidence and prediction when determining which course of action to take in an hypothetical production workflow. In particular, we show that TAC strictly improves the value derived from models allowed to reject/defer. We provide further empirical evidence that TAC works well on multiple types of architectures and data modalities and that it is at least as good as state-of-the-art alternative confidence scores derived from existing models.

Published as a conference paper at ICLR 2023
CONSTRAINING REPRESENTATIONS YIELDS MODELS THAT KNOW WHAT THEY DON'T KNOW
João Monteiro, Pau Rodríguez', Pierre-André Noël, Issam Laradji, David Vázquez ServiceNow Research (Fi (FirstName. La sattNamm) @servi cenow.co.. com
"Currently at Apple.
ABSTRACT
A well-known failure mode of neural networks is that they may confidently return erroneous predictions. Such unsafe behaviour is particularly frequent when the use case slightly differs from the training context, and/or the presence of an adversary. This work presents a novel direction to address these issues in broad, general manner: imposing class-aware constraints on a model's internal activation patterns. Specifically, we assign to each class unique, fixed, vector hereafter called class code and train the model so that its cross-deptht activation patterns predict the appropriate class code according the input sample's class. The resulting predictors are dubbed total activation classifiers (TAC), and TACS may either be trained from scratch, or used with negligible cost as a thin add-on on top of a frozen, pre-trained neural network. The distance between a TAC's activation pattern and the closest valid code acts as an additional confidence score, besides the default unTAAC'ed ed prediction head's. In the add-on case, the original neural network's inference head is completely unaffected (so its accuracy remains the same) but we now have the option to use TAC's own confidence and prediction when determining which course of action to take in an hypothetical production workflow. In particular, we show that TAC strictly improves the value derived from models allowed to reject/defer. We provide further empirical evidence that TAC works well on multiple types of architectures and data modalities and that it at least as good as state-of-the-ar alternative confidence scores derived from existing models.
geeeerraee binary
INTRODUCTION
Recent work has revealed interesting emerging properties for representations learned by neural networks (Papernot& & McDaniel, 2018 Kalibhat et al. 2022; Bäuerle et al.) (022). In particular, simple class- class-dependent patterns were observed after training: there are groups of representations that consistently activate more strongly depending on high-level features of inputs. This behaviour can be useful to define predictors able to reject/defer test data that do not follow common patterns, provided that one can efficiently verify similarities between new data and common patterns. Well known limitations of this model class can then be addressed such as its lack of robustness to natural distribution shifts (Ben-David et al. (2006), or against small but carefully crafted perturbations its inputs (Szegedy et al.] Goodfellow et al. 2014).
These empirical evidences and potential use cases naturally lead to the question: can we enforce simple class- dles-dee structure rather than hope it emerges? In this work, we address this question and show that one can indeed constrain representations to follow simple, class-dependent, and efficiently verifiable patterns on learned representations. In particular, we turn the label set into set of hard-coded class-specific binary codes and define models such that activations obtained from different layers match those patterns. In other words, class codes constrain representations and define a discrete set of valid internal configurations by indicating which groups of features should be strongly activated for given class. At testing time, any measure of how well a model matches some of the valid patterns can be used to reject potentially erroncous predictions. Codes are chosen a priori and designed to approximate pair-wise orthogonality.

---

Published as a conference paper at ICLR 2023
Motivation. The motivation for constraining internal representations to satisfy simple class- dependent structure is two-fold:
1. Given data, we can measure how close to valid pattern the activations of a model are, and finally use such a measure as a confidence score. That is. if the model is far from a valid activation pattern, then its prediction should be deemed unreliable. Moreover, than standard one-hot representations. Long enough codes enable us to represent classes with very distinct, hence discriminative, features.
make codes higher-dimensional
2. Tying internal representations with the labels adds constraints to attackers. To illustrate the advantage of this scheme, consider that an adversary tries to fool standard classifier: its only job is to make it so that any output unit fires up more strongly than the right one, and any internal configuration that satisfies that condition is valid. In our proposal, an attack is only valid if the entire set of activations matches the pattern of the wrong class, adding constraints to the attack problem and effectively making it harder for an attacker to succeed under a given compute/perturbation budget as compared to a standard classifier for which decisions are based solely on the output layer.
Intuitively, we seek to define model classes and learning algorithms such that intermediate represen- tations follow a deas-dependdnt struclure that can be efficiently verified. Concretely, we introduce total activation classifiers (TAC): a component that can be added to any class of multi-layer classifiers. Given data and set of class codes, TAC decides on output class depending on which class code best matches an observed activation pattern. To obtain activation patterns, TAC slices and reduces (e.g., sum average) the activations of stack of layers. Concatenating the results of the slice/reduce steps across the depth of the model yields a vector that we refer to as the activation profile. TAC learns by matching activation profiles the underlying codes. At inference, TAC assigns the class with the closest code to the activation profile that a given test instance yields, and the corresponding distance behaves as a strong predictor of the prediction's 's quality so that, at testing time, one can decide to reject when activation profiles do not match valid codes to threshold.
Contributions. Our contributions are summarized as follows:
1. We introduce a model class along with a learning procedure referred to as total activation classifiers, which satisfies the requirement of representations that follow class- deas-dee patterns. Resulting models require no access to -it-of-distributiin data during training, and offer inexpensive easy-to- obtain confidence scores without affecting prediction performance.
2. We propose simple and efficient strategies leveraging statistics of TAC's activations to spot low confidence, likely erroneous predictions. In particular, we empirically observed TAC to be effective in the rejection setting, strictly improving the value of rejecting classifiers.
3. We provide extra results and show that TAC's scores can be used to detect data from unseen classes, and that it can be used as robust surrogate of the base classifier if it's kept hidden from attackers, while preserving its clean accuracy greater extent than alternative robust predictors.
2 TOTAL ACTIVATION CLASSIFIERS (TAC)
2.1 REPRESENTING LABELS AS CODES
We focus on the the setting of K-way classification. In this case, data instances correspond to pairs x, X' x D, with X' C Rd and y 2,, K} K E N. Usually, model families targeting such a setting parameterize data-conditional categorical distributions over y That is, a given model F X + A will project data onto the the probability simplex A AK1
Alternatively, we will consider classes of predictors of the form f' E with L > K, iee, models that map data onto the unit cube in R. We thus associate each element in } with a vertex of the cube. In other words, we represent class labels with a set of binary codes C {C1, C2 Ca..., Cx). and training can be performed via searching arg min ((z,9)~(X.)) (x))) (=)) (z),,,),,, C,), for a distance D : R x R + R. In doing so, we seek models able to project data such that results close to the vertex corresponding to the correct class label in terms of distance D. At prediction time, one can predict via arg min, (f C.) or
[0, 1
leverage (x) C E C) to estimate confidence.
2

---

Published as a conference paper at ICLR 2023
The main advantage of defining such class of predictors and an encoding scheme for labels is the fact that one can control the properties of the set C to maximize the discriminability of its elements. This is not possible for more common cases such models that project onto A AK-1, where the set C is given by one-hot codes. In fact, for a large enough code length I. and D given by the T4 distance, we can easily choose C. such that D(C Cу) L/2 A j- That is to say that model needs to make several mistakes (along different dimensions of the label code) for its predictions to flip.
2.2 MODEL DEFINITION
To realize the model class F' briefly introduced above, we will leverage the set of n--ayered neural net- works. The outputs of layer i within any chosen f' E F are represented by a'. w E [1, 2, 3, .... Wi]. where W; indicates the width of layer given by its number of output features. We then parti- tion the set of representations a into disjoint subsets (or slices) of uniform sizes, denoted by S I {1. 2.. L/n]] }. Intuitively, our goal is to make it so that groups of high-level features "fire up" more strongly depending on the underlying class of the input. To enforce that property, we consider the sequence of total activations of slices, That is, for the feature slice S on layer i, we will consider its total activation
x) - E
Σ
(1)
CLES h',""
where features arbitrarily considered 2-dimensional objects (h' and h² are, for example, spatial dimensions in convolutional architectures for images). Note that total activations are defined similarly for features of any dimension by simply reducing away the extra dimensions. The sequence of total activations obtained from slices across all layers, denoted A(J", .²), will be referred to as the activation profile of x. An illustration provided in Figure [ where a generic convolutional model has the outputs of its layers partitioned into slices S which are then reduced to yield the sequence A( x.). the activation profile of as induced by f'. Figure 16]in in the Appendix shows Pytorch (Paszke et al. 2019) implementation of feature slicing and activation profile computation. Given a stack of layers yielding activation profiles of dimension L, a set of class codes of dimension L, and a distance D, we define total activation classifiers (TAC) such that they make the prediction
V/ == arg min D(A(f' I), Ct) with associated confidence -(A(f()) Og).
(2)
KEY
Code Label
1 0 ...
0
01
0 ...
D
Codes table
1 A A
A ALE.M
AT A
ALLmol
1011-..1011 0001--10000
#Classes
1011-00100 0101-
S1 $11 S1
St [L/NJ
ST S
Sn [L/nn]
# Channels Input
W1 Com:: I
W
Conv. n
Figure Illustration of Total Activation Classifier (TAC). For simplicity, we illustrate the case where aw is 2-dimensional. The set of codes is chosen a priori and defines valid activation patterns.
2.3 TRAINING
Training is carried out to search for f' E J' with activation profiles close to the correct code. To enforce that behaviour, we minimize the training objective
1 N Lbin = NL
Σ
CT yi
log o (A(f', mx)) + +11 Cy.)) log - (A(f,
(3)
3

---

Published as a conference paper at ICLR 2023
where N is the size of the batch of data used to compute the objective, L is the length of the code and activation profile, and σ is the sigmoid function. Luin computes the binary cross-entropy loss for each dimension of the code and is minimized for perfect match between activation profiles and the correct codes. While minimizers of Lain yield the properties we need into a model, we empirically observed that training against Liin is not trivial in that it requires the of aggressive learning rates, rendering training unstable, especially so for large (i.e. 2 1000) label sets y. To overcome that issue, we define second training objective by parameterizing a conditional categorical distribution over the label set using the distances between a given activation profile and all the codes and performing maximum likelihood estimation. More formally, the alternative training objective is
Lee
log K Σ 2k=1 -.AA((
(4)
N
where 7 is a scaling parameter.
While both Chin and Lee are reasonable choices for the training of TAC, each presents challenges. As mentioned above, Lsin is such that it yields unstable training but we found that, in the cases where one can minimize discriminative confidence scores. On the other hand, training Lee is less unstable in that more common hyperparameter configurations work out of the box. However, minimizers of Lee are not as good at matching vertices tightly since the objective only requires activation profiles to lie closer to the correct code than they are from other codes and, in that case, using distances as confidence scores is not as effective. We thus consider the linear combination of these training objectives
produces models able to match codes very closely, and as such provide
(5) where a and I are hyperparameters. This choice takes advantage of the relative easiness of training against Loe and the pressure of Loin towards solutions able to match codes more closely. Remark that both objectives share minima, so minimizing their sum does not introduce any real trade-off.
a Loin + BL
2.4 CODES DEFINITION
The only requirement we consider for the class codes is that they are as dissimilar as possible in some sense. To build that set, one could use deterministic procedures such as, for instance, computing Hadamard matrices, but doing so adds constraints to the models since implies codes of length 2", n E N. We observed that, given large enough codes, randomly building C suffices for us to get discriminative codes. We thus define C as a random matrix of dimensions | x L, where entries independent Bernoulli random variables, observed initialization. We remark that binary codes are chosen for simplicity. In particular, we can know exactly which features matter for each class each layer, and this can be used for inspection of representations, or determine features in the inputs implying predictions.
2.5 ATTACHING TAC TO PRE-TRAINED MODELS
Provided that one has their ready-to-use classifier, TAC can be added on top of its features, and the base classifier is preserved as-is (i.e., "frozen")). To do so, we isolate TAC from the base predictor via trainable projection layers applied in the representations of the base classifier. Then, we apply the slicing and reducing operations defined in Eq. 3 on those projections to obtain activation profiles. Projections correspond to independent MLP networks, one for each layer of the base classifier. The projection layers are then trained following the loss in Eq. 3 but gradients are not propagated to the base classifier, which remains unchanged. We remark that, while this approach is intended to simplify TAC training and enable its use in more practical scale, it also enables the combined use of confidence scores obtained by TAC and at the output layer of the base classifier (e.g., the maximum softmax probability (MSP) (Hendrycks & Gimpel, 2016) or the maximum logit score (MLS) (Vaze et al.) 2022)), i.e., one can reject inputs that have too low confidence w.r.t. either of these casy-to-obtain scores. In other words, adding TAC on top of an existing classifier only improve upon any confidence score that was already in use.
EVALUATION
Evaluations split into three main parts:
4

---

Published as a conference paper at ICLR 2023
Section 3.1
We start with a proof-of cooff-f--onneptt and show that TAC can match activation patterns defined by class codes. We further show that small norm attackers are not able to match codes as well as clean data, rendering the distance between activation profiles and codes good confidence score.
Section 3 We then proceed to the main evaluation and use TAC as an add-on to existing classifiers. In this case, evaluate performance under the rejection setting and show TAC to improve upon the base classifier. We further evaluate TAC when used to detect test data from unseen classes.
Section 3.3 We seek additional applications of TAC and put it to test robust surrogate to the base classifier. We assume the threat model where only the base predictor is exposed to attackers, and observe that simple adversarial training results in competitive robust accuracy against strong attacks.
For a fair comparison, we consider approaches that do not require access to auxiliary data of any sort. Directly relying upon statistics of the output layer of classifiers was observed recently to yield state-of-the-art performance on different OOD detection tasks (cf. MLS, (Vaze et al. 2022 We thus consider MLS obtained from the base classifier as our main performance target, but also account for other similar recent methods for a broader comparison such as DOCTOR (Granese et al. (021). as well as simple but effective baselines such as MSP (Hendryeks & Gimpel 2016). addition, we consider generative approaches that use likelihoods in the representation space as detection scores. For the robustness case, we use state-of-the-an robust classifiers as a reference of performance.
Baselines:
Evaluation metrics:
Besides prediction accuracy, for the detection cases, evaluations are carried out in terms of the area under the operation curve (Det. AUROC) as well as in terms of the detection rate measured at the threshold where the false negative and false positive rates match, as commonly done to compute the Equal Error Rate. In Figure 15 we provide examples of the two metrics computation.
3.1 PROOF-OF-CONCERS
To test for whether commonly used models are able to match activation patterns given by class codes, train TAC' ed WideResNet-28- 10 (Madry al. 2017) on CIFAR-10 (Krizhevsky et al. 2009). starting from random weights. TAC's slice/reduce operations are applied three layers that output 160, 320, and 640 2-dimension feature maps. We use 16 slices in each layer so that the resulting code length L is 48. Upon training to convergence, we inspect the representations obtained from a random draw of the test set, as displayed in Figure 2 Each column in the figure contains information about a different layer (layer depth grows from left to right). In the first row, we plot the raw activations after average pooling spatial dimensions. The activation profile A is displayed in the second row, where one can see tight match with the ground-truth codes, as displayed in the third row. The differences between and code are shown in the bottom row. We further test how good of a confidence score one can get by measuring some distance between A and code. In Figure 3 we plot histograms of L1 distances for clean data and adversarial perturbations of the test set of CIFAR-IO. Attackers correspond to subtle PGD perturbations (Madry et al. 2017) obtained under a Loo budget of 255 consider the white-box access model in which the attacker has full access to the target predictor and the table of codes. Attackers are created so that the activation profile A moves towards the code of wrong class. Attackers fail in matching codes as tightly as clean data. TAC can then spot attackers and defer low-confidence predictions. Confidence intervals for the distances at different layers shown in Figure in the appendix, showing similar behaviour throughout depth. We remark that we've ve empirically observed that one can sometimes improve performance by picking a particular layer, and a task/model dependent analysis could be carried out in real application to select the
8 We
target layer as reported for other cases in Figures I Ha 11b
12a
and 12b
3.2 ADDING TAC TO PRE-TRAINED CLASSIFIERS
TAC preserves the accuracy of the base classifier. We consider intent prediction tasks of Di-
aloGLUE (Mehri et al., 2020). Namely, we conduct experiments on HWU64 (Liu et al.] (021). Banking77 (Casanueva et al. 2020). and CLINC150 (Larson et al. 2019). which correspond to sentence-level multi-class classification problems. We train base classifiers corresponding to the AGBERTa-BASH architecture Liu et al. (0019 to convergence. TAC is added afterward, and trained while the base model is frozen. We apply TAC across depth in 13 different points of the base model. The trainable parameters in this case correspond to the projection layers, one for each of the 13 layers, used to isolate the TAC operations from the pre-trained models. The number of slices in each layer is
5

---

Published as a conference paper at ICLR 2023
P4
Am
Cean
L1 Abtaco imagis
0012
0000
0008
0004
0002
Figure 2: Activations of ed Wide- ResNet-28-10 trained on CIFAR-10. Each column in the figure contains information about a different layer.
30
100
150
220
250
Figure 3: Distance histograms for attacks and natural images on CIFAR-10.
HWU64
Banking77
CLINCI50
RoB-TAC
RoB-BASE
BERT++
RoB-TAC
RoB-BASS BASE
BERT ++
RoB-TAC
RoB-BASE
BERT+
93.9
92.6
92.9
94.2
93.9
93.4
97.4
97.0
97.1
Table 1:
Prediction accuracy for three intent prediction tasks. TAC is added on pre-trained RoBERTa -EAEE, and the base model is kept frozen during training. BERT++ indicates the best performance reported in Mehri et al. (2020) for each dataset.
treated hyperparameter and tuned with cross-validation. Projection layers correspond stacks of fully-connected layers followed by ReLU activations, and features at each layer are extracted the <eos> token. The depth of the projection stacks along with further details the model architecture as well on the datasets can be found in the Appendix. as well as additional results on image datasets further supporting our conclusions (cf. Table Results are reported in Table TT] In this case, we compare the performance obtained from TAC's predictions with the base classifier it relies on (indicated as RoB-BASE, in the table) as well as with the best performing case reported in Mchri et al. (2020) (BERT++). TAC outperforms the basclines in all three cases, which suggests the additional constraints imposed at early layers do not significantly affect representation capacity.
Value analysis of predictors able to reject. We evaluate TAC as well as different statistics of the output layer of base predictors when those are all treated rejecting classifiers, i.e., models that can abstain if not sufficiently confident in their predictions. We then leverage the framework proposed by Casati al. (021), which defines the application-speccfic error cost capturing the ratio of how much we dislike accepting incorrect classifications over how much we like accepting correct ones. This framework then proposes to evaluate different predictors in terms of their value V N as function of where Ne and N; correspond to the number of correct and incorrect error detection predictions made by model over a sample of size N = N: + N, + Ny. for N, has maximized at N, = 0, in which case this value matches the standard prediction accuracy. In Figure 4 we plot the Value Operating Characteristic (VOC, ef. definition in Casati et (202 II) curves for TAC, MLS, and MSP. We pre-trained a ViT Base-166 16666 6 (Dosovitskiy et al. (020)) as the base predictor, and TAC operations are performed in 13 different layers throughout the model. We perform k-fold (k: 5) random splits on the validation set of ImageNet and, for given split and value of we then use the k left-out splits to select the confidence rejection threshold that maximizes V. Curves averaged over splits are plotted for the data used for threshold selection (indicated as train in the plot) well as for the left-out splits. One can then note that TAC's scores yield the highest value throughout a broad range of w., if not all of it.
rejections. Note that the extreme
While we defer most of the implementation details to the Appendix due to space constraints, we highlight a few practical observations. In particular, we observed that setting the distance D to the L distance along with using the full activation profiles and codes to yield good default, working well across most cases. In some particular cases, on ImageNet specifically, we observed gains moving to cosine similarity and focusing recommend the default setting in general but adjusting to the specific data of interest should be done whenever feasible. All of our design decisions were made with left-out validation data.
the last layer only. As these choices case dependent,

---

Published as a conference paper at ICLR 2023
Method
Det. AUROC (%)
Det. Rate (%)
HWU64
MPS
89.39
81.43
DOCTOR
89.39
81.43
MLS
89.36
80.12
0.6
TAC (7,)
89.25
83.25
(Cos)
CLINC150
0.4
MPS
90.56
82.77
DOCTOR
90.56
82.96
0.2
MLS
91.36
82.66
TAC (L1)
93.25
85.70
0.0 100
1.50
ImageNet
MPS DOCTOR
80.74
73.63
80,91
73.69
Figure 4: VOC curves for different predic- tors on ImageNet as a function of w. the application-specfficc error cost characterizing how undesirable it is to accept incorrect clas- sifications. The overall value V of TAC-based predictors dominates for broad range of w.
MLS TAC (Cosine) 89.41
53.69
52.75
81.64
Table 2: Detection of prediction errors. Detec- tion rates are measured at the threshold where false positive and false negative rates match.
To further assess how good different scores are when used to define rejecting classifiers, the perfor- mance of different scores for detecting prediction errors reported in Table ] TAC's distances as measured between activation profiles and codes yield a higher detection rate in all cases. Finally, accuracy vs. rejection level curves are shown in Figure for different models trained on ImageNet (refer to Appendix A.7 for results on other datasets). The horizontal axis represents the fraction of the test set the model must predict from, and comparisons are carried such a curve, which would be for perfect model. We remark that predictions for the whole test set are fixed, and different results are given by how good different scores are predicting errors.
terms of the area under
Out-of distiiiiiitto detection with TAC. We further assess performance under the setting where test instances might belong to classes unseen during training. That is, for an unseen class, a safe model would be expected to abstain from classifying it. We test this behavior by using TAC's distances to decide when not to predict. Performance is evaluated on the CLINC150 150 benchmark. The training sample is such that UNKNOWN label is used to indicate classes not represented in the remainder of the lahel set. We train our models on all classes except UNKNOWN, but try to detect unknown classes testing time. Detection performance is reported in Table 3 We compare with MLS Vaze al. (202)), recently shown state-of-the- in this setting, and the approach introduced by Shao et al. 2020) where likelihoods of features under a Gaussian mixture model (GMM) are used as detection scores. TAC improves detection performance relative to those other unsupervised approaches that rely solely on the outputs the base classifiers, especially so terms of detection rate, indicating that one can leverage low-level outputs to design better confidence metrics. In Figure 8J Appendix A.3] we report the detection performance obtained when different distances are used between TAC's activation activation profiles and class codes, and observe somewhat consistent behaviour.
Method
Det. AUROC (Sr
Det. Rate (Sr
Shao 2020
96.55
91.51
0.93
Vaze et al. 2012
97.00
91.78
IAC (21)
97.28
92.82
0.93
MLSS 427%
87.45%% (area
Table Comparison of different scoring strategies used for detection of examples from classes unseen during training. The detec- tion rate measures the fraction of correctly detected attackers at the threshold where false positive and false negative rates match. For TAC, we report results considering the best performing distances, L, in this case.
0.85
0.89
02
2.5
Figure 5: Accuracy of TAC'ed ed ViTs on Ima- geNet at different levels of allowed rejection.
7

---

Published as a conference paper at ICLR 2023
3.3 ADDITIONAL APPLICATION: TAC AS A ROBUST SURROGATE TO THE BASE CLASSIFIER
TAC is meant as a way to detect prediction errors, not as a robust solution against while-box adversarial attacks (i.e., for which the attacker has unlimited, direct access to the full model). In fact, our results (Table more details below) indicate that TAC performs quite poorly in such a white-box adversarial context. That being said, we also consider a more limited adversary, monitored white-box, in which black-box (i.e., secret) TAC module monitors how a public, white-box model is being used. In other words, in the context of monitored white-box API call to a publicly-available model, an outsider may observe that the API behaves identically to the white-box model, but the monitor may take actions following attacks, e,g., request human assistance. Using TAC as a monitor proves to be a strong baseline, competing with the top performers for "true" white-box robustness.
Table 3 reports clean and robust prediction accuracy for Atto- Note that each of the two top rows report our results for both TAC and the base model, for both attack classes. The rest of the table provides reference results for the top-3 entries in Robust Bench /s lcaderboard (in grey in the table). Also note that auto-attack comprises set of attack models evaluated individually. For completeness, for our models, we reported both the overall performance as well as results specific to each attack strategy. Adversarial training is employed using PGD perturbations obtained from the base classifier. To be clear, our goal here is not to outperform these robust baselines, and comparisons between the two access models are unfair in favor of the monitored white-box. Rather, we seek indications of other use cases where adding a TAC to an existing model might be useful. TAC preserves clean accuracy to a greater extent than the alternative approaches and, in the monitored white-box setting, TAC performs particularly well. In fact, up to improvements on the case of apgd-dlr, TAC is able to raise the performance of the base classifier to the level of the top performers the leader board.
Model
Attack
Clean Accuracy
apgd-cv
Individuual attacks from Auto-Altack (9)
upgd--lly
Jubst
square
Average
TAC/ model TAC/ Base model
Monitonrd wi-h White-bax
93.23/95.55 .9
0.00
53. 13.74/0.00 0:00
70.00
75.86/ 0.00
72.01 0.00
61.44/ 1,83
74.34/
38.90 0.46
73.68
53.74/ 0.00
2.42/ 00.00
Performance reference: Rohust baselines
Overall Auto-Atttcc (%)
Robuli
White-bux
While-bax
88.74
66.10
Ualei BI m.
2020
White-hax
91.10
65.87
Table 4: Robust accuracy of TAC under different access models for attackers. Depending on how much information the attackers has access to, an added TAC can raise the accuracy of the base model to close to tfateoff-the-an level. Overall for baselines should be compared with our Min. results.
RELATED WORK
Error-correcting output codes. ECOCs assign unique code per class (Dietterich & Bakiri 1994
Gareia-Pedrajas & Fyfe| 2008 Rodriguez et al.) 2018). Codes are commonly designed to maximize the Hamming distance between all pairs of classes so that errors can be detected and corrected by remapping incorrect predictions to the nearest code (Bautista et al.) 2012). Verma & Swamil (2019) Song al. (2021) leveraged ECOC overcome adversarial attacks in multi-class classification. To do so, they train an independent binary classifier for each of the dimensions in a code. Rather than training an ensemble of binary classifiers, in our case we propose slice the activations of already-existing architectures to obtain the elements of a code. As a result, TAC is significantly more efficient since it requires single feature extractor. In addition, constraining outputs along the depth of a deep neural network makes adversarial perturbations more difficult to execute since it would require an attacker simultaneously change the network's 's activations at multiple layers.
Models that rely on low-level features. The use of low-level (close to input) representations has been explored in recent work (Rodríguez et al., 2017 Evei et al. 2022; Monteiro et al. 2022a). However, their main goal is to improve the performance of classifiers rather than attaining more robust architectures. In Wu et al. (202I), on the other hand, classifiers are trained along with a decoder, and learned features are enforced to be such that inputs can be reconstructed. At testing time, reconstruction error can be used as a test statistic to detect anomalies. In Evci et al. (2022). features collected throughout the model rather than at the outputs of specific layer. which improves downstream performance under domain shift. Audio representations were shown in (Tang et al. 2019) Monteiro et al. 2022al to improve downstream performance once low-level features incorporated.
8

---

Published as a conference paper at ICLR 2023
Out-of-distribution detection. OOD detection methods aim to detect samples that deviate con- siderably from the training distribution. Previous methods such as (Hendrycks & Gimpel 2016
Liang et al.] 2017 Hsu et al. 2020) focus on the maximum of the softmax activations, which tend
to be smaller for OOD samples. OOD detection is also related to novelty detection (Abati et al.
2019 Perera et al. 2019 Tack al., (020) and anomaly detection (Hendryeks et al. 2018 K Wwon
ct al. 2020; Bergman & Hoshen 2020). Generative approaches were also considered in recent literature. Shao al. (2020) and Jiang et al. (2022), for instance, train generative models on top of representations of in in-distribution data and use the likelihoods of test examples as a confidence score. Most of these methods propose finding OOD samples in existing classifiers or propose new models to detect OOD examples explicitly. In contrast, we propose new model component, allowing for better OOD detection by inspecting activations. The classification strategy discussed in (Monteiro) et al. 2022b) also uses a slice/reduce type of representation. However, in their case, only a single layer is considered and activation patterns are matched to standard one-hot codes at the gradient level as opposed to the activations, with standard classifiers, any activation pattern apart from the output layer is valid, which we solve with TAC. More recently, Grrnnee et al. (202T) and Vaze et al. (0222) showed that, in opposition to approaches that require access to auxiliary data (Jiang et al. 2018) or gradients (Liang et al., 2017), directly using simple statisties of the output layer of well tuned classifiers would outperform complex methods and reach state-of -frtth--rtttt OOD detection performance on number of benchmarks. Similarly to our proposal. Haroush al. (202I) leverage class-dependent structure in representations across layers to detect out-of- -i-diitribttion data. In that case however, pre-trained models are used and the focus lies on determining good detection scores. In our case, detection scores correspond to simple vector distances, and our focus lies on the training side. The two approaches are orthogonal and could be combined.
attackers still have room to find effective perturbations since, as
A Adversarial Robustness. Adversarial attacks perturb the input in such way that pushes the model's output towards an erroneous prediction. Several attack strategies were proposed specifically targeting the case of image classifiers (Goodfellow et al., 2014| Szegedy et al. 2013 Carlini & Wagner 2017]
Tu et al. 2019]] Xiao et al., (2018] | Zhang et al.] 2020), but also for other domains, such as sentiment
analysis systems (Jin et al. [2020] 3D point cloud models (Hamdi et al.
2020), audio recognition
systems (Abdullah et al. 2019), and text classification (Morris et al. 2020 Zeng et al. 2020). Zhang
et al. (2021) describe different strategies to make models more robust to adversarial attacks. Some examples are input transformation methods (Guo et al. 2017 Xie et al. 2018), stochastic defense
methods (Cao & Gong 2017 Papernot et al. 2016), and adversarial training (Goodfellow al.
2014; Miyato et al.] 2018). However, defense strategies cannot indicate that system is under attack to carry out corrective/preventive measures. Thus, detection methods have been proposed such as supervised cases where small binary classification network is used to that end (Metzen et al. (017). Other supervised detection approaches operate by monitoring statisties of natural and perturbed samples (Feinman et al.) 2017 Xu et al.) 2017 Akhtar et al., 2018) or by combining outputs of indepdently trained classifiers and using that as inputs to a detector (Monteiro et al.) 2019). Gradient- based detection methods were proposed such that adversarial data is spotted through the norm of the
gradients of a model relative to the input (Lust & Condurache 2020). For text classification, recent work (Rawat et al. 2021) generates oft-of-domain samples to perform OOD detection.
CONCLUSION
We introduced TAC: a model component that can be attached to an existing predictor without affecting its accuracy to provide scores indicating how likely it is that predictions are incorrect. We showed that distances, as measured between activation profiles and class codes, define effective detection scores of problematic predictions. Notably, we successfully trained TAC on top of models as large as variations of BERT and ViT for text and image classification tasks respectively, and observed resulting scores to yield more effective rejecting classifiers than state-of-the-art alternatives that use the same base models TAC builds upon. We further showed that TAC can be applied to detect data from unseen classes and improves the detection rate relative to strong alternatives. We finally evaluated the robust accuracy of ad adversarially trained TAC on an additional task out of the original scope of our work to test for further applications where this kind of model can be useful, observing a promising performance when only the TAC component is kept private from attackers.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
In order to facilitate reproducibility of our empirical assessment, we developed all of our experiments on top of openly accessible data, models, and software tools. Moreover, both training and evaluation across all applications we considered were performed in single-GPU hardware. Implementation details are reported in Appendix B In particular, code snippets of critical components are displayed
in Figures T6 and 17
REFERENCES
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In Conference on Computer Vision and Pattern Recognition, 2019.
Hadi Abdullah, Washington Garcia, Christian Peeters, Patrick Traynor, Kevin RB Butler, and Joseph Wilson. Practical hidden voice attacks against speech and speaker recognition systems. arXiv preprint 10te::9900.00744,, 2019.
Zahid Akhtar, João Monteiro, and Tiago H Falk. Adversarial examples detection using no-reference image quality features. In 2018 international Carnahan conference on security technology (ICCST), pp. 1-5. IEEE, 2018.
Alex Bäuerle, Daniel Jönsson, and Timo Ropinski. Neural activation patterns (naps): Visual explain- ability of learned concepts. arXiv preprini arXiv:2206.10m 2022.
Miguel Ángel Bautista, Sergio Escalera, Xavier Baró, Petia Radeva, Jordi Vitriá, and Oriol Pujol. Minimal design of error-correcting output codes. Pattern Recognition Letters, (3(6)6933-702, 2012.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Liron Bergman and Yedid Hoshen. Classification- based anomaly detection for general data, arXiv preprint arXiv:2005.02335 023599 2020.
Xiaoyu Cao and Neil Zhenqiang Gong. Mitigating evasion attacks to deep neural networks via
region-based classification. In Annual Computer Security Applications Conference, 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE
symposium on security and privacy, 2017.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807, 2020.
Fabio Casati, Pierre-André Noël, and Jie Yang. On the value of ml models. arXiv preprint arXiv:21 2021.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
Thomas G Dietterich and Ghulum Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal of artificial intelligence research, 2:263-286, 1994.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Hcad2toe: Utilizing intermediate representations for better transfer learning. arXiv preprint arXiv:2001.03529, 2022.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv: arXiv:1703.08 1703.004 10, 2017.
10

---

Published as a conference paper at ICLR 2023
Nicolás Garcia-Pedraee and Colin Fyfe. Evolving output codes for multiclass problems. 1EEE Transactions on Evolutionary Computation, 12(1):93 10099----6,--0- 2008.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint (xXiv:441266572 2014.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
Sven Gowal, Sylvestre- Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34:4218-4233, 2021.
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. Doctor: A simple method for detecting misclassification errors. Advances in Neural Information Processing Systems, 34:5669-5681, 2021.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using imput transformations. arXiv preprint arXiv:1711.00111 2017.
Abdullah Hamdi, Sara Rojas, Ali Thabet, and Bernard Ghanem. Advpe: Transferable adversarial perturbations on 3d point clouds. In European Conference Computer Vision, 2020.
Matan Haroush, Tzviel Frostig, Ruth Heller, and Daniel Soudry. A statistical framework for efficient out of distribution detection in deep neural networks. In International Conference on Learning Representations, 2021.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rXiv:1610.02111 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 18/ 2.04666, 2018
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: distribution image without learning from out-of-distribution data. In Conference on Computer Vision and Pattern Recognition, 2020.
Detecting out-of-
Dihong Jiang, Sun Sun, and Yaoliang Yu. Revisiting flow generative models for duitt--distribution detection. In International Conference on Learning Representations, 2022.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. Advances in neural information processing systems, 31, 2018.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. AAAI conference on artificial intelligence, 2020.
Neha Mukund Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Towards better understanding of self-supervised representations. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Citeseer, 2009.
Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Backpropagated gradient representations for anomaly detection. In European Conference on Computer Vision, 2020.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and oft-of-scope prediction. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNL.P-JCCNIP), 2019.
11

---

Published as a conference paper at ICLR 2023
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-oofdistributiin image detection in neural networks. arXiv preprint arXiv:1706.02699, 2017.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Springer, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907v 11622, 2019.
Julia Lust and Alexandru Paul Condurache. Gran: an efficient gradient-norm norm based detector for adversarial and misclassified examples. arXiv preprint wrXiv:2004.09179 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1oXiv::706.00 770.00608, 2017.
Shikib Mehri, Mihail Eric, and Dilck Hakkami-Tur. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. arXiv preprint arXiv:2009.1333 2020.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff, On detecting adversarial perturbations. arXiv preprint arXiv: aeee:04002275 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and cemi-super learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993, 2018.
João Monteiro, Isabela Albuquerque, Zahid Akhtar, and Tiago H Falk. Generalizable adversarial examples detection based bi-model decision mismatch. In 2019 JEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 2839-28441 IEEE, 2019.
João Monteiro, Jahangir Alam, and Tiago H Falk. Multi-level self-attentive tdnn: A general and efficient approach to summarize speech into discriminative utterance-leve representations. Speech Communication, 2022a.
João Monteiro, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Greg Mori. Monotonicity regularization: Improved penalties and novel applications to disentangled representation learning and robust classification. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022b.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.0540 2020.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803. /80300465.. 2018.
Nicolas Papernot, Patrick McDaniel, Xi Wu. Somesh Jha, and Ananthram Swami. Distillation as defense adversarial perturbations against deep neural networks. In IEEE symposium on security and privacy (SP), 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Conference on Computer Vision and Pattern Recognition, 2019.
Mrinal Rawat, Ramya Hebhalaguppe, and Lovekesh Vig. Pnpood: Out-of-distribbttion detection for text classification via plug andplay data augmentation. arXiv preprint arXiv:21 1.00566, 2021.
Sylvestre-Alvie Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint 103.v:2103.01946, 2021.
12

---

Published as a conference paper at ICLR 2023
Pau Rodríguez, Guillem Cucurull, Josep M Gonfaus, F Xavier Roca, and Jordi Gonzalez. Age and gender recognition in the wild with deep attention. Pattern Recognition, 72:563-571, 2017.
Pau Rodríguez, Miguel A Bautista, Jordi Gonzàlez, and Sergio Escalera. Beyond one-hot encoding: Lower dimensional target embedding. Image and Vision Computing, 75:21-31, 2018.
Laëtitia Shao, Yang Song, and Stefano Ermon. Understanding classifier mistakes with generative models. arXiv preprint arXiv:2010.02364 2020.
Yang Song, Qiyu Kang, and Wee Peng Tay. Error-correcting output codes with ensemble diversity for robust learning in neural networks. AAAI Conference on Artificial Intelligence, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1r 2013.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in neural information processing systems, 2020.
Yun Tang, Guohong Ding, Jing Huang, Xiaodong He, and Bowen Zhou. Deep speaker embedding learning with multi-level pooling for rext-independent speaker verification. In JEEE International Conference on Acoustics, Speech and Signal Processing. 2019.
Chun-Chen Tu, Paishun Ting, Pin-Yo Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-based zeroth order optimization method for attacking black-box neural networks. In AAAI Conference on Artificial Intelligence, 2019.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
Gunjan Verma and Ananthram Swami. Error correcting output codes improve probability estimation and adversarial robustness of deep neural networks. Advances in Neural Information Processing Systems, 2019.
Mike Wu, Noah Goodman, and Stefano Ermon. Improving compositionality of neural networks by decoding representations to imputs. arXiv preprint arXiv:2106.00769 2021.
Chaowei Xiao, Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed
adversarial examples. arXiv preprint arXiv: 1801 1001.8001222 022112,22222 2018
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects
through randomization. In International Conference Learning Representations, 2018.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint 17040:170..01155, 2017.
Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Zixian Ma, Bairu Hou, Yuan Zang, Zhiyuann Liu, and Maosong Sun. Openattack: An open-source textual adversarial attack toolkit. arXiv preprint arXiv:2009.09191, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprini 17Xiv:1710.0040
2017.
Xingwei Zhang, Xiaolong Zheng, and Wenji Mao. Adversarial perturbation defense on deep neural networks. ACM Computing Surveys (CSUR), 54(8):1-36, 2021.
Yanghao Zhang, Wenjie Ruan, Fu Wang, and Xiaowei Huang. Generalizing universal adversarial
attacks beyond additive perturbations. In International Conference Data Mining (ICDM), 2020.
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
A.1 FURTHER RESULTS ON TRAINING TAC FROM SCRATCH
In Table we report clean prediction accuracy and detection performance for adversarial perturba- tions. In this case, we compare standard classifiers with their corresponding TAC considering both MNIST and CIFAR-10. PGD adversarial perturbations are obtained with Madry al. (0177) attacks under Lo budgets of 0.3 and 255 for MNIST and CIFAR-10 10, respectively. The same WideResNet- 28- 10 discussed in Section is used for CIFAR- 10, while a 4-layered convolutional model is used for MNIST. We evaluated different detection scores corresponding Lo, L1, L2, and Lo distances measured between and codes, and reported the one that performed the best each case (Lec and L1 for MNIST and CIFAR-10, 10, respectively). We evaluated both MSP and MLS for the base classifiers and reported the best performer. TAC improves the detection performance at the cost of slight drop in prediction accuracy. We investigate the drop in accuracy by testing the capacity of the model after the TAC operations are included. To do that, we train models to overfit randomly assigned labels on the training of MNIST. Figure Figure&Jillustren iilllustraess the in-sample error rate, where one can notice that while both models manage to achieve a minimum error rate, one class converges much faster than the other. Thus, we propose applying TAC to trained base classifiers avoid such training difficulties. We Il show that, in that case, we avoid accuracy drops even in much larger models and datasets.
Accuracy (%)
AUROC (%)
0.8-
MNIST
Base model
99.0
75.7
0.6
(L..)
98.6
80.8
CIFAR-10
Eme
0.4
Base model
95.6
93.7
(A,)
94.9
95.7
Table 5: Prediction performance as well as adversarial detection evaluation for PGD at- tackers under Loe budgets of 0.3 and 255 the cases of MNIST and CIFAR-10, respec- tively.
200
4000
000
1000
Ezech
8 for
Figure 6: Capacity test MNIST. Models overfit randomly assigned labels.
A larger size version of Figure is displayed in Figure 7 for easier visualization.
A.2 PREDICTION PERFORMANCE ON VISION DATASETS
In Table 6 we report prediction performance of TAC compared to the base classifiers builds upon. Similarly to the cases of text classification reported in Table [ the performance of the base classifier matched by the additional component.
MNIST
CIFAR-10
ImageNet
TАC
BASE
TAC
BASE
TAC
BASE
99.3
99.0
95.0
95.6
80.6
80.0
Table 6: Prediction accuracy for three object classification tasks. TAC is added on pre-trained classifiers and closely matches their accuracy (referred to as BASE).
A.3 COMPARING TAC UNDER VARYING D
In Figure 8 we report the detection performance obtained when different distances are used on top of TAC. Performance is consistent across different choices for D, except for the case of Lee, which indicates that there is always some dimension with a loose match between activation and code, no matter whether data is in- distr---ittiittioon.
14

---

Published as a conference paper at ICLR 2023
Layer2
200
650
Figure 7: Activations of TAC'ed Mide-ResNNe-28-111 trained on CIFAR-10. Each column in the figure contains information about a different layer (layer depth grows from left to right). In the first row, we plot the raw activations after average pooling spatial dimensions. The activation profile is displayed the second row, where one can see tight match with the ground- truth codes, as displayed in the third row. The differences between A and code are shown in the bottom row.
1.9
TAC
MLS
GVW
0.8
0.7
6.5
Figure 8: Detection of out-of-sample test instances in terms of AUROC.
A.4 MATCHING OF ACTIVATION PROFILES AND CODES
In Figures Da and 9b report further evidence showing that TAC succeeds in matching activation profiles and class codes. To do so, we compare codes with the set of class-wise average activation profiles given by the following for a particular class label y and data sample D:
1
A("
Σ A(' (. 11)
Da
2.yED
where [[:] the indicator function and D_ is the subset of D that belongs class
D. { 3) E D c).
(7)
For both MNIST and 10, we then build the heatmaps shown in Figures aaland 9bby computing the cosine distances between class average activation profiles and codes. That is, heatmap H will be a | [}] matrix such that entry Hij will be:
Hiy(y)) = cos(A(f Cz))
(8)
In both the cases of MNIST and CIFAR-10, observe that H is such that its main diagonal is highlighted. indicating an effective matching between activation profiles averaged for given class and the corresponding class code.
15

---

Published as a conference paper at ICLR 2023
12
11
0.9
0.8
0.7
2
(a) MNIST
(b) CIFAR-10
Figure Heatmaps indicating the distances between activation profiles averaged per class and different class codes.
150
Glen impes
cataree
Figure 10: Layer-wise distance confidence intervals on CIFAR-10.
A.5 PERFORMANCE ANALYSIS PER LAYER
12b present per layer prediction accuracy along with per layer error detection performance for the cases of CLINCİ50 and ImageNet. Interestingly, for ImageNet, we observe that performances grow monotonically with depth. That's not the case for CLINC150 where a non-mono mon-monotonic relationship is observed and the peak is not at the last layer. These results support a general recommendation of setting as default the use of the full activation profiles and codes, but determine the best layers on data of interest whenever feasible.
In Figures 1la 11b
2a
0.8
6.8
6.6
AUMOC
0.4
bat.
0.6
6.2
0.5
12
10
Layer
Layer
(a) Accuracy
(b) Error detection AUROC
Figure 11: Performance at each layer individually for TAC trained on C5I
16

---

Published as a conference paper at ICLR 2023
6.7-
0.6
AUROC
0.4
6.3
6.0
12
10
Layer
Layer
(a) Accuracy
(b) Error detection AUROC
Figure 12: Performance each layer individually for TAC trained on ImageNet.
A.6 ABLATIONS
Results prediction accuracy and error detection performance are shown in the table below for ECOC baselines. We consider those to be ablations of TAC since they similarly project data to match binary codes, though they focus on the last layer only. In particular, of (Dietterich & Bakiri 1994) proposed by (Rodríguez et al., 2018) where an ensemble of binary classifiers is trained top of representations extracted at the last layer. The ensemble trained on top of the same pre-trained base classifiers we considered for TAC, and the same hyperparameter sweep was accounted for. In addition, we further considered a variation of that bascline where the LCe we proposed is further included in an attempt to obtain strong baselines.
considered the extension
Prediction accuracy was severely affected in all ablation cases, as was the error detection performance in two out of three datasets. Specifically in the case of ImageNet, we observed an improvement error detection performance (at severe cost in prediction accuracy), which inline with other experiments since we observed the last layer to yield the best performance only in this case.
Accuracy (%)
Det. AUROC (%)
Detection rate (%)
Dataset
TAC
ECOC L+LCE
TAC
ECOC/ +LCE
TAC
ECOC / LCE
HWU64
93.9
73.6 85.2
89.3
86 86/83.1
83.3
77.3 76.2
CLINC150
97.4
83.4 / 87.8
93.3
83.5 82.6
85.7
75.6 / 75.1
ImageNet
80.6
/4.4/77.8
89.4
90.06 / 83.0
81.6
83.5 / 74.9
Table 7: Ablation results. For ECOC, we present results with and without the use of LCE-
A.7
EXTRA DETAILS AND RESULTS ON THE PERFORMANCE OF REJECTING CLASSIFIERS
A.7.1 VOC CURVES
We provide further results comparing TAC with commonly used confidence scores such as output layer statisties when models are expected to reject or abstain from predicting if confident enough. We then repeat the procedure we used to create the VOC curve reported in Figure 4 for other datasets. To do that, we split the test sets into splits. For each such split, we use the larger portion of the data (four splits) to find the confidence threshold that maximizes the value Y of the underlying predictor under the given error detection scoring strategy. We then plot the average value curves as a function of w (the error cost) for both the larger (train) and smaller (test) data portions when we refrain from predicting from any exemplar for which the confidence score is below the threshold for the given w. Figures 13a 13a and 13c correspond to VOC curves for HWU64, CLINC150, and ImageNet (same as Figure but with matching range of w for consistency with other cases), respectively.
Results support previous conclusion: the best detection score is iesiicattio-dependee In other words, best performers depend on the value of w as well as on the underlying data. any case, the use of TAC's scores results in improvements in several cases making it clear that the proposed approach
17

---

Published as a conference paper at ICLR 2023
should be used in cases where prediction errors are costly/unsafe and rejecting is possibility requirement.
1.0
MSP Train
MSP-
0.8
(L3)
MSP
test
MEL Less
MILS- Fisl-
TAC (L1) Test
TAC (L3) Test
0.4
0.4
0.2
0.2
10
105
102
1.2
1.0
10t
10a
10P
1.0
1.0"
1.07
1.00
(a) HWU64
(b) CLINC150
0.8
MSP
6.7-
TAC (Cos) train
MSP tast
MLS Test
TAC (Cos)
0.4
0.3
6.2
6.1
10-4
10r
(c) ImageNet
Figure 13: VOC Curves.
A.7.2 ACCURACY UNDER VARYING REJECTION LEVELS
To further assess the effect in performance given by rejecting classifiers when evaluated only on high confidence data points, we plot in Figures 14a 14b and test accuracy for various levels of rejection. In more detail, while in the vertical axis we plot the standard prediction accuracy on the underlying task, the horizontal axis corresponds to the fraction of the test set that used for evaluation, and we keep the test instances yielding the highest confidence. For example, if data fraction at the value 0.3 in any of those curves, it indicates that 30% of the test data was used for evaluation, and the selection is made after sorting data points in decreasing order of confidence. We further measure the area under the curves, which is better when higher and upper bounded by the value of 1.
general observation is that, for almost all cases, we observe an expected behaviour: the more reject the better is the resulting prediction accuracy. This suggests that all of the considered confidence scores correlate well with prediction correctness. As before, results suggest that the best confidence score is data-dependent and also dependent on which level of rejection one is able to tolerate in a given application. Nonetheless, TAC can be beneficial over compared statistics.
A.8 ADVERSARIAL DETECTION ON IMAGENET
To further assess the utility of confidence scores derived from TAC, we verify their performance when used to detect dlcision-flipping adversarial perturbations in images from ImageNet Deng et al.) (2009). We consider the black-box attack access model so that an independently trained classifier used generate attacks, and those created from the validation partition of ImageNet. Three attack strategies are considered: Goodflllw et al. (2014) and PGD Madry et al. (2017). both under J, budgets, as well very subtle T.2 CW Carlini & Wagner (0077) perturbations. Predictors
18

---

Published as a conference paper at ICLR 2023
3.98
0.950
0.97
0.96
0.985
0.95
0.980
0.94
(area 93.74%%
0.905
MLS (armu 835%
MSP (area -93.611%
MSP (area 94.541%
TAC (L1) janta 92.651%
0.970
TAC (L11 larca - 94.701%
0.2
0.4
0.6
0.8
1.0
0.2
0,4
0.6
0.8
100
Data fraction
Data fraction
aa) HWU64
(b) CLINC150
0.95
0.90
ML5 (area 74,421
MSP (area =87.49%%
TAC (Cos) 29.697%
0.2
0.4
3.6
6.8
3.0
Data fraction
(c) ImageNet
Accuracy/rejection curves. The horizontal axis indicates the fraction of the test set that was used to measure prediction accuracy, Only the highest confidence test points are used.
Figure 14:
correspond to a pre-trained ResNet-50 where a TAC module is applied on top of projections of its features obtained at four different layers. Further experimental details are discussed in the Appendix. In Table g we compare the scores obtained from TAC with those obtained at the output layer of the base classifier. Namely, we compare L1 distances measured between TAC's activation profiles and the code of the predicted class, with output layer statistics such MPS and MLS. We note that Li distances are more discriminant of input perturbations than alternatives for the three perturbation cases considered.
FGSM
FGD
CW
Det. AUROC (%)
Det. Rate (%)
Det. AUROC (%)
Det. Rate (%)
Det. AUROC (%)
Det. Rate (%)
MPS
71.49
65.70
62.01
58.65
59.36
$6.89
MLS
77.49
70.47
60.24
57.43
61.21
58.25
TAC (T.))
77.94
70.90
63.11
59.67
61.92
58.92
Table 8: Performance on the detection of adversarial perturbations ImageNet images.
B IMPLEMENTATION DETAILS
B. ADDITIONAL TRAINING DETAILS
Optimization and empirical observations Training was performed with Adam in all cases except for models trained on MNIST and CIFAR-10, where SGD with momentum was employed. grid- search over hyperparameters was performed in each dataset, and models reported in the main text correspond to the configuration reaching the highest prediction accuracy on in-distribution validation were performed in single GPU hardware, which required few hours for all
data. All training cases except for models trained on ImageNet, which continued improving until 2-3 days. Overall, we noticed that TAC tends to perform better when weight decay is not applied when its coefficient is
19

---

Published as a conference paper at ICLR 2023
set to very small values (< 50. Moreover, training against Coin required relatively large learning rates compared to commonly used ranges for SGD. For MNIST and CIFAR-10 10 for instance, training to accuracy close to the base non-TAC model required learning rates greater than 1, which usually rendered training unstable. We observed this behaviour to change once we introduced Lee, after which common ranges of learning rate values used in popular recipes would work without change, though TAC still required moderate to low values of the weight decay parameter. Also, We usually needed to put pressure on one of the loss components, and we observed that the combination (a, 3) = (1, 10) worked consistently well.
Choices of distance functions To define D, we either use some Ly distance for E 0, 1, 2, xo or a cosine distance. While we used p = in most cases, we found that the choice of D does not affect TAC's performance significantly. For ImageNet, we found that setting D to the cosine similarity for this model resulted the best-performing approach. Moreover, at testing time, we used only the deepest of the 13 layers to compute TAC's confidence scores since that resulted in improved performance.
Mixup For models trained on CIFAR-10 and ImageNet, we performed Mixup (Zhagg et al. 2017) between pairs of inputs and their corresponding codes. That is, given pair of exemplars ( ('',y', (x", y²). we compute the mixtures of data and codes as given by z az + (1 a a)=", C Cymia aCy' + (1 Cy",", and a Beta(0.2. 0.2). Pairs are created by pairing training mini-batch with random interpolation of its copy, and a is drawn independently for each pair. We present an implementation of the mixing approach we used for training in Figure 17
Adversarial perturbations Adversarial perturbations were created using Foolbox For experi- ments reported in T5 attackers correspond to PGD under Lox distortions up to a maximum of 0.3 for MNIST and 255 for CIFAR-10. Moreover, in this case, we assumed the white-box access model so that attackers have full access to our classifiers, including the code table. Attacks are then created so that activation profiles are moved towards the code of a wrong class (whichever one other than the correct). For the cases reported in Table on the other hand, we consider the black-box case where an external predictor used generate attackers, which are then presented to our models. We created attacks using Torchvision's 's pre-trained ResNet-5( In this case, attackers corresponded to T.o FGSM and PGD, and T.2 CW. The perturbation budget given to attackers in each case was 0.05, 0.02, and 0.1 for FGSM, PGD, and CW respectively. Moreover, evaluation was performed subset of 10 images per class out of the validation sample of ImageNet.
Examples of the types of perturbations we consider are shown in Figure 18 where one can notice how subtle the considered transformations are, and hence how difficult of a problem it is to detect them. Except for the case of FGSM, one can barely notice any difference among those images by inspecting them visually, which is inline with the different detection performances we observed across attack strategies.
B.2 MODEL ARCHITECTURES
B.2.1 MNIST
Models trained on MNIST correspond to 4-layered convolutional stacks where each layer is followed by LeakyReLU non-linearity. The numbers of channels in each layer 64, 128, 256, and 512. To define TAC, we sliced post-activation features output by each such layer into 16 slices such that activation profiles and codes have dimension 64.
B.2.2 CI
Experiments on CIFAR-10 10 were carried out using a WideResNet-28. as described in Madry et al. (2017). Most ResNet's implementations (and its variants) split the model stack into four main blocks. We thus use the outputs those blocks to define TAC. More specifically, in this case, we used the
https://foollbxx how.eeooabbbo..oooorrrreee...
50.
20

---

Published as a conference paper at ICLR 2023
three blocks closer to the output, each outputting 160, 320, and 640 2-dimensional features. Once more, we defined TAC by slicing each set features into 16 slices such that activation profiles and codes have dimension 48.
B.2.3 IMAGENET
For ImageNet, we trained both a ResNet-50 as well as a VIT under the BASE-1 6x1 6 configuration. For the ResNet case, we used the outputs of its four blocks to define TAC, and dimensions of representations output in each such part of the model are 256, 512, 1024, 2048, each split in 256 slices compose activation profiles of size 1024. For the VIT, we used all the 13 transformer layers after averaging across the spatial dimension. That is, we collect 13 768-dimensional vectors across depth define TAC, and set the number or slices in each layer to 256. As discussed in the main text, for the VIT case, we noticed that using a cosine distance during training helped improve performance. Moreover, at testing time, using only the final part of the activation profile and code resulted in the best scoring strategy in this case. In both models, base predictors are pre-trained and TAC is defined on top of projections of features as described in Section 3.2.
B.2.4 TEXT CLASSIFICATION
Architectures for the three text classification datasets we considered corresponded to the RoBERTa BASE configuration. We train base models and use the projection approach described in Section 3.2 to define TAC. As in the case of ViT, we collect 768- (68-dimensional feature vectors across the 13 layers of the model. However, in this case, rather than averaging out the sequential component as in the VIT, we use only the elements corresponding to the -ed-uf-eeueecce token, and 16 slices are considered in this case for each of features.
B.2.5 PROJECTION LAYERS
Projections used to enable the strategy described in Section 3.2 correspond to stacks of fully-connected layers followed by ReLU activations. Independent projections are defined in each layer used to feed TAC. We considered 5 projection configurations named small, large, very- large, x x-1axgeae large, and 2x-1a rge, and the choice amongst those options is treated with cross- validaaoiiation for each dataset we trained on. The numbers of fully connected layers for each configuration is 1, 2, 3, 3 and The x largg and 2x 2x-1arge configurations include LayerNorm operations in the inputs and outputs.
hyperparameter to be selected
B.3 TEXT CLASSIFICATION DATA
For text classification experiments, train models HWU64 Liu al. (2021) }. Bank- ing77 Casanueva et al. (2020), and CLINCI501 Larson et al. 20199) which correspond to sentence- level multi-class classification problems. HWU64 composed of 257 16 sentences and contains 64 classes, and those correspond to different commands for virtual assistants. BANKING77 contains 13,083 data exemplars and 77 classes corresponding to different intents. CLINC150 contains 23,700 utterances and 150 different intent classes, being one of them an out-of-scope or UNKNOWN class, which is ignored during training of TAC, but used testing time for detection evaluations.
B.4 COST OF TRAINING
Training TAC on top of a pretrained model is rather fast. For instance, for the text classification applications, it takes only couple of hours train TAC on single GPU. Here, much of the training time is actually the pretrained frozen model doing inference (so that its activations may be provided as input TAC). If one were to cache these activations as a pre-processing step, this should provide major speed-up on top of the Already-quick training time.
'The e-large configuration includes layerNorm operations in addition the fully connected layers.
21

---

Published as a conference paper at ICLR 2023
B.5 DETECTION RATE IMPLEMENTATION EXAMPLE AND COMPARISON WITH THE EQUAL ERROR RATE
We present Python code snippets in Figure 15 showing that our choice of threshold used compute the detection rate matches that of the standard Equal Error Rate.
import numpy as np from sklearn import metrics
def compute_detectioon
y:
y_score: list [fleat]
float:
"""oomputes the detection rate at the EER threshold.
Args:
Detection (binary) labels.
y_score: Scores.
Returns:
float: Detection rate at a certain threshold.
fpr, tpr, - cew.iicssrooccuveeeeeeo ly, y_scor,, pos [abel=1 fnr = tpr
= np-nanargmin (np. abs (fnr fpr))
return tpr[t]
def compute_ees {
y: list(fl y_score: list [float]
-> float:
"""Computes the Equal Error Rate metric.
Args:
Detection (binary) labels. y_score: Scores.
Returns:
float: Equal error rate.
fpr, tpr, metriest........... roc_curee (y, y_score, pos_label=1)
fnr = 1 tpr
= np. nanargmin (np abs (fnr-fpr))
eer_low, eer_high = min (pr max (fnr(t),, (eer_lowteer_high))o)
return eer
Figure 15: Python implementation of the detection rate reported and the Equal Error Rate (EER).
Our choice of threshold matches the threshold used for EER.
B.6 TAC's SLICE/REDUCE IMPLEMENTATION EXAMPLE
In Figure 16 we show an example of an implementation of TAC's slice and reduce operations on top of 2-dimensional features. These operations repeated in all the layers that be TAC'ed e'dd in model stack and results are concatenated to define the complete activation profiles, and those should match in dimension with class codes.
22

---

Published as a conference paper at ICLR 2023
def cempputt_activatioo.oro.....
features: torch. FloaTTensr,, s_slices: int torch.FloatTensor:
"*"Compute TAC's slice-reduce operations. Expects features with shape [N,C, H, where: N is the batch size. C the number of channels the conv. layer. H and W are spatial dimensions.
Args:
features (torch.PloatTensor): Input features. n_slices (int): Number slices.
Returns:
Forchh.FootTensor: sliced and reduced activations of layer. ***
sitch_size, feature_dimension fia (0), features.
slices_size size feature_dimensicn // n_slices
cotal_ssices length slices size slsiie
(total slices_indiee torch arange (tctal siiiees__llieeeeengtth vvew (i_ewin__cces,, slicos_size)
activation_profile features(;, slices_indices :]
(batch_size, n_slices, -1) activation_profile cativaticn_rrofilee sum (-1, keepdin-True)
activation_profile
return activation_profft
Figure 16: Pytorch implementation of feature slicing and activation profiles computation for a TAC
induced by 2-dimensional convolution layers.
23

---

Published as a conference paper at ICLR 2023
def nixxu__interpolatiinn
cat Forch...oatTense:: one_hot_labelg, torch. code_labels: torch.FloatTensee: interpolation__angee float, 1ist "*"Mixup style data interpolation.
Args:
cata batch Storch.FloaTTensehi batch of data.
one wve_ho__labes (torch.Floateesoor, batch labels in one-hot format.
cocc_labels (torch.Floateensee batch of binary class codes.
Interpolation_range (float, eptional): Concentration param for the Bet distribution.
Returns:
Tuple with batches of interpelated pairs from data_batch. codes, and labels.
permutation_idx torch. carchhrandperm.. device-data__aachh.eeee (aaaaaradddaaaatcchsssssese size () [0),
data_pairs data batch (prrutttpetttt _dd_iixxiiiiiii ...]
label_pairs one htt__hot____beee labell p idx, ...]
code_labcl_pairs = code_labcls oooe__aacollpprrrrtttoooo.o.o (permutation_idk, ...]
Mixup interpolator: random convex combination of pairs
interpolation_fattaa
torch. barrch.istributionn.. wtt.rrrrutioss.eeaaaaat........ Beta (interpolation_ranyy interpolatiion_aagee
rsample (sample_stie shape= (data pairs.size (0),))
to (data batch.
Create extra dimensions in the Interpolation_factors tensor interppolation_ffetrrsssssssssss interpolation_fcctosss (....) (None,) (data_batch.
fatatttornfa__s____at_aa__ san_hht_llatee
= interpolation_ factors
(...,) + (None,) mem_hhetllbellsndde ndim 1)
interpolation
wwwpolaattoooacttosssooo factors_code labels interpolation (...,) + (None,) label_s nddm 1)
interpolation for pair x_0, x_1 and factor given by E* (x (1-t) interpolated batch =
interpelation factors data data_batch
(1.0 interpolationn_acctorsssss
data_pairs
interppllated_labels =
anterpolation__acttrrs__uu + one hat_hhot_lbbbl (1.0 tatteppolation__cctrrrsss_o
label_pairs
interpolaaee__ode_ Labels
interpolation_facter
labels * code labels cade_labels
(1.0 - interpolation www.ooottttttttteeeeeeeeeeooeeet code labels) code Label label_paire
return interpolated interpolated__aabel.. interpolated_coob_iatee
Figure 17: Pytorch implementation of Mixup interpolations.
24

---

Published as a conference paper at ICLR 2023
(a) Clean
(b) FGSM
(c) PGD
(d) CW
Figure 18: Examples of adversarial perturbations.
25


Reviews:
[{"review_id": "zQhORIRWDy", "reviewer": "Reviewer_rL3k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the use of L-length binary codes (TAC) for the use of multi-class classification. Instead of having a one-hot representation, the authors' approach is to produce a binary value from each sliced-up section of the neural network's activations (Figure 1). Models can be trained to output TAC from scratch or via post-hoc add-ons to existing classifiers. ", "summary_of_the_review": "This is clear and interesting work. It needs to spend more time exploring how depth is related to code quality and utility for out-of-distribution detection, and needs to compare to other ECOC work. As is, the work is a 6. If both of my concerns are addressed, it would turn into an 8.", "strength_and_weaknesses": "Thanks for the authors for the hard work on this paper. \n\nStrengths\n------------\n- Appealingly simple idea\n- Clear writing\n- Strong performance\n- Seems quite generally applicable\n\nWeaknesses\n----------------\n- If I understand correctly, each of the binary codes is associated with a slice of the activations. However, activations in the earlier layers are generally worse at predicting the class (and perhaps one code in a vector of them). As such, shouldn't latter (in terms of neural network topology) codes be upweighted (in terms of loss computation) during training, during the distance computation (for OOD detection), and during inference? To assess whether the premises of my question is correct, it would be great to see a plot of (a) training error and (b) OOD distribution per layer: are earlier codes harder to learn than later ones and are they worse at OOD detection? You have something like this in Figure 2, but these are per-sample: how bad is the error vs depth overall? I think this statement in the appendix supports my concern: \"Moreover, at testing time, we used only the deepest of the 13 layers to compute TAC’s confidence scores since that resulted in improved performance\" - what fraction of the code is this? Did you have to do this kind of trimming for any other datasets? This is something important to understand and should be lifted out of the appendix.\n\n- Sec 2.4 - but why *not* use Hadamard matrices? Is there any advantage in using random binary codes specifically?\n\n- How easy is your method to adapt to multi-label classification tasks? How about regression? Would be nice to have a few sentences about this, but I'm not asking for any new experiments here.\n\n- Table 2 - L1 is used for HWU64 and CLINIC150 but cos is used for ImageNet. It's noted in the appendix that you chose cos for ImageNet, Did you choose it on a held-out validation set? Is that the procedure you generally recommend? If so, please make explicit.\n\n- The related work discusses error-correcting output codes. Why not compare to 1-2 of these methods that can be applied to neural networks? I understand they are not as efficient, but it's important to understand how good your performance is compared to previous similar ideas and trade that off with performance gains.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iRWP_hNJhhF", "reviewer": "Reviewer_sw7u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper presents a robust multiclass classification \"add on\" scheme for deep neural networks called TAC (total activation classifiers) in which multiple hidden layers of a network are trained to output error correcting codes (ECOCs) that are used to identify the proper class of an example and guard against spurious predictions in the case of out-of-distribution (OOD) examples and adversarial examples designed to trick the underlying model. Because ECOCs require more bits to be correctly predicted when determining the class label, TAC-augmented models are claimed to be more robust. Specifically, with TAC, the hamming (or cosine) distance to the closest code in the code book can used as a measure of model confidence, and predictions with at least a certain distance can be flagged for review or rejected.\n\nToy experiments on CIFAR-10 show that TAC-augmented neural networks can be trained to output ECOCs at multiple network layers, and that the distances of predicted codes is discriminative at identifying adversarial examples (though not perfectly so).\n\nLanguage model experiments on 3 intent-classification tasks show that TAC improves accuracy over and above the original pretrained RoBERTA and BERT++ models. And experiments on ImageNet data with a pretrained ViT model shows that TAC better identifies missed predictions than other confidence detection measures like the max-logit-score (MLS), maximum-softmax-prediction (MPS), and a recently published approach called \"DOCTOR\".\n\nFinally, on an adversarial benchmark dataset (RobustBench), TAC performs well at identifying attacks in a gray-box scenario where the attackers know the inner workings of the base neural network, but do not have access to the classification codebook. In the full white-box scenario where attackers have the code book as well, TAC is not competitive in providing protection against adversarial attacks.\n", "summary_of_the_review": "Overall, this paper presents a simple and promising extension for adding robustness to both new and existing deep learning models. Theoretical justification is lacking, and some of the experimental results are hard to interpret in real terms. At the same time, some of the experimental results are compelling and show real improvements above commonly used SOTA baselines for misprediction and out-of-distribution identification. Therefore, I am in favor of accepting this paper to ICLR, but not strongly so.\n", "strength_and_weaknesses": "Strengths:\n- TAC is a simple extension that allows one to add safety and robustness to new and existing neural network models.\n- Experiments with TAC show promise in terms of classification accuracy and mis-prediction detection across a range of datasets and misprediction penalty ranges.\n- In gray-box adversarial settings, experiments show that TAC is effective at detecting adversarial examples.\n\nWeaknesses:\n- TAC is an empirical method with intuition, but not theory, justifying the use of ECOCs output by multiple network layers for preventing against spurious predictions. The paper's experiments are suggestive, but are not extensive enough to determine whether TAC provides robust and superior decision support across a wide range of tasks, and compared to other commonly used methods like temperature calibration.\n- While the value-of-classification curve (VOC) was cool and something I have not seen before, many of the experimental results regarding misclassification detection rates are presented in a manner that is rather opaque and hard to interpret in real terms (see comments on paper clarity below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WK2GOZaggZ", "reviewer": "Reviewer_eB98", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the robustness of deep classification models, noting that out-of-the-box models can yield arbitrarily high-confidence predictions on out-of-distribution data. As a proposed remedy, “total activation classifiers” (TAC) are introduced to capture “class-dependent patterns” which are claimed to offer a reliable way to identify test data that do not conform to such patterns. Notably, the approach does not require access to out-of-distribution data at training time.\n\nIn more detail, TAC consists of a predictor mapping data onto the unit cube. The features are extracted from the neural network by taking “slices” from all intermediate activations which are then reduced (via summation) to 2-dimensional objects (e.g., spatial dimensions in convolutional architectures for images). The goal is to “make it so that groups of high-level features fire-up more strongly depending on the class of the input.” The sequence of all such slices is called the “activation profile” whose distance is measured to the classes. The class labels are represented as distinct binary codes; the set of binary codes is chosen in this way to maximise the “discriminability of its elements.” To train TAC, the binary-cross entropy loss is used to obtain class activation profiles which minimise the coordinate-wise distance to the corresponding class; variations on this loss are explored to improve training stability.\n\nThe overall TAC procedure can be applied to both “frozen” pre-trained models to obtain a secondary confidence score as well as trained “from scratch,” at a slight drop in accuracy (A.1). Thus, the recommended approach is to apply TAC to frozen models.", "summary_of_the_review": "Overall, this paper tackles an important problem and the experimental evaluations on robustness to adversarial attacks, ability to abstain, and out-of-distribution detection seem fairly convincing overall. However, I have a few concerns about motivation for the proposed approach and there is an important omission in the discussion of related work.", "strength_and_weaknesses": "Overall, this paper tackles an important problem and the experimental evaluations on robustness to adversarial attacks, ability to abstain, and out-of-distribution detection seem fairly convincing overall. However, I have a few concerns about motivation for the proposed approach as well as some lacking discussion (and perhaps experiments) of related prior work.\n\nThe approach is not well-motivated. In the introduction, the question is posed, “can we enforce this property rather than hope it emerges?”, but the reader is left wondering what property is actually under discussion. It would be helpful to include further discussion of the property of interest to motivate why it’s sufficient to target this to obtain “models that know that they don’t know.” \n\nThe use of binary codes is motivated by the need to have well-separated targets for optimization. However, in S2.4 it is stated that the codes are randomly selected with coordinates being independent binary random variables. Do the codes actually need to be binary? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster