Improving Compositionality of Neural Networks by Decoding Representations to Inputs
Authors: Mike Wu
Noah Goodman
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative models
decoder
representations
out-of-distribution
compositionality

Abstract: In traditional software programs, it is easy to trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together. Although deep learning programs have demonstrated strong performance on novel applications, they sacrifice many of the functionalities of traditional software programs. With this as motivation, we take a modest first step towards improving deep learning programs by jointly training a generative model to constrain neural network activations to "decode" back to inputs. We call this design a Decodable Neural Network, or DecNN. Doing so enables a form of compositionality in neural networks, where one can recursively compose DecNN with itself to create an ensemble-like model with uncertainty. In our experiments, we demonstrate applications of this uncertainty to out-of-distribution detection, adversarial example detection, and calibration --- while matching standard neural networks in accuracy. We further explore this compositionality by combining DecNN with pretrained models, where we show promising results that neural networks can be regularized from using protected features.

Improving Compositionality of Neural Networks by Decoding Representations to Inputs
Mike Wu, Noah Goodman, Stefano Ermon
Department of Computer Science
Stanford University
Stanford, CA 94303
{wumike, ngoodman, ermon Astanfordd edu
Abstract
In traditional software programs, it easy trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together. Although deep learning programs have demonstrated strong performance on novel applications, they sacrifice many of the functionalities of traditional software programs. With this as motivation, we take modest first step towards improving deep learning programs by jointly training generative model to constrain neural network activations "decode" back inputs. We call this design Decodable Neural Network, or DecNN. Doing so enables form of compositionality in neural networks, where one can recursively compose DecNN with itself to create an ensemble-like model with uncertainty. In our experiments, we demonstrate applications of this uncertainty to out-of-distribution detection, adversarial example detection, and calibration - while matching standard neural networks accuracy. We further explore this compositionality by combining DecNN with pretrained models, where show promising results that neural networks can be regularized from using protected features.
Introduction
Traditional hand-written computer programs are comprised of a computational graph typed variables with associated semantic meaning. This structure enables practitioners interact with programs in powerful ways (even if they are not the author) such as debug code by tracing variables back to inputs, apply assertions block errors, and compose programs together for more complex functionality. However, traditional software has its limitations: it difficult to hand-write programs classify images extract sentiment from natural language. For these functionalities, deep learning and neural networks
[1 have become the dominant approach 23 46
While neural networks have made impressive progress complex tasks, they come at sacrifice of many of the desirable properties of traditional software. Specifically, the closest approximation to "variable" in neural network is activation. Yet it is difficult to understand neural network's computation from an activation value, and there little to associate an activation with semantic meaning, A practitioner cannot write assertion statements constrain valid neural network logic: checking the values that an activation takes is usually not enough to gauge correctness nor meaning. Moreover, given multiple neural networks, composing them together requires retraining from scratch.
In this paper, we take modest first step towards bridging the expressivity of deep learning with the engineering practicality of traditional software. We aim to uncover ways for practitioners to build and use neural networks by leveraging compositionality. Specifically, we propose to train a neural network classifier jointly with generative model whose role is to map the classifier's activations back to the input, approximating invertibility. For any input, a neural network's 's computation can
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

be represented by sequence of activations (from each layer), each of which can now be mapped back to the input space. It is this insight that enables special form of compositionality for neural networks, as inputs derived from activations can be fed back into other models.
In our experiments, study this compositionality by (1) recursively composing neural network with itself, creating an ensemble-like model with measure of uncertainty that is useful for out-of- distribution detection, adversarial example detection, and model calibration; (2) composing neural networks with pretrained models as forms of regularization and distillation; and (3) using decodable activations to discourage neural network from using protected attributes. Throughout, we show that decodability comes at low cost as we find equivalent accuracy to a standard neural network.
2
Background
Neural Networks We will focus on supervised neural networks for classification, although is simple to extend to regression tasks. We denote a neural network by fa where 0 represents its trainable parameters. A neural network Sa maps an example æ to predicted probabilities over K' classes. Typically, neural network is composed of L blocks. For example, if So is multi-layer perception, each block linear layer followed by a non-linearity. If fe is a ResNet [17), each layer is residual block. In computing its prediction, neural network fe produces activations (/1....,hy. where hy is the output of the (-th block.
To train a neural network, we solve an optimization function of the form
(an(a; 0) = log pogp(yl(((x)) + (x;;;;;
using stochastic gradient descent. In Equation [ the notation denotes an input example and its label. The function 52 represents an auxiliary objective such as a regularizer. The hyperparameter 0 is used to scale the auxiliary loss. For classification, log p(v) -) is cross entropy.
Generative Models Broadly, we are given latent variable and an observed variable X. generative model Is captures the distribution p(z) h), mapping samples of the latent code h to samples
Bdder byart of the nec natuert, fo
hat
Uncertatee
Facanire Shanng
no hat
Hisfer
corl-cetvoy lnss
has hat
Decantsaction
has hat
Tris deoode Icveetrr at iiput
Sharna
los
Trae libe
fu
no hat
Douddd octvetuus souh roco stuuction
Coetiaanss roceeS2
inturee ard ppens
(a) Decodable Neural Networks
(b) Ensembling through self- Composition
Thas hat
has fast
Conotran rifa, be deso unifer S.A. protected Nemure
has hair
has hat
Pratesined clasthen Ing hair cortancingg
Prstrateed Steas deccdded sctivssin trpect
Unfoon
(c) Composition with Pretrained Classifiers
(d) Constraints with Protected Features
Figure 1: Decodable Neural Nenworks and Applications: The decodable neural network (DecNN) is composed of main network. f and generative model, Each activation in ʃ is decoded back to inputs through Subfigure (a) shows an incorrect classification: the network predicted the man not wearing a hat despite true label. Consequently. we see the decoded activations "interpolate" as the hat is removed, Subfigures (b-d) show the compositionality DecNNs. First, (b) shows self-composition, where decoded activation can fed input back into the network f create ensemble with uncertainty. Second, (c) shows how a pretrained classifier, m can composed with DecNNs for regularization. Finally, (d) shows how to constrain DecNNs to ignore "protected features" through pretrained networks by regularizing for uniformity.

---

of the observed variable r. For a deep generative model, Is is parameterized by neural network. In the simplest case where Ie is an autoencoder, we are also given an encoder q that maps observed samples to latent codes h. Then, we optimize the objective
(a) Φ) = log peppp(1gg ((=)))(((()))
(2)
where for continuous v, the expression log p(r(4) is the log Gaussian probability with mean l and fixed variance (or equivalently, mean squared error). More complex generative models might impose a prior distribution on h [22), an adversarial discriminator [29 [1]. or invertibility of g [35]
3 Neural Networks with Decodable Activations
In Section [ motivated our desiderata for activations that can be mapped back to the input space. We now propose to do this by "inverting" a neural network at a particular activation. Concretely, given an input 2; neural network fo. can compute activation h by computing the forward pass (x)). Then, suppose we could construct the set:
Shh = {2: ~ PD. fo(ă)= = h}
(3)
where Pp represents the empirical distribution that x sampled from. Now, any input à E S(h) is a valid mapping of h back to the input space. (Trivially, activation h. Unfortunately, in practice we do not know how to construct such a set S(h).
S(h).) We call this ê decoding of the
But, we do know how to (approximately) sample from S(h). We can train a generative model Jo to reconstruct the original input x from neural network activations (recall can sample r ~ (h), which we know how to do, to sample from S(h). If we jointly train this generative model Io with the neural network fa. we can optimize for activations that both solve the classification task and decode back to inputs once.
S(h)). Then, we
So far, we have only considered single activation despite our neural network having T. such activations. Although one could train a separate generative model per layer, we found it efficient and performant to share weights ø across all layers. Our joint objective is:
C( 0, の) = log log (o +
log pogg(z)go)
fi1..., hy come from fa(z)
(4)
where the first term is cross entropy as in Equation and the second term is a reconstruction error (Equation 2) averaged over layers. Comparing Equation 4 to Equation D can interpret the generative loss as a regularization term Ω. Intuitively, in a standard neural network, the activations hi,..., hr are unconstrained, chosen only minimize the loss. However, in Equation 4 the network fa is constrained to produce activations that the generative model Io can invert. In practice, is Re decoder with upsampling layers (see Appendix A.I. We call this setup Decodable Neural Network, or DecNN. Figure T depicts an illustration of the DecNN architecture.
We highlight that although Equation 4 requires good reconstruction, it does not enforce that recon- structed inputs f (g)( must map to the same label y as fzz). We show a few examples of this by visualizing decoded activations throughout network in Appendix A.3
4 Composing Decodable Neural Networks
In traditional software, we can easily compose two compatible functions Si and f2 together by (x)) for some input x. With this as loose inspiration, we wish to similarly compose neural networks together. As motivation, we may want to do this to faciliate interaction between networks, such as for regularization or distillation. But given two image classifiers, it is not clear how to tie outputs from one neural network to inputs for the other? Both networks expect images as input.
The workaround comes from decodable activations. Since they are images of the same shape as inputs, we are free to treat them as inputs for another model. So, we can compose neural networks together using decoded activations as an intermediary: given two models fi and fz, and an input x, we can take an activation h coming from (x), decode it, then provide to the other model, f2(go(h)). In the following subsections, we explore two instances of this compositionality: one by composing a model with itself (i.e., fi f2), and the other by composing a model fi with a pretrained one, f2. See Figure Ib.c for graphical depiction of both compositions.
3

---

4.1 Recursive Self-Composition
Our first composition will be to compose a DecNN model with itself. But if we can do that, we can also recursively compose DecNN model with itself infinitely:
Given a decoded activation go (h( (0)) for any layer I, we can feed it as input back into the classifier,
(0)) )) and maximize agreement with the true label y. Intuitively, if go (hy) is a good reconstruc- tion of the input x (i.e. (h()) and belong to the same set S(h)),, then good classifier fo should
(o(gs(h)
predict the correct label with it as input. Now, we also observe that the computation fo (1x (0)) produces I. additional activations, 11 ,..., .i (1) where we use the superscript to represent the number of compositions. We can repeat this exact process with the same intuition, decoding ht and passing it to fa to produce h Computational limits aside, this can be repeated ad infinitum.
One interpretation of this procedure is as data augmentation: we saw in Section A.3 that decoded activations share features relevant to the prediction task (e.g. generated celebrities wear hats) but vary other irrelevant features (e.g. face shape. hair color, etc.). Notably. these generated images not in the original dataset. Thus, training a model using decoded activations as inputs is similar to augmenting the initial dataset. The only difference is that gradients are propagated back through recursion to the inputs, which is also a form of regularization.
Admittedly, what we have described so far is intractable. Besides recursing infinitely, each activation produces L more activations, creating tree (with root node x) with large branching factor L. In practice, we make two adjustments - First, we limit the recursion to max depth D. Second, instead of constructing the full graph of activations, which would have OLLP) nodes, we randomly sample
"paths" in the graph. Specifically, starting the root, we compute {11 ..... / through fo(x)) and (0)
randomly choose an integer 4 E [1, L]. Then, we compute (h ..... hL through (o (h(2)) ie. we classify the lo-th decoded activation from depth O. Again, we randomly pick 4 to compute ................................ [h!")...,,h, { through (g (h,,)) and repeat until depth D.
Building on Equation 4
we can write this "recursive" objective
L(:; 0, Φ) = (ogp(y)f((x))
Σ ad logg(yl (1
+33 Σ a log (11111)))) (5) d=1
where the sequence of integers lo.... 1D1 are each sampled uniformly from [1. L]. We introduce a new hyperparameter a E [0, 1] that geometrically downweights later depths (if 0. Equation reduces to maximum likelihood). The first term in Equation the usual classification objective. The second term sum of cross entropy terms that encourage the decoded activation depth d to predict the correct label through fe. The third and final term is a sum of reconstruction losses that encourage the decoded activation at depth d to approximate the original input x.
We call this model a "Recursively Decodable Neural Network", abbreviated ReDecNN. Note that ReDecNN is a special case of an ensemble network. That is, every path sampled in the activation tree by picking lo.... I builds legitimate classifier. There are ((T,P) such classifiers implicit in the ReDecNN, We can interpret optimizing Equation (5 aa training individual classifiers that are Monte Carlo sampled from the tree every gradient step. See Figure Ip for an illustration.
Our goal in the next few paragraphs is to utilize this ensemble to measure uncertainty.
Method
MNIST
Fashion
ColebA
Method
MNIST
Fashion
CelehA
Standard
97.1 (0.11)
87.7 (0.19)
90.8 (et..)
ReDecNN
0.869
0.795
0.692
DeeNN
97.8 (0.17)
88.8
90.8 (c0.1)
MC-Dropon
0.878
0.818
0.657
ReDocNN
97.5 (0.20)
88. 1 (0.16)
(0.8
BayesNN 0.537
0.574
0.587
Dropout
95.9 (0.11)
80.7 (0.34)
Ensemble 0.531
0.559
0.605
MC-Dropour
91.33
78.6 (0.23)
87.8 (0.19)
BayesNN
96.3 (0.51)
87. 0. (0.36)
88.2 (0.22)
(b) ROC-AUC of separating correctly classified
Ensemble
97.6 (0.11)
88.4 (0.12)
90.9
and misclassified examples using Eg. a score.
(a) Classification performance on test set.
The stdev. for all entries over 3 runs are <0.01.
Measuring Uncertainty As a first step, we compare ReDecNN to several baselines: a standard neural network, neural network with dropout, two kinds of Bayesian neural networks - Monte

---

Carlo dropout [TI]. abbreviated MC-Dropout, and weight uncertainty 4 abbreviated BayesNN and naive ensemble network where we train D copies of the standard neural network with different initializations. For all experiments, see Appendix A. and A.2 for training and task details. Table la shows accuracies over a held-out test set, averaged over three runs. While we observe lower accuracy from Bayesian NNs, we observe (perhaps surprisingly) equal performance of DecNN and ReDecNN to a standard neural network. Typically, adding uncertainty to neural networks comes at a cost to performance (as is true for other baselines), but this seems to not be the case here.
The next step to compare the quality of model uncertainty. For the ReDecNN, for given input æ, we measure uncertainty as follows: sample N different classifiers from the activation tree, and make predictions with each classifier on z. The uncertainty is the entropy of the predictions:
Uncertainty (1) --) Σ = c)g pog c))
c
(6)
where ply = c) is the empirical probability (e.g. normalized count) of predicting class out of N classifiers. A higher uncertainty metric would represent more disagreement between classifiers. We can compute the same metric for naive ensemble, as well as for MC-Dropout (where repeated calls drop different nodes) and BayesNN (by sampling weights from the learned posterior).
We want to test if the model's uncertainty is usefud. One way is to correlate the uncertainty with when the model makes prediction errors (we call this "misclassification")) It would be useful if the model was less uncertain on correct predictions and more uncertain on incorrect ones. A practitioner could then use uncertainty to anticipate when a model might make a mistake. Table Ib reports the area under the ROC curve, or ROC-AUC of separating correctly and incorrectly classified examples in the test set (using uncertainty as a score). A higher ROC-AUC, closer to I, represents a better separation. We find that ReDecNN is competitive with MC-Dropout, even out performing it on CelebA. In Section A. we include examples of images with low and high uncertainty.
Distribbbiinn Detection A second application of uncertainty is detecting -uttof-distribution (OOD) inputs. Given an input that is far from the training distribution, model is likely to make a mistake. Instead, the model should recognize its uncertainty, and refuse to make a prediction. There a rich literature on detecting OOD inputs using downstream computation on trained model 19). Unlike those works, we study OOD detection using uncertainty alone.
18 25 24 52
MNIST
MosSS MC
CelebA MC
ReDecNN
Ensemble
ReDecNN
Ensemble
ReDeeNN
Ensemble
Advensarial
0.787
0.817
0.540
0.711
0.760
OOD (MNIST)
0.793
0.864
0.501
0. 0.847
0.548
0.591
ODD ()ashionMNISS)
0.812
0.888
0.334
0.641
0.572
0.399
OOD (ColehA)
0.893
0.943
0.675
0.753
0.793
0.704
Comput (Mean)
0.727
0.678
0.691
0. 0.886
0.569
0.632
Corrupt (Sidev)
0.075
0.113
0.087
0.072
0.108
0.106
0.03d
0.017
0.016
Table 2: ROC-AUC of predicting which examples are out-of-distriuttion (OOD). We vary OOD examples to be adversarial, corrupted, or from a different dataset. Standard deviation for all entries are < 0.01 over three runs.
We explore three kinds of OOD examples: (1) adversarial examples crafted for a single classifier using FGSM (15), (2) examples from different dataset (e.g. train on MNIST and use FaSH as OOD) as done in (25,, and (3) corrupted examples by 14 image transformations (e.g. adding pixel noise), borrowed from BT. See Appendix Jppendix/A.4ffor expanded corruption results.
Table reports the ROC-AUC of separating inlier examples taken from the test split of the dataset the models were trained on, and outlier examples. From Table 2 we observe ReDecNN is just under MC-Dropout, but while MC-Dropout achieves these results at the cost of classification accuracy, ReDecNN does not (Table Tap. Furthermore, we find ReDecNN generalizes better to CelebA, more complex image dataset, where it outperforms MC-Dropout (and other baselines).
Focusing on CelebA, we study a domain OOD challenge by holding out all images with positive annotations for a single attribute as nut-of-distribution. Here, inlier and outlier examples are very similar in distribution. We report ROC-AUCs in Tar three held-out attributes (randomly chosen): is the celebrity in the image "wearing a hat", has "blonde hair", or "bald"? As this is more challenging problem (since inlier and outlier examples are from the same dataset versus different But while MC-Dropout, BayesNN, and a naive ensemble all perform near chance, ReDecNN is consistently near 0.6.
datasets), the performance lower than in Table 2
5

---

OOD: Wearing a Hat
OOD: Bland Hair
OOD: Bald
Method
ROC-AUC
Method
ROC-AUC
Method
ROC-AUC
ReDecDNN
0.604 (41.01)
ReDecDNN
0.593 (c0.01)
ReDecDNN
0.615 (4h.11)
MC-Dropout
0.519 (ct01)
MC. Drapout
0.502 (c001)
MC: Droppot
0.502 (et0l)
BayesNN
0.510 (001)
BayesNN
0.508 (0.01)
BayesNN
0.508 (c001)
Ensemble
0.526 (c0.1)
Ensemble
(c0.1)
Ensemble
0.503 (c001)
Table 3: We hold out a group of CelebA attributes, such as those wearing a hat, when training. We compute the ROC-AUC of labeling the held-out group as OOD (mean and stdev. over runs).
Calibration A third application of uncertainty we explore is calibration of predicted probabilities. Standard neural networks are notoriously over-confident and are incentived in training to predict with extreme probabilities (e.g. 0.99). A model with useful uncertainty would make predictions with proper probabilities. Although many calibration algorithms exist [33 [6), we want to measure how calibrated each model is out-of-the-box to compare the quality of model uncertainties.
Table 4a reports the expected calibration error, ECE (32), which approximates the difference in expectation between confidence and accuracy using discrete bins. A lower number (closer to 0) represents a more calibrated model. We observe that while standard neural network has a relatively high ECE, many of the approaches (dropout, MC-Dropout, BayesNN, and ReDecNN) reduce ECE. On the other hand, naive ensembles increase the calibration error, as they are more prone to overfitting. In two of three datasets, ReDecNN achieves the lowest ECE whereas MC- MC-Dropout achieves the lowest in the third. These results should be viewed in parallel to Table Ia which measures "sharpness". Otherwise, we can trivially reduce ECE to zero by ignoring the input.
Method
MNIST
Fashion
CelebA
Depth
Acc.
OOD (Fashion)
0OD (ColehA)
Standard
1.18 (0.10)
0.56 (0.14)
2.75 (0.12)
ReDecNN
0.33 (0.05)
0.21 (0.0))
(0.13)
97.7
0.665
0.752
Dropout
0.63 (0.00)
0.48 (0.09
2.51 (0:13)
4
97.6
0.788
0.804
MC MC-Dropaul
0.41 (0.06)
0.13 (0.03)
2.99 (0.15)
97.7
0,805
0.868
BayesNN
0.91
0.18 (0.0)
2.41
97.5
0.812
0.893
Ensemble
1,80 (0.12)
0.89 (0.07)
3.12 (0.12)
(b) Effect of recursion depth D on classification accuracy and OOD detection for a ReDecNN model (8-layer MI.P) trained on MNIST.
(a) We compare the expected calibration error (ECE) of ReDecNN to standard neural network with var- ious regularization and ensembling.
Finally, we study the effect of depth on the efficacy of ReDecNN. For the experiments above, we chose the recursive depth to be equal to the number of layers the MLP (D = T, = 8). In Table 4b) we vary the depth from 2 to 8 (while keeping the number of layers fixed to and study the effect on classification accuracy and OOD detection. We find increasing OOD performance as D increases, although with marginally decreasing gains. Moreover, we observe constant accuracy, matching standard neural networks regardless of the choice of D.
Effeet of Depth
4.2 Composing with Pretrained Models
Apart from self-com celf-composiiinn, we can also compose our neural networks with off the--hh-hhf pretrained models as a form of regularization or distillation. Suppose we have a pretrained model m that maps an input to class prediction. We can modify the ReDecNN objective as follows:
6(x;0,0) Φ) ((r;0,0) + Σ ad (oggttolooo000""""00m0110 (((-))))
d=1
where Ć is as defined in Equation The additional term acts as a divergence, bringing the classifer's predictions close to those of the pretrained model. A similar edit can be made the DecNN. We revisit Section experiments now using two pretrained models - a linear classifier or a ResNet-18 classifier. For each, optimize Equation 77 and compute performance
OOD detection.
Table 5 compares the results composing with either (1) a linear model, (2) a residual network, or (3) no composition (None). Using linear model, we observe slight drop in test accuracy but an increase across OOD experiments and calibration. fact, these new results rival surpass
MC-Dropou from Table 2 Conversely, with ResNet-18, observe point increase in test

---

MNIST
CelebA Linear
Method
None
Linear
ResNet
Nome
ResNet
Accuracy
97.50 (0.20)
97.17 (0.12)
98.30 (0.24)
90. (d0.1)
89.76 (0.11)
91.28 (0.13)
Misclassfication
0.869 (chol)
0.904 (c0.1))
0.742
0.692 (ch.01)
0. 0.732 (c001)
0.615 (c0.01)
Adversorial
0.787 (c0.0))
0.805 (0.00)
0.729 (c0.01)
(MNIST)
0.647 (c001)
0.693 (4001)
0.609 (c0.01)
OOD (Fashion)
0.812 (cb.0l)
0.845 (<0.01)
0.650
0.641 (c0.01)
0.673 (c001)
0.585 (c0.01)
OOD (CelebA)
0.893 (c0.01)
0.928 (c0.01)
0.802 (c0.01)
Corruption (Meun)
0.727 (c0.01)
0.790 (x0.01)
0.694 (cb,01)
0.686 (c0.01)
0. 708 (<0.0)
0.616 (<0.01)
Calibration (ECE)
0.334 (0005)
0.296 (0.0))
0.385 (0.02)
1.927 (0.13)
1.937 (0:04)
3.002 (0.10)
Table 5: OOD detection using uncertainty for ReDecNN composed with pretrained models.
accuracy but notable drops in OOD and calibration results. These polarizing outcomes show two use cases of pretrained models: using simpler models allow for regularization that trades off accuracy for robustness while more complex models encourage distillation, making the opposing tradeoff.
An open question how much Equation 7 ben-
efits from regularizing fe (h)) to be close to mk) versus regularizing it to be close to m(x). The former is a pretrained embedding of recon- struction whereas the latter of a static input. Ta- ble compaare the two on MNIST, where we call the latter "direct regularization". We observe that Equation ( outperforms direct regularization (other than calibration), often by a large margin. We rea- this to be because the input and the recon- struction (h) can be very different, especially in later layers shown in Appendix [A.3] This im- plies that mxxx and m(g(h) can be very different as well. Intuitively, constraining the model be similar what a pretrained model thinks of the reconstruction (which dynamic value that varies with model weights) is a stronger learning signal than what the pretrained model thinks of the original input (which is static). In other words, regularizing fe ((()) to be like m(z) at all layers / is too strict of a constraint.
Method
Direct Reg.
Accurecy
97.17
96.72
Misclass.
0,904
0.828
Adversarial
0.805
0.707
0OD (Fashion)
0.841
0.699
0OD (CelebA)
0,928
0.782
Corruption (Mean)
0.790
0.667
Calibration (ECE)
0.296
0.280
Table Composition (Eq.
versus direct regular-
ization with pretrained models on MNIST.
5 Constraining Decodable Computation
A milestone for deep learning would be the ability to specify what information an activation should not capture. This is especially important in cases when we have protected attributes that should not objective. While this milestone remains out of grasp, we take small step towards it by optimizing a neural network to "be indifferent" with respect to a protected attribute through composition with pretrained models.
be abused in optimizing
Suppose we have access a pretrained classifier m that predicts the assignment for K-way protected feature. Then, building on Section 4.2] we can optimize our classifier to "ignore" information about the protected feature through m. To do this, we define the objective:
[(x; = ((z;0,0) La logp(m(g
(M (M 1k)
(8)
K
dml
where 1x is K-dimensional vector of ones. That is, Equation 8 encourages the decoded activation go (oo("("))) (2-1)) when passed through the protected classifier m, to predict chance probabilities i.e. have no information about the protected attribute. high performing classifier trained in this manner must have solved the prediction task without abusing the protected attribute(s).
To test this, we extract two attributes from CelebA: "beardedness" and "baldness". Suppose we wish to design a neural network to predict beardedness. If we were to naively do so, there could be discrepancy performance between groups of individuals for example, between bald individuals and individuals with hair, the latter group is more common in the CelebA dataset. Indeed, Table shows a point difference Fl and a point difference in average precision (AP) in classifying beardedness between the two groups. The second row of Table provides bascline ReDecNN
7

---

Model
Fl (Bald Not Bald)
AP (Bald Not Bald)
ECE (Bald Not Bald)
Standard
0.338 0.338/0.440 /0,438 (40.11 <0.0:
19.4/23.9 (0.080009)
4.76/3.50 /3.56 (0.0/0.00)
ReDecNN (Eq.))
0.327 /0.44 (cha1
18.6/22.3 (0.11/0.44)
4.74/3.54 (008
ReDecNN (Eq.
0.461/0.501 1001 (001)
27.2/28.9 (0.10/0.10)
3.80/3.28 3.203 (0.12/0.09)
Table 7: We pretrained model on CelebA that predicts baldness optimize the activations of second classifier to ignore baldness when predicting beardedness.
optimized without knowledge of the protected attribute. Unsurprisingly, we find similar discrepancy between groups, as with standard neural network. In the third row, we evaluate ReDecNN that was optimized to ignore the "baldness" attribute using pretrained classifier for baldness. Critically, we find much more balanced (and higher) F1, AP, and ECE across two groups.
One unsatisfying but plausible explanation for Table 7 is that the generative model learned construct adversarial examples such that the pretrained model m cannot predict the protected feature, but that the protected feature still being used to make predictions by f. To show this is not the case, we take a trained ReDecNN optimized by Equation g freeze its parameters, and small linear head on top of the Jast hidden layer to predict the protected feature. If the model learned ignore this feature, it should not be able to perform the task well. Table reports a poor FI score (0.1) 1) whereas doing the same with standard neural network or with unconstrained ReDecNN gets 3 times the FI score.
Methad
F1
Standard
0.383
ReDecNN (Eq.
5
0.371
ReDecNN (Eq. 8
0.108
Table 8:
Quantitatively accessing
ability to classify baldness.
6 Generalizing to other Modalities
While we have focused image classification, the proposed approach is more general and can be extended to other modalities. We apply the same decodable representations to speech, in particular utterance and action recognition. We compare DecNN and ReDecNN to the same baselines, measuring accuracy and uncertainty through similar experiments as we did for images.
Method
Acc
Misclass.
O0D
BCE
Methol
Acc:
Misclass.
OOD
ECE
Standard
(0.1)
Standard
(0.4)
DecNN
93.8 (0.2)
DecNN
41.4 (0.7)
ReDecNN
93.4 (0:2)
0.766
0.705
0.225
(0.1) ReDecNN
41.2 (0.5)
0.642
0.605
0.523 (0.2)
MC. Dropout
82.1
0.788
0.745
0.429
MC Dropout
34.5
0.629
0.562
0.515
BayesNN
91.6 (0:7)
0.545
0.518
1.039 (0:4)
BayesNN
40.3
0.523
0.500
0.918 (0.1)
Ensemble
96.3
0.529
0.506
(0.2) Ensemble
44.1 (0.1)
0.541
0.522
1.189
(a) AudioMNIST
(b) Fluent Speech Commands
Table 9: Performance on speech classification. If not specified, stdev. is <0.01 over three test runs.
We utilize the AudioMNIST 3 and Fluent speech com- mands 27 datasets, the former being a corpus of 60k utterances of the digits 0 through 9, and the latter corpus of 100k recordings 97 speakers interacting with smart- home appliances for one of six actions. Audio waveforms are preprocessed to log Mel spectrograms, outputting 32 by 32 matrix [47]. Figure shows the input on the left- most column along with the 2nd, 4th, 6th, and 8th decoded activations for random test examples. Table reports the findings, where like the image experiments, we find ReDecNN has strong performance on OOD detection and calibration return for only a small drop in performance compared to standard neural network.
Figure 2: Decoding activations to im- ages for Fluent spectrograms.

---

7 Related Work
Although DecNNs are reminiscent of autoencoders, there are differences: while autoencoders only reconstruct the final hidden layer, DecNN has a reconstruction for every single hidden layer. This, and because DecNNs have a supervised objective (which autoencoders do not have) critically changes what information is captured in the hidden layers. Whereas the autoencoder seeks perfect reconstruction, the DecNN does not, as evidenced by loss of information for reconstructions later in the network (see Appendix A.3) Finally, we point out that the applications of autoencoders are very different from the DecNN. The former is not used for calibration or composition.
Autoencoders
Invertible and Ensemble Networks The DecNN is akin to invertible generative models 6 as DecNN is training generative model to "invert" the classifier up to hidden layer. However, unlike invertible flows, DecNN does not impose an explicit prior on the activations (i.e. latent variables), and further, DecNN has supervised objective. Our proposed recursive network, ReDecNN, has similarities to Bayesian neural networks [I 4 and ensemble networks 48 two baselines we compare against for evaluating model uncertainty. However, we find NeDecNN to be easier to train than doing inference over weights, and cheaper in parameter count than naive ensembles.
Probing Neural Networks A related line of work seeks to probe neural network post-training to understand the underlying computation []. Most relevant to our work is an approach that inverts
CNN activations through reconstruction (28). Unlike this approach, we are not proposing any additional computation post-hoc. Rather, we are interested optimizing neural networks such that their representations are more easily mapped back to the input domain.
Robustness and Protected Attributes
Out-of-distribution detection
26
24
52
40
20
selective classification 12
13
adversarial perturbation detection 49
T0
30
34
and neural
network calibration [6
51
50
32
38 each have a rich subfield of algorithms. While task-specific
methods likely outperform our results, we are excited about representations that enable uncertainty without task information or additional algorithms. Our approach also shares similarities where the authors minimize the mutual information between learned representations and a protected attribute. We view Equation as an approximation of this where we treat the pretrained classifier for the protected attribute as proxy for the mutual information.
8 Limitations and Future Work
To close, we discuss few limitations. First, approach is bottlenecked by the quality of the gen- erative model. Without good reconstruction, op- timization will be intractable. However, in light recent work [45] 44 5 21 43), this is becoming less of a problem as new generative models surface. Second, in the main text, we only explored feedfor- ward classifiers for simplicitly. Our approach ex- tends naturally residual and transformer blocks, and future research could explore this direction. As a start, we provide experiments for extensions to CNN architectures in Appendix A.6 Third, opti- mizing recursive networks, although parameter efficient, costs more compute as backpropagation more expensive. See Table 1O for timings of epoch in seconds. The proposed method does have significant computational overhead, averaging about 6x cost, while baseline methods like Dropout and MC-Dropout impose little to overhead. On the other hand, ReDecNN has comparable cost to DecNN despite requiring recursive gradients. Also, both DecNN and ReDecNN are both cheaper than naive ensembles, significantly in the CelebA case. Future work could explore weight sharing across layers (not just depth) to reduce compute.
Method
MNIST
Fashion
ColobA
Stundard
13.6 (0.3)
13.5 (0.)
82.6 (6.3)
DecNN
80.5 (6.8)
99.2 (96)
486.2 (129)
ReDecNN
96.7
(6.6
(21.6
Dropout
13.9
(4.0
(9.5
MC-Dropout
10.9
14.0
79.5 (5.7)
Ensemble
10..6
132.6 (8.5)
659.4 (82)
Table 10: Cost (seconds) of 1 epoch on a Titan X GPU (averaged over 10 epochs).
In summary, we explored building neural networks with decodable representations, and leveraged this new property to compose models together. By re-purposing decoded activations as novel inputs, we were able to join neural networks together in useful ways for distriibitii detection and calibration, as well as influence what information a network retains in optimization. We are optimistic about promising results and look to future research for more applications of decodability.

---

Acknowledgments and Disclosure of Funding
We thank the Ermon group for their comments and suggestions. We thank the reviewers for their many iterations of feedback that helped this paper improve significantly. This research was supported in part by NSF (#1651565, #1522054, #1733686), ONR (N000141912145) ONR MURI (N000141612007 AFOSR (FA955019 ARO (W91 INF-21-1-0125), and Sloan Fellowship. MW is supported by the Stanford Interdisciplinary Graduate Fellowship as the Karr Family Fellow.
References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprini
2014.
[3] Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, 0tts/1807.03415 2018.
[4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[5] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233 2021.
[6] Laurent Dinh, Jascha Sohl- Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605v 2016.
[7] Piero Esposito. Blitz - bayesia layers in torch zoo (a bayesian deep learing library for torch). https://sg http://ggtttttt..ee/eee httpb///////ppppttoo///tttt--- http://tobbbttt-bbbabia-e-deeeee -ayessann-de---erniing 2020.
[8]
et al. Falcon, WA.
Pytorch lightning. 3, 2019.
GirHub.
Note:
[9] William Falcon and Kyunghyun Cho. A framework for contrastive self-supervised learning and designing new approach. arXiv preprint arXiv:2009.00104 2020.
[10] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint
000 2017.
[11] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[12] Yonatan Geifman and Ran EI- Fa-iv..v. Selective classification for deep neural networks. arXiv preprint arXiv: meo:: 2017.
[13] Yonatan Geifman and Ran El- Yaniv.... Selectivenet: A deep neural network with an integrated reject option. International Conference on Machine Learning. pages 2151-2159, PMLR. 2019.
[14] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprins arXiv:1406.2660 2014.
[15] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv: 1412,6441226572 2014.
[16] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pages 1321-1330. PMLR, 2017.
10

---

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. arXiv preprint 1610.0: 36, 2016.
[19] Dan Hendryeks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: ww../......00.. 2.0666, 2018.
[20] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances Neural Information Processing Systems, pages 15663-15674, 2019.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[23] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1 1105, 2012.
[24] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[25] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of detection in neural networks. In International Conference on Learning Representations, 2018.
image
[26] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. arXiv preprint arXiv: 1 170.02269, 2017.
[27] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. arXiv preprint 10X4v:990400366 2019.
[28] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by invert- ing them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 51 3188-5196, 2015.
[29] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adver- sarial autoencoders. arXiv preprint 15Xiv:1511.05664 2015.
[30] Jan Hendrík Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint
2017.
[31] Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv:
2019.
[32] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[33] Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625-632, 2005.
[34] Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In NeurIPS, 2018.
[35] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019.

---

[36] George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. arXiv preprint orXiv:/705.07057. 2017.
[37] Guim Peramau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355 2016.
[38] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
[39] Robin Rombach, Patrick Esser, and Bjorn Ommer. Network-to-netwonl translation with conditional invertible neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[40] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with in-distribution examples and gram matrices. arXiv preprint
2019.
[41] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
[42] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference Arlificial Intelligence and Statistics, pages 2164 2173. PMLR, 2019.
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[44] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint 190.v0190..05600 2019.
[45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020.
[46] llya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014.
[47] Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. arXiv preprint arXiv:2010.07432, 2020.
[48] Sean Tao. Deep neural network ensembles. In International Conference on Machine Learning, Optimization, and Data Science, pages 1-12. Springer, 2019.
[49] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint /7Xiv:/704.01155 155, 2017.
[50] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Ieml, volume 1, pages 609-611 616. Citeseer, 2001.
[51] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.
[52] Ev Zisselman and Aviv Tamar. Deep residual flow for of distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13994 14003, 2020.
12


Reviews:
[{"review_id": "Kenfkjf3YEm", "reviewer": "Reviewer_3B1Z", "summary": "This paper explores jointly training generative model probes alongside a neural architecture to imbue the model with capabilities analogous to \"debugging\" in traditional software. By training the generative model probes to reproduce the model inputs, every intermediate activation can be decoded to input space (images). The authors use this (1) to interpret the partial progress of the model made at intermediate layers, (2) to restrict the model from using sensitive information from the inputs, and (3) as an interface to compose the model with itself and with other pretrained image models. The authors further use self-composition to produce an ensemble from a tree of connected classifiers, which in turn gives uncertainty measurements that enable (4) out-of-distribution detection and (5) calibration. Experimental evidence backs each of these contributions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I outline here the main strengths of the paper. The core idea that the paper proposes -- training generative model probes jointly with an image classification -- is novel and conceptually simple, but with powerful implications. This technique allows for several applications, some of which resemble traditional software debugging capabilities (interpreting the partial progress of the model is like a step-through debugger; restricting the model's use of sensitive information is analogous to an assert statement; composing two models or the model with itself using the image domain as the interface is like traditional function composition), and some of which are desirable properties of neural models (out-of-distribution detection, calibrated predictions).\n\nNeural networks usually lack these capabilities. That a single conceptually straightforward addition to neural networks allows for all of these capabilities concurrently is a significant result. That it does this without undue degradation in performance of the original underlying model further improves the significance. The paper does a good job of covering the several benefits the method confers and defending each one with experimental evidence. These benefits span interpretability, fairness, flexibility through model composition, and uncertainty measurement (OOD-example detection and calibration).\n\nThis is a comprehensive paper presenting a novel type of optimized neural probe, showcasing the method and several capabilities that it enables in image neural networks. It leverages generative image models as neural probes. Of course neither generative models nor neural network probes for interpretability individually are new, but I am not aware of generative models being used as neural network probes previously. To my knowledge, the use of generative models in this way is original. I recommend investigating the \"linear probes\" line of work, not discussed in your (otherwise overall quite comprehensive) discussion of the literature. [1]\n\nAs with the \"observer effect\" in physics, the probes that you consider alter the behavior of the underlying network. This is because they are optimized in conjunction with the main model. Fortunately, the paper finds that performance is not degraded by optimizing the probes jointly with the rest of the network. Future work could explore the degree to which the solutions found by the original network change when optimized in conjunction with these generative probes. (Note, unlike with the observer effect, a model trained with the probes can be used for inference with the probes removed, without changing the network's output.)\n\nA key drawback of the approach is that there is a significant increase in compute for training and inference from the addition of the generative model to the underlying model, and accordingly there is a decrease in training speed. The paper suggests otherwise at line 122, and I think it is incorrect to do so; I recommend rewording. The paper would benefit from quantifying this drawback, or at least addressing it in the text.\n\nAt line 96 the paper describes sampling from $g_\\theta(h)$ as approximating sampling from $S(h)$. I don't think this approximates sampling from $S(h)$, though, since the training data doesn't include any members of $S(h)$ other than $x$. We never once see another input that produces $h$ besides the one that originally produced $h$. Further, I don't think defining $S(h)$ was necessary for your argument.\n\nOn the fairness application at line 275, you state that \"a high performing classifier trained ... without abusing the protected attribute.\" Alternatively the model could have constructed adversarial examples such that the pretrained classifier m cannot predict the protected feature, despite the protected feature still being present and still being used to make predictions. An investigation into whether this alternative behavior occurs would benefit the paper or future work. One way to approach this investigation would be to retrain the sensitive attribute classifier to see if the sensitive attribute can still be detected, rather than assuming the pretrained classifier is still a good measure of whether the sensitive attribute is being considered.\n\nThe main text of the paper does not mention the details of the generative model. This information is contained in the Appendix. I encourage the authors to include at minimum a reference to this Appendix in the body of the paper.\n\n---\n\n[1] Understanding intermediate layers using linear classifier probes https://arxiv.org/abs/1610.01644\n\n---\n\nNit at line 71: an non-linearity -> a non-linearity\n\nNit: activations have same type as encoded training examples, not the training examples themselves.\n\nTypo at line 283: discrepancy is spelled wrong\n\nNit: No entry is highlighted in OOD column for AudioMNIST.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors consider both technical limitations and potential societal impact of their work in Section 9.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "8"}, "extra_text": {"ethical_concerns": "The paper considers the task of predicting attractiveness in the CelebA dataset while ignoring skin color, using a pretrained model on CelebA that predicts pale skin. I understand that this is a sensitive topic, and accordingly have selected Yes for sending this paper for an ethics-focused review."}}, {"review_id": "FniH6NO_hOi", "reviewer": "Reviewer_gKMX", "summary": "This paper argues for inverting intermediate latent representations of convolutional classifiers -- predicting the original input -- as a tool for improving robustness and interpretability. The ability of the model to learn invertible intermediate hidden representations acts as a regularizer during training.\n\nThis basic premise is used in several ways. For example, a recursive version of this idea (using the reconstructed input to classify, and the intermediate hiddens of that to re-reconstruct the input, adding that reconstruction loss into the overall loss calculation) leads to an interesting form of drop out: choose which reconstructions (from which layer, and which recursive depth) to use in the final output. That can be used for enhanced forms of Monte Carlo Dropout, improving uncertainty computations over regular MC Dropout.\n\nAnother use is in placing constraints on intermediate hidden layers. For a fairness goal, such a constraint might be that, given a good classifier for some protected attribute, that classifier has poor performance on the reconstructed inputs from the original model's hidden layers. In a sense, this constrains the training of the model to lose the information that would allow the external classifier to predict the protected feature.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall Assessment\n\nOriginality: Although autoencoders, invertible networks, and interpretability tools are not new, I found the formulation here interesting and elegant. I would have appreciated some statement on the contrast to vanilla autoencoders here. I see some differences, but I'd like to see those in the paper.\n\nQuality: I appreciated the evaluation. It seemed sound. I questioned whether some of the evaluated aspects (e.g., OOD detection) are really a contribution of this work, or a secondary artifact of the improved uncertainty estimation.\n\nClarity: The write-up is excellent.\n\nSignificance: The results seem of academic importance for now, given the lack of a cost-benefit analysis of the approach. However, the idea of getting uncertainty out of the main computation, without further training past the original training seems useful in its own merit, perhaps analogously to Monte Carlo Dropout. I find the novelty also pushes the significance up in my eyes.\n\n\n# Feedback\n\nI enjoyed reading your work. It's an interesting use of autoencoders that I hadn't seen before. I found the combination of the basic reconstruction-loss term in more intricate formulations fun to read, and a great demonstration of creativity.\n\nMost of my feedback is on the positioning of this work. I found the introduction (and even the title) a bit of an oversell. Obviously, this work isn't solving the problem of neural network interpretability, or enabling arbitrary composition with other neural networks. Your example mentioned \"segmentation and classification\" which is more about functional compositionality (exciting!), but you only talk in the paper about self-composition, and regularization with a pre-trained model that does the same function. That's why I found that story a bit oversold. Your work does interesting things (using reconstruction from partial hiddens to augment and regularize, which is great!) but after your abstract and intro, I was expecting to see the Great Software 2.0 debugger, and this it is not.\n\nThe Self Composition section was exciting to read. It seems you get a big boost in uncertainty estimation using the \"random walks through the reconstruction layers\" trick, which is awesome. However, given the great cost of this self-composition (not to mention the cost of the decoder) it would be very useful to show in Table 1 what the relative costs in terms of computation time and model weights go along with the DecNN's and the various baselines. I imagine your approaches still win over Ensembles on both counts, but it would be useful to see the details to contextualize the cost of your idea.\n\nThe *Out-of-Distribution Detection* section was informative, but I was having trouble distilling how much of its results are due to the improved uncertainty estimation of RecDecNN and how much of it due to other reasons inherent in the DecNN architecture. Unless I misunderstand the experiments in this section, any black-box uncertainty estimator that gave improved results, would also improve OOD detection, regardless of the presence of input reconstruction, or the decoder sampling via recursion paths. In other words, although I buy this as an outcome of your work improving uncertainty estimation, I see it as a secondary benefit that's not a primary contribution of your work. Please clarify if I've misunderstood your contribution here. The same applies to Calibration.\n\nI have a somewhat similar concern about section 5.2 and your composition with pre-trained models. It seems that this composability argument is, again, rather superficial. Perhaps a way to argue that DecNNs are better suited to this would be to show that if you move the $m$ model into $f$, you get inferior results. Perhaps another way to ask this is how much would standard models have benefited from the pre-trained regularizer (if your new term in equation 7 didn't involve $g_\\phi$, but just $(f_\\theta(x) | m(x)))? Is there something inherent to DecNNs that makes this regularization better?\n\nOne question about your fairness section is how the experiments shown in Table 6 affect primary performance (overall accuracy in Table1). I'd love to see that in a future version of this work, although I find the results here very welcome.\n\n\n# Smaller comments\n\n* Lines 87--92. This is a nit really, but I find the notation for $h$ confusing, given that there's lots of $h$'s we're dealing with in most of the paper. I think it would be much cleaner to just change the expression $x \\in S(h)$, into something like $x \\in S(f_\\theta(x))$? It changes nothing in your point here, and it doesn't add more cognitive muddiness.\n\n* Line 256 is confusing. It sounds like you're incorporating a linear model to a residual network. Please rephrase.\n\n* You use both ((Re)DecDNN and (Re)DecNN) in the paper. Please use one (I recommend the latter).\n\n* Your Table 1 has no caption. Oops.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Bg3inCf8ga4", "reviewer": "Reviewer_9qhL", "summary": "Rather than only building the forward model P(y|X), this work propose to also learn a generative \"inverse\" model P(X|h) for any hidden layer h. This ability mirrors traditional software debugging, where upon a failure computation, one can \"pull back\" from the point of failure and detect at which point is the program behaving erroneously. This paper explores how to cleverly leverage this trained \"sampled pullback\" for various applications such as probing, uncertainty estimation, calibration, and fairness.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# significance\nI believe this work is significant. It is using a very simple idea, that of training an inverse sampler in conjunction with the forward NN. This work use the inverse in a clever way: realizing that with the inverse one can arbitrarily \"compose\" different neural networks (of same type signature) by pulling back the first NN to a sample of prototypes in the input space then passing it to the second NN. This clever insight enabled a range of applications, from building an ensemble for uncertainty estimation and calibration, to information-flow style privacy.\n\n# originality\nI cannot comment on this as I am not an active practitioner in explainable neural networks. However I have not seen approaches like this from my experience, so if anything this approach (using inverse as a mean of composition) is novel to me. I will defer this to other reviewers who are more fluent in the field.\n\n# training method\nthis work is simple technically, train a conditional-vae-ish inverse sampler, and section 2 and 3 read very clear.\n\n# applications\nthis paper claims that the simple training of inverse would allow application of probing, composition, and constraints. \nbelow are some detailed comments\n\n## probing\nthis section is clear, by looking at the inverted X from different layer, a user can reason about where in the network did something important such as \"wearing hat\" got lost. \nI feel some results of probing can be relegated to the appendix, as probing is a mere consequence of training the inverse, and it is not very novel / exciting compared to composition. probing can be in fact a sub-section of section 3, entirely written in 4 sentences and the reader won't miss it.\n\n## composition\noverall this is the coolest, and most insightful part of the paper (section 5), it should have a higher prominence. It is currently buried very deep.\n\n\"But given two image classifiers, it is not clear how to do so\" - why do I even want to do this? it'll be good to have a concrete use-case to motivate this upfront.\ngiven f1 and f2, what is the interpretation of the \"semantics\" of f2(f1_inv(f1(x)) exactly? If the inverse is bijective, the semantics is simply f2(x), but f1 loses information along the way, so what is the intuition here, and can you write about it?\nthe prose \"\"\"in other words ... joint objective\"\"\" reads a bit dense, I was able to understand it after a few reads, but it would benefit from a figure, as it is central to understanding this work.\n\nThe application of using self-composition to get an ensemble is good, and the explanation is very clear (I especially enjoyed the part on sampling from a space of O(L^D), very cool).\n\nI think parts of section 5 is the actual main thesis of this paper, instead of this \"building software 1.5\" story which feels out of place. \n\nto me, this paper is asking a fairly philosophical question: \"What does it mean to compose two arbitrary functions which has no business of being composed, by sampling the inverse of the first one, f2(f1_inv(f1(x))?\" The answer to this (abstract) question then opened the door for a host of insights on how to build neural network compositions that solve a range of tasks. it is unfortunate that, such an interesting insight is buried so deep in section 5, and the out of place \"building software 1.5\" would have prominence over this in the introduction.\n\nanyone working on explainable NN can claim they are working on \"software 1.5\", but not anyone has the unique insight of composing functions by leveraging a sampled inverse, and using it to build an ensemble.\n\n## constraining\nthis section is clear, the training objective function is very clever.\nthe analogy to assertions is not good, more details in clarity section.\n\n## other modalities\nsure, this is obvious.\n\n# clarity\nthis is where I'd like to see some major improvements\n\n### figure 1 looks like a hastily cropped figure 3 \nthis figure should carry way more information, as this is the first thing the reader will draw attention to, yet it is the worst figure of them all.\ncurrently, it is 1)vague caption, 2)has no annotation, 3)and the axis are not labeled, and 4)tells an incomplete story\n1) what is misclassification mean? only after reading the whole paper did I realize that, the bottom row is an input that should be a \"6\" that is being missclassified as a \"4\".\n2) the first 3 rows need caption as well, was the mistake \"hair - nohair\" for the first row? was it \"man - woman\" for the second row? was it \"bag - dress\" for the third row? \n3) it is also unclear that the horizontal axis represents how \"deep\" into the network the latent vector / activation was when used to invert (figure 3 does a better job). \n4) why are composition and constraints not highlighted equally here, why is only \"probing\" being highlighted? you have the phrase \"improving compositionally\" in the title, presumably this is the important story to tell, and should be reflected in figure1 somehow.\n\nincidentally, the \"probing\" capability is the _least_ interesting, most obvious part of this paper, at least to me.\n\nI am inclined to raise my score if the authors can provide an anonymous url to an updated figure_1.png that clearly explains what decNN is on a high level, how does it allow for arbitrary NN composition, and its applications (essentially, sets up the main technical contribution and \"promise\" of this work). maybe consult a colleague who is proficient at effective visual communication.\n\n### the analogy to software 1.0 feels strained\n\nFor instance, \n\"\"\"In traditional software, assertion statements are a popular way to make sure the program is executing\nas expected. Ideally, we would have the same functionality in neural networks: the ability to specify\nwhat information an activation should not capture. \"\"\"\nthis analogy doesn't quite work, as \"executing as expected\" =/= \"specify what information is kosher\"\na better analogy is 'information flow\", take a look : https://www.cs.cornell.edu/andru/papers/jsac/sm-jsac03.pdf\n\nIn general the analogies to software 1.0 feel strained (here, and in the introduction). My (personal) opinion is that the story of composition you have in this paper is already good enough, there really is no point trying to shoe-horn a story with software1.0, 2.0, 1.5 at all, which \"get in the way\" of the main thesis of this paper, which is composition of two functions of the same input type and the consequence of this composition.\n\n### minor phrasings\n\"to expect “useless” information to be dropped deeper into the network\" could change the word \"drop\" to \"discard\" as drop can also mean \"to place\" (drop me off at the zoo) which is confusing.\n\n# overall \noverall this is a solid paper with a simple approach that is easily applicable (at least in theory, provided that the inverse sampler is easy to train) to a range of important problems. the experimental results look solid to me as well, I especially liked the section on using self composition as a way of building an ensemble, which in turn can be used for uncertainty estimation and calibration. \n\nit is being held back by poor organization and story telling (perhaps a lack of re-writes on the paper) and a negligence of effective visual communications, which hampers its readability in its current form. this is unfortunate, because this work is very inspiring to me, both on a conceptual level (composing two arbitrary functions that takes the same input type using inverse) and on a practical level (the preliminary results looked promising). Practically, I think this work would benefit by moving some contents into the appendix to free up more space for better story telling.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "it is sufficient", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision has been reviewed and the paper is officially accepted.

----

After extensive discussions among SACs, ACs, and the program chairs, we have decided to conditionally accept this paper. The primary concerns were around over-claiming, the interpretation of results, and the analogy to programming (which was viewed as a distraction from the main contributions of the paper).  There was some debate around whether these issues constitute a "fatal flaw" that should prevent the paper from being published at NeurIPS, but all agreed that these issues need to be addressed. We hope that the authors will take into account feedback from the reviewers and make the changes that were promised as part of the rebuttal.

Additionally, in order to be accepted, the following changes must be made in the revision:

Introduction:

L52: "This joint training optimizes the classifier not only to be highly performant, but constraints its activations to a manifold that the generative model is able to decode. Consequently, the networks entire computation is constrained to a 'decodable' vector space... " This is misleading and should be rephrased or removed; just because the activations at a given layer can decode the input doesn't mean that that information is being used by the downstream layers to actually make decisions.

L59: This paragraph should focus on what the paper actually demonstrates empirically; essentially remove the first sentence. 

Section 3: 

Highlight that equation (4) doesn't require that the reconstructed images $g_\phi(h_l)$ actually produce the same output y, so the reconstruction may not actually be highlighting the features that were important for f to make its decision in the first place.

Section 4: 

This section should be significantly rewritten to avoid making claims unsupported by the evidence. Specifically:

L116: "In practice, this theory leads us to expect 'useless' information..." This would be true if Equation (4) wasn't rewarding the network for keeping the useless information to help support the decoding task. 

L119: This paragraph is misleading and should be removed. While it is true that you can visualize the information at each layer, you cannot actually tell which of that information is actually being relied upon to make the decision. 

L130: This is the only paragraph in this section that states an empirical fact; the whole section should focus more on what this experiment objectively shows, but avoid making claims not supported by the observations made in this experiment.

L138: This paragraph again misrepresents what can really be inferred from the generated images. 

Section 6. 

The analogy with assertions is misleading as it implies a level of guarantees that is not supported by the evidence. The section should focus more closely on what the experiments actually demonstrate for this application. The attractiveness task should also be replaced with the "bald vs not bald", and "bearded vs not bearded" experiment presented in the rebuttal. 

Abstract: 

The abstract should be consistent with all the changes made to the rest of the paper. 

----

The original meta-review for this copy of the paper follows:

I am really torn about this paper. On the one hand, I think this paper presents an intriguing and original idea: train a generative model to reconstruct the input from the intermediate layers of the network, and then feed those reconstructed inputs either to the same network or to other networks that perform related tasks, possibly even recursively. The paper then presents empirical evidence that these compositions can be engineered to have some useful properties, such as identifying when their input is out of distribution, or avoiding the use of certain types of information in a classification. 

On the other hand, the paper as written suffers from extreme over-claiming and interprets many of its results in a way that I don't think is warranted by the data. The gap between the clear contributions of the paper and the narrative that the authors built around them was pointed out by multiple reviewers. While the authors did significant work through the rebuttal period to address many of the reviewer concerns and made some significant promises in how they were going to rewrite the paper. I think the issues are significant enough that I really think a fresh version of this paper that incorporates all the proposed changes needs to be reviewed by a fresh panel. The issues with his paper are too big to accept it without further review. 

I now describe some of the issues that I found most questionable about this paper. 

The paper is organized around an analogy to traditional programming and program debugging. The claim is that the generative models allow us to interpret what each of the layers in the network mean so that they can be debugged probed, asserted, etc. However, the analogy doesn't really stand up to scrutiny, and in arguing for this interpretation of its results, the paper fails to explore alternative explanations for the observed behaviors. 

For example, consider section 4. The objective empirical observation is that for this set of images and image classification tasks, when an image is misclassified, the reconstructed images progressively look more and more like the incorrect label. This is interesting, but the authors try to claim something much bigger; they claim that these images can allow one to understand "what features a neural network is attending to in making a prediction, or to understand what features make it susceptible to making an error". I find this highly unlikely. In fact, the loss function in Equation 4 almost guarantees that this would not be the case. For example, suppose that in the case of the handbag, the neural network actually learns that a handful of pixels making up the handle of a handbag are what characterizes it as a handbag. An image that allows one to understand what features a neural network is attending would show just the handle of the handbag and discard everything else if that's what the network is using for its classification. However, equation 4 rewards the network for maintaining enough information at each layer to reconstruct the whole handbag, even if most of that information is not actually being used to make the prediction, only for the reconstruction. In fact, the handle of the handbag has very few pixels, so the reconstruction network would not be significantly penalized if it failed to display the handle; there is nothing in equation 4 that actually requires the reconstruction to focus on the features that are important for classification. 

There are similar issues with each of the subsequent sections. They are motivated and explained in terms of this analogy to programming, but the analogy just doesn't hold up to scrutiny. I think section 6 is particularly problematic because the analogy with assertions implies a level of guarantees that is simply not supported by the evidence. Now, I actually think this section is the most interesting one in the paper, and I think it is unfortunate that the authors chose such a bad example to make their point. However, I think a lot more evidence is needed to show that this approach can be used to reliably prevent the network from using a protected characteristic when making a decision (that would be a great paper by itself). For example, Equation 8 forces the generative model to optimize three criteria: faithfulness to the input image, randomness with respect to the protected characteristic, and ability to recreate the correct classification output. How these competing goals are balanced out will depend on some hyperparameter tuning that may determine the extent to which the resulting network actually satisfies the desired constraint. For example, a protected characteristic that affects a lot of pixels in the image will increase the cost of adding diversity relative to the goal of being faithful to the input image and may require different hyperparameters from one that only occupies a few pixels. 

Overall I think there is an important and novel contribution in this paper, and I do believe the authors understand some of the issues with the paper as submitted and have demonstrated a willingness to rewrite the paper based on the criticisms in the reviews. This is reflected in the high scores given to the paper by all the reviewers. However, multiple reviewers raised the same concern highlighted above, and the Area Chair feels that this concern is quite substantial and requires a major revision to address. 
Decision: Accept (Poster)