Improving Compositionality of Neural Networks by Decoding Representations to Inputs
Authors: Mike Wu
Noah Goodman
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative models
decoder
representations
out-of-distribution
compositionality

Abstract: In traditional software programs, it is easy to trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together. Although deep learning programs have demonstrated strong performance on novel applications, they sacrifice many of the functionalities of traditional software programs. With this as motivation, we take a modest first step towards improving deep learning programs by jointly training a generative model to constrain neural network activations to "decode" back to inputs. We call this design a Decodable Neural Network, or DecNN. Doing so enables a form of compositionality in neural networks, where one can recursively compose DecNN with itself to create an ensemble-like model with uncertainty. In our experiments, we demonstrate applications of this uncertainty to out-of-distribution detection, adversarial example detection, and calibration --- while matching standard neural networks in accuracy. We further explore this compositionality by combining DecNN with pretrained models, where we show promising results that neural networks can be regularized from using protected features.

Improving Compositionality of Neural Networks by Decoding Representations to Inputs
Mike Wu, Noah Goodman, Stefano Ermon
Department of Computer Science
Stanford University
Stanford, CA 94303
{wumike, ngoodman, ermon Astanfordd edu
Abstract
In traditional software programs, it easy trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together. Although deep learning programs have demonstrated strong performance on novel applications, they sacrifice many of the functionalities of traditional software programs. With this as motivation, we take modest first step towards improving deep learning programs by jointly training generative model to constrain neural network activations "decode" back inputs. We call this design Decodable Neural Network, or DecNN. Doing so enables form of compositionality in neural networks, where one can recursively compose DecNN with itself to create an ensemble-like model with uncertainty. In our experiments, we demonstrate applications of this uncertainty to out-of-distribution detection, adversarial example detection, and calibration - while matching standard neural networks accuracy. We further explore this compositionality by combining DecNN with pretrained models, where show promising results that neural networks can be regularized from using protected features.
Introduction
Traditional hand-written computer programs are comprised of a computational graph typed variables with associated semantic meaning. This structure enables practitioners interact with programs in powerful ways (even if they are not the author) such as debug code by tracing variables back to inputs, apply assertions block errors, and compose programs together for more complex functionality. However, traditional software has its limitations: it difficult to hand-write programs classify images extract sentiment from natural language. For these functionalities, deep learning and neural networks
[1 have become the dominant approach 23 2 46
While neural networks have made impressive progress complex tasks, they come at sacrifice of many of the desirable properties of traditional software. Specifically, the closest approximation to "variable" in neural network is activation. Yet it is difficult to understand neural network's computation from an activation value, and there little to associate an activation with semantic meaning, A practitioner cannot write assertion statements constrain valid neural network logic: checking the values that an activation takes is usually not enough to gauge correctness nor meaning. Moreover, given multiple neural networks, composing them together requires retraining from scratch.
In this paper, we take modest first step towards bridging the expressivity of deep learning with the engineering practicality of traditional software. We aim to uncover ways for practitioners to build and use neural networks by leveraging compositionality. Specifically, we propose to train a neural network classifier jointly with generative model whose role is to map the classifier's activations back to the input, approximating invertibility. For any input, a neural network's 's computation can
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

be represented by sequence of activations (from each layer), each of which can now be mapped back to the input space. It is this insight that enables special form of compositionality for neural networks, as inputs derived from activations can be fed back into other models.
In our experiments, study this compositionality by (1) recursively composing neural network with itself, creating an ensemble-like model with measure of uncertainty that is useful for out-of- distribution detection, adversarial example detection, and model calibration; (2) composing neural networks with pretrained models as forms of regularization and distillation; and (3) using decodable activations to discourage neural network from using protected attributes. Throughout, we show that decodability comes at low cost as we find equivalent accuracy to a standard neural network.
2
Background
Neural Networks We will focus on supervised neural networks for classification, although is simple to extend to regression tasks. We denote a neural network by fa where 0 represents its trainable parameters. A neural network Sa maps an example æ to predicted probabilities over K' classes. Typically, neural network is composed of L blocks. For example, if So is multi-layer perception, each block linear layer followed by a non-linearity. If fe is a ResNet [17), each layer is residual block. In computing its prediction, neural network fe produces activations (/1....,hy. where hy is the output of the (-th block.
To train a neural network, we solve an optimization function of the form
(an(a; 0) = log pog((yl(((x)) + (x;;;;;
using stochastic gradient descent. In Equation [ the notation denotes an input example and its label. The function 52 represents an auxiliary objective such as a regularizer. The hyperparameter 0 is used to scale the auxiliary loss. For classification, log p(v) -) is cross entropy.
Generative Models Broadly, we are given latent variable and an observed variable X. generative model Is captures the distribution p(z) h), mapping samples of the latent code h to samples
Bdder byart of the nec natuert, fo
hat
Uncertatee
Facanire Shanng
no hat
Hisfer
corl-cetvoy lnss
has hat
Decantsaction
has hat
Tris deoode Icveetrr at iiput
Sharna
los
Trae libe
fu
no hat
Douddd octvetuus souh roco stuuction inturee ard
Coetiaanss roceeS2
(a) Decodable Neural Networks
(b) Ensembling through self- Composition
Thas hat
has fast
Conotran rita, be deso unifer S.A. protected Nemure
has hair
has hat
Pratesined clasthen Ing hair cortancingg
Prstrateed Steas deccdded sctivssin trpect
Unfoon
(c) Composition with Pretrained Classifiers
(d) Constraints with Protected Features
Figure 1: Decodable Neural Nenworks and Applications: The decodable neural network (DecNN) is composed of main network. f and generative model, Each activation in ʃ is decoded back to inputs through Subfigure (a) shows an incorrect classification: the network predicted the man not wearing a hat despite true label. Consequently. we see the decoded activations "interpolate" as the hat is removed, Subfigures (b-d) show the compositionality DecNNs. First, (b) shows self-composition, where decoded activation can fed input back into the network f create ensemble with uncertainty. Second, (c) shows how a pretrained classifier, m can composed with DecNNs for regularization. Finally, (d) shows how to constrain DecNNs to ignore "protected features" through pretrained networks by regularizing for uniformity.

---

of the observed variable r. For a deep generative model, Is is parameterized by neural network. In the simplest case where Ie is an autoencoder, we are also given an encoder q that maps observed samples to latent codes h. Then, we optimize the objective
(a) Φ) = log peppp(1gg ((=)))(((()))
(2)
where for continuous v, the expression log p(x(4) is the log Gaussian probability with mean l and fixed variance (or equivalently, mean squared error). More complex generative models might impose a prior distribution on h [22), an adversarial discriminator [29 [1]. or invertibility of g [35]
3 Neural Networks with Decodable Activations
In Section [ motivated our desiderata for activations that can be mapped back to the input space. We now propose to do this by "inverting" a neural network at a particular activation. Concretely, given an input 2; neural network fo. can compute activation h by computing the forward pass (x)). Then, suppose we could construct the set:
Shh = {2: ~ PD. fo(ă)= = h}
(3)
where Pp represents the empirical distribution that x sampled from. Now, any input à E S(h) is a valid mapping of h back to the input space. (Trivially, activation h. Unfortunately, in practice we do not know how to construct such a set S(h).
S(h).) We call this ê decoding of the
But, we do know how to (approximately) sample from S(h). We can train a generative model Jo to reconstruct the original input x from neural network activations (recall can sample r ~ (h), which we know how to do, to sample from S(h). If we jointly train this generative model Io with the neural network fa. we can optimize for activations that both solve the classification task and decode back to inputs once.
S(h)). Then, we
So far, we have only considered single activation despite our neural network having T. such activations. Although one could train a separate generative model per layer, we found it efficient and performant to share weights ø across all layers. Our joint objective is:
C( 0, の) = log log (f +
log pogg(z)go)
fi1..., hy come from fa(
(4)
where the first term is cross entropy as in Equation and the second term is a reconstruction error (Equation 2) averaged over layers. Comparing Equation 4 to Equation D can interpret the generative loss as a regularization term Ω. Intuitively, in a standard neural network, the activations hi,..., hr are unconstrained, chosen only minimize the loss. However, in Equation 4 the network fa is constrained to produce activations that the generative model Io can invert. In practice, is Re decoder with upsampling layers (see Appendix A.I. We call this setup Decodable Neural Network, or DecNN. Figure T depicts an illustration of the DecNN architecture.
We highlight that although Equation 4 requires good reconstruction, it does not enforce that recon- structed inputs f (g)( must map to the same label y as fzz). We show a few examples of this by visualizing decoded activations throughout network in Appendix A.3
4 Composing Decodable Neural Networks
In traditional software, we can easily compose two compatible functions Si and f2 together by (x)) for some input x. With this as loose inspiration, we wish to similarly compose neural networks together. As motivation, we may want to do this to faciliate interaction between networks, such as for regularization or distillation. But given two image classifiers, it is not clear how to tie outputs from one neural network to inputs for the other? Both networks expect images as input.
The workaround comes from decodable activations. Since they are images of the same shape as inputs, we are free to treat them as inputs for another model. So, we can compose neural networks together using decoded activations as an intermediary: given two models fi and fz, and an input x, we can take an activation h coming from (x), decode it, then provide to the other model, f2(go(h)). In the following subsections, we explore two instances of this compositionality: one by composing a model with itself (i.e., fi f2), and the other by composing a model fi with a pretrained one, f2. See Figure Ib.c for graphical depiction of both compositions.
3

---

4.1 Recursive Self-Composition
Our first composition will be to compose a DecNN model with itself. But if we can do that, we can also recursively compose DecNN model with itself infinitely:
Given a decoded activation go (h( (0)) for any layer I, we can feed it as input back into the classifier,
(0)) )) and maximize agreement with the true label y. Intuitively, if go (hy) is a good reconstruc- tion of the input x (i.e. (h()) and belong to the same set S(h)),, then good classifier fo should
(o(gs(h)
predict the correct label with it as input. Now, we also observe that the computation fo (1x (0)) produces I. additional activations, 11 ,..., .i (1) where we use the superscript to represent the number of compositions. We can repeat this exact process with the same intuition, decoding ht and passing it to fa to produce h Computational limits aside, this can be repeated ad infinitum.
One interpretation of this procedure is as data augmentation: we saw in Section A.3 that decoded activations share features relevant to the prediction task (e.g. generated celebrities wear hats) but vary other irrelevant features (e.g. face shape. hair color, etc.). Notably. these generated images not in the original dataset. Thus, training a model using decoded activations as inputs is similar to augmenting the initial dataset. The only difference is that gradients are propagated back through recursion to the inputs, which is also a form of regularization.
Admittedly, what we have described so far is intractable. Besides recursing infinitely, each activation produces L more activations, creating tree (with root node x) with large branching factor L. In practice, we make two adjustments - First, we limit the recursion to max depth D. Second, instead of constructing the full graph of activations, which would have OLLP) nodes, we randomly sample
"paths" in the graph. Specifically, starting the root, we compute {11 ..... / through fo(x)) and (0)
randomly choose an integer 4 E [1, L]. Then, we compute (h ..... hL through (o (h(2)) ie. we classify the lo-th decoded activation from depth O. Again, we randomly pick 4 to compute ................................ [h!")...,,h, { through (g (h,,)) and repeat until depth D.
Building on Equation 4
we can write this "recursive" objective
L(:; 0, Φ) = (ogp(y)f((x))
Σ ad logg(yl (1
+33 Σ a log (11111)))) (5) d=1
where the sequence of integers lo.... 1D1 are each sampled uniformly from [1. L]. We introduce a new hyperparameter a E [0, 1] that geometrically downweights later depths (if 0. Equation reduces to maximum likelihood). The first term in Equation the usual classification objective. The second term sum of cross entropy terms that encourage the decoded activation depth d to predict the correct label through fe. The third and final term is a sum of reconstruction losses that encourage the decoded activation at depth d to approximate the original input x.
We call this model a "Recursively Decodable Neural Network", abbreviated ReDecNN. Note that ReDecNN is a special case of an ensemble network. That is, every path sampled in the activation tree by picking 4o.... I builds legitimate classifier. There are ((T,P) such classifiers implicit in the ReDecNN, We can interpret optimizing Equation (5 aa training individual classifiers that are Monte Carlo sampled from the tree every gradient step. See Figure Ip for an illustration.
Our goal in the next few paragraphs is to utilize this ensemble to measure uncertainty.
Method
MNIST
Fashion
ColebA
Method
MNIST
Fashion
CelehA
Standard
97.1 (0.11)
87.7 (0.19)
90.8 (et..)
ReDecNN
0.869
0.795
0.692
DeeNN
97.8 (0.17)
88.8
90.8 (c0.1)
MC-Dropon
0.878
0.818
0.657
ReDocNN
97.5 (0.20)
88. 1 (0.16)
(0.8
BayesNN 0.537
0.574
0.587
Dropout
95.9 (0.11)
80.7 (0.34)
Ensemble 0.531
0.559
0.605
MC-Dropour
91.33
78.6 (0.23)
87.8 (0.19)
BayesNN
96.3 (0.51)
87. 0. (0.36)
88.2 (0.22)
(b) ROC-AUC of separating correctly classified
Ensemble
97.6 (0.11)
88.4 (0.12)
90.9
and misclassified examples using Eg. a score.
(a) Classification performance on test set.
The stdev. for all entries over 3 runs are <0.01.
Measuring Uncertainty As a first step, we compare ReDecNN to several baselines: a standard neural network, neural network with dropout, two kinds of Bayesian neural networks - Monte

---

Carlo dropout [TI]. abbreviated MC-Dropout, and weight uncertainty 4 abbreviated BayesNN and naive ensemble network where we train D copies of the standard neural network with different initializations. For all experiments, see Appendix A and A.2 for training and task details. Table la shows accuracies over a held-out test set, averaged over three runs. While we observe lower accuracy from Bayesian NNs, we observe (perhaps surprisingly) equal performance of DecNN and ReDecNN to a standard neural network. Typically, adding uncertainty to neural networks comes at a cost to performance (as is true for other baselines), but this seems to not be the case here.
The next step to compare the quality of model uncertainty. For the ReDecNN, for given input æ, we measure uncertainty as follows: sample N different classifiers from the activation tree, and make predictions with each classifier on z. The uncertainty is the entropy of the predictions:
Uncertainty (1) --) Σ = c)g pog c))
c
(6)
where ply = c) is the empirical probability (e.g. normalized count) of predicting class out of N classifiers. A higher uncertainty metric would represent more disagreement between classifiers. We can compute the same metric for naive ensemble, as well as for MC-Dropout (where repeated calls drop different nodes) and BayesNN (by sampling weights from the learned posterior).
We want to test if the model's uncertainty is usefud. One way is to correlate the uncertainty with when the model makes prediction errors (we call this "misclassificationn)) It would be useful if the model was less uncertain on correct predictions and more uncertain on incorrect ones. A practitioner could then use uncertainty to anticipate when a model might make a mistake. Table Ib reports the area under the ROC curve, or ROC-AUC of separating correctly and incorrectly classified examples in the test set (using uncertainty as a score). A higher ROC-AUC, closer to I, represents a better separation. We find that ReDecNN is competitive with MC-Dropout, even out performing it on CelebA. In Section A. we include examples of images with low and high uncertainty.
Distribbbiinn Detection A second application of uncertainty is detecting -uttof-distribution (OOD) inputs. Given an input that is far from the training distribution, model is likely to make a mistake. Instead, the model should recognize its uncertainty, and refuse to make a prediction. There a rich literature on detecting OOD inputs using downstream computation on trained model 18 25
Out-of-
24 52
19). Unlike those works, we study OOD detection using uncertainty alone.
MNIST
MosSS MC
CelebA MC
ReDecNN
Ensemble
ReDecNN
Ensemble
ReDeeNN
Ensemble
Advensaiial
0.787
0.817
0.540
0.711
0.760
OOD (MNIST)
0.793
0.864
0.501
0. 0.847
0.548
0.591
ODD ()ashionMNISS)
0.812
0.888
0.334
0.641
0.572
0.399
OOD (ColehA)
0.893
0.943
0.675
0.753
0.793
0.704
Comput (Mean)
0.727
0.678
0.691
0. 0.886
0.569
0.632
Corrupt (Sidev)
0.075
0.113
0.087
0.072
0.108
0.106
0.03d
0.017
0.016
Table 2: ROC-AUC of predicting which examples are out-of-distriuttion (OOD). We vary OOD examples to be adversarial, corrupted, or from a different dataset. Standard deviation for all entries are < 0.01 over three runs.
We explore three kinds of OOD examples: (1) adversarial examples crafted for a single classifier using FGSM (15), (2) examples from different dataset (e.g. train on MNIST and use FaSH as OOD) as done in (25,, and (3) corrupted examples by 14 image transformations (e.g. adding pixel noise), borrowed from BT. See Appendix Jppendix/A.4ffor expanded corruption results.
Table reports the ROC-AUC of separating inlier examples taken from the test split of the dataset the models were trained on, and outlier examples. From Table 2 we observe ReDecNN is just under MC-Dropout, but while MC-Dropout achieves these results at the cost of classification accuracy, ReDecNN does not (Table Tap. Furthermore, we find ReDecNN generalizes better to CelebA, more complex image dataset, where it outperforms MC-Dropout (and other baselines).
Focusing on CelebA, we study a domain OOD challenge by holding out all images with positive annotations for a single attribute as nut-of-distribution. Here, inlier and outlier examples are very similar in distribution. We report ROC-AUCs in Tar three held-out attributes (randomly chosen): is the celebrity in the image "wearing a hat", has "blonde hair", or "bald"? As this is more challenging problem (since inlier and outlier examples are from the same dataset versus different But while MC-Dropout, BayesNN, and a naive ensemble all perform near chance, ReDecNN is consistently near 0.6.
datasets), the performance lower than in Table 2
5

---

OOD: Wearing a Hat
OOD: Bland Hair
OOD: Bald
Method
ROC-AUC
Method
ROC-AUC
Method
ROC-AUC
ReDecDNN
0.604 (41.01)
ReDecDNN
0.593 (c0.01)
ReDecDNN
0.615 (4h.11)
MC-Dropout
0.519 (ct01)
MC. Drapout
0.502 (c001)
MC: Droppot
0.502 (et0l)
BayesNN
0.510 (001)
BayesNN
0.508 (0.01)
BayesNN
0.508 (c001)
Ensemble
0.526 (c0.1)
Ensemble
(c0.1)
Ensemble
0.503 (c001)
Table 3: We hold out a group of CelebA attributes, such as those wearing a hat, when training. We compute the ROC-AUC of labeling the held-out group as OOD (mean and stdev. over runs).
Calibration A third application of uncertainty we explore is calibration of predicted probabilities. Standard neural networks are notoriously over-confident and are incentived in training to predict with extreme probabilities (e.g. 0.99). A model with useful uncertainty would make predictions with proper probabilities. Although many calibration algorithms exist [33 [6), we want to measure how calibrated each model is out-of-the-box to compare the quality of model uncertainties.
Table 4a reports the expected calibration error, ECE (32), which approximates the difference in expectation between confidence and accuracy using discrete bins. A lower number (closer to 0) represents a more calibrated model. We observe that while standard neural network has a relatively high ECE, many of the approaches (dropout, MC-Dropout, BayesNN, and ReDecNN) reduce ECE. On the other hand, naive ensembles increase the calibration error, as they are more prone to overfitting. In two of three datasets, ReDecNN achieves the lowest ECE whereas MC- MC-Dropout achieves the lowest in the third. These results should be viewed in parallel to Table Ia which measures "sharpness". Otherwise, we can trivially reduce ECE to zero by ignoring the input.
Method
MNIST
Fashion
CelebA
Depth
Acc.
OOD (Fashion)
0OD (ColehA)
Standard
1.18 (0.10)
0.56 (0.14)
2.75 (0.12)
ReDecNN
0.33 (0.05)
0.21 (0.0))
(0.13)
97.7
0.665
0.752
Dropout
0.63 (0.00)
0.48 (0.09
2.51 (0:13)
4
97.6
0.788
0.804
MC MC-Dropaul
0.41 (0.06)
0.13 (0.03)
2.99 (0.15)
97.7
0,805
0.868
BayesNN
0.91
0.18 (0.0)
2.41
97.5
0.812
0.893
Ensemble
1,80 (0.12)
0.89 (0.07)
3.12 (0.12)
(b) Effect of recursion depth D on classification accuracy and OOD detection for a ReDecNN model (8-layer MI.P) trained on MNIST.
(a) We compare the expected calibration error (ECE) of ReDecNN to standard neural network with var- ious regularization and ensembling.
Finally, we study the effect of depth on the efficacy of ReDecNN. For the experiments above, we chose the recursive depth to be equal to the number of layers the MLP (D = T, = 8). In Table 4b) we vary the depth from 2 to 8 (while keeping the number of layers fixed to and study the effect on classification accuracy and OOD detection. We find increasing OOD performance as D increases, although with marginally decreasing gains. Moreover, we observe constant accuracy, matching standard neural networks regardless of the choice of D.
Effeet of Depth
4.2 Composing with Pretrained Models
Apart from self-com celf-oomposiiinn, we can also compose our neural networks with off the--hh- pretrained models as a form of regularization or distillation. Suppose we have a pretrained model m that maps an input to class prediction. We can modify the ReDecNN objective as follows:
6(x;0,0) Φ) -(x;0, + Σ ad (o (((-))))
d=1
where Ć is as defined in Equation The additional term acts as a divergence, bringing the classifer's predictions close to those of the pretrained model. A similar edit can be made the DecNN. We revisit Section experiments now using two pretrained models - a linear classifier or a ResNet-18 classifier. For each, optimize Equation 77 and compute performance
OOD detection.
Table 5 compares the results composing with either (1) a linear model, (2) a residual network, or (3) no composition (None). Using linear model, we observe slight drop in test accuracy but an increase across OOD experiments and calibration. fact, these new results rival surpass
MC-Dropou from Table 2 Conversely, with ResNet-18, observe point increase in test

---

MNIST
CelebA Linear
Method
None
Linear
ResNet
Nome
ResNet
Accuracy
97.50 (0.20)
97.17 (0.12)
98.30 (0.24)
90. (d0.1)
89.76 (0.11)
91.28 (0.13)
Misclassfication
0.869 (chol)
0.904 (c0.1))
0.742
0.692 (ch.01)
0. 0.732 (c001)
0.615 (c0.01)
Adversorial
0.787 (c0.0))
0.805 (0.00)
0.729 (c0.01)
(MNIST)
0.647 (c001)
0.693 (4001)
0.609 (c0.01)
OOD (Fashion)
0.812 (cb.0l)
0.845 (<0.01)
0.650
0.641 (c0.01)
0.673 (c001)
0.585 (c0.01)
OOD (CelebA)
0.893 (c0.01)
0.928 (c0.01)
0.802 (c0.01)
Corruption (Meun)
0.727 (c0.01)
0.790 (x0.01)
0.694 (cb,01)
0.686 (c0.01)
0. 708 (<0.0)
0.616 (<0.01)
Calibration (ECE)
0.334 (0005)
0.296 (0.0))
0.385 (0.02)
1.927 (0.13)
1.937 (0:04)
3.002 (0.10)
Table 5: OOD detection using uncertainty for ReDecNN composed with pretrained models.
accuracy but notable drops in OOD and calibration results. These polarizing outcomes show two use cases of pretrained models: using simpler models allow for regularization that trades off accuracy for robustness while more complex models encourage distillation, making the opposing tradeoff.
An open question how much Equation 7 ben-
efits from regularizing fe (h)) to be close to mk) versus regularizing it to be close to m(x). The former is a pretrained embedding of recon- struction whereas the latter of a static input. Ta- ble compaare the two on MNIST, where we call the latter "direct regularization". We observe that Equation ( outperforms direct regularization (other than calibration), often by a large margin. We rea- this to be because the input and the recon- struction (h) can be very different, especially in later layers shown in Appendix [A.3] This im- plies that mxxx and m(g(h) can be very different as well. Intuitively, constraining the model be similar what a pretrained model thinks of the reconstruction (which dynamic value that varies with model weights) is a stronger learning signal than what the pretrained model thinks of the original input (which is static). In other words, regularizing fe ((()) to be like m(z) at all layers / is too strict of a constraint.
Method
Direct Reg.
Accurecy
97.17
96.72
Misclass.
0,904
0.828
Adversarial
0.805
0.707
0OD (Fashion)
0.841
0.699
0OD (CelebA)
0,928
0.782
Corruption (Mean)
0.790
0.667
Calibration (ECE)
0.296
0.280
Table Composition (Eq.
versus direct regular-
ization with pretrained models on MNIST.
5 Constraining Decodable Computation
A milestone for deep learning would be the ability to specify what information an activation should not capture. This is especially important in cases when we have protected attributes that should not objective. While this milestone remains out of grasp, we take small step towards it by optimizing a neural network to "be indifferent" with respect to a protected attribute through composition with pretrained models.
be abused in optimizing
Suppose we have access a pretrained classifier m that predicts the assignment for K-way protected feature. Then, building on Section 4.2] we can optimize our classifier to "ignore" information about the protected feature through m. To do this, we define the objective:
[(x; = ((z;0,0) La logp(m(g
(M (M 1k)
(8)
K
dml
where 1x is K-dimensional vector of ones. That is, Equation 8 encourages the decoded activation go (oo("("))) (2-1)) when passed through the protected classifier m, to predict chance probabilities i.e. have no information about the protected attribute. high performing classifier trained in this manner must have solved the prediction task without abusing the protected attribute(s).
To test this, we extract two attributes from CelebA: "beardedness" and "baldness". Suppose we wish to design a neural network to predict beardedness. If we were to naively do so, there could be discrepancy performance between groups of individuals for example, between bald individuals and individuals with hair, the latter group is more common in the CelebA dataset. Indeed, Table shows a point difference Fl and a point difference in average precision (AP) in classifying beardedness between the two groups. The second row of Table provides bascline ReDecNN
7

---

Model
Fl (Bald Not Bald)
AP (Bald Not Bald)
ECE (Bald Not Bald)
Standard
0.338 0.338/0.440 /0,438 (40.11 <0.0:
19.4/23.9 (0.080009)
4.76/3.50 /3.56 (0.0/0.00)
ReDecNN (Eq.))
0.327 /0.44 (cha1
18.6/22.3 (0.11/0.44)
4.74/3.54 (008
ReDecNN (Eq.
0.461/0.501 1001 (001)
27.2/28.9 (0.10/0.10)
3.80/3.28 3.203 (0.12/0.09)
Table 7: We pretrained model on CelebA that predicts baldness optimize the activations of second classifier to ignore baldness when predicting beardedness.
optimized without knowledge of the protected attribute. Unsurprisingly, we find similar discrepancy between groups, as with standard neural network. In the third row, we evaluate ReDecNN that was optimized to ignore the "baldness" attribute using pretrained classifier for baldness. Critically, we find much more balanced (and higher) F1, AP, and ECE across two groups.
One unsatisfying but plausible explanation for Table 7 is that the generative model learned construct adversarial examples such that the pretrained model m cannot predict the protected feature, but that the protected feature still being used to make predictions by f. To show this is not the case, we take a trained ReDecNN optimized by Equation g freeze its parameters, and small linear head on top of the Jast hidden layer to predict the protected feature. If the model learned ignore this feature, it should not be able to perform the task well. Table reports a poor FI score (0.1) 1) whereas doing the same with standard neural network or with unconstrained ReDecNN gets 3 times the FI score.
Methad
F1
Standard
0.383
ReDecNN (Eq.
5
0.371
ReDecNN (Eq. 8
0.108
Table 8:
Quantitatively accessing
ability to classify baldness.
6 Generalizing to other Modalities
While we have focused image classification, the proposed approach is more general and can be extended to other modalities. We apply the same decodable representations to speech, in particular utterance and action recognition. We compare DecNN and ReDecNN to the same baselines, measuring accuracy and uncertainty through similar experiments as we did for images.
Method
Acc
Misclass.
O0D
BCE
Methol
Acc:
Misclass.
OOD
ECE
Standard
(0.1)
Standard
(0.4)
DecNN
93.8 (0.2)
DecNN
41.4 (0.7)
ReDecNN
93.4 (0:2)
0.766
0.705
0.225
(0.1) ReDecNN
41.2 (0.5)
0.642
0.605
0.523 (0.2)
MC. Dropout
82.1
0.788
0.745
0.429
MC Dropout
34.5
0.629
0.562
0.515
BayesNN
91.6 (0:7)
0.545
0.518
1.039 (0:4)
BayesNN
40.3
0.523
0.500
0.918 (0.1)
Ensemble
96.3
0.529
0.506
(0.2) Ensemble
44.1 (0.1)
0.541
0.522
1.189
(a) AudioMNIST
(b) Fluent Speech Commands
Table 9: Performance on speech classification. If not specified, stdev. is <0.01 over three test runs.
We utilize the AudioMNIST 3 and Fluent speech com- mands 27 datasets, the former being a corpus of 60k utterances of the digits 0 through 9, and the latter corpus of 100k recordings 97 speakers interacting with smart- home appliances for one of six actions. Audio waveforms are preprocessed to log Mel spectrograms, outputting 32 by 32 matrix [47]. Figure shows the input on the left- most column along with the 2nd, 4th, 6th, and 8th decoded activations for random test examples. Table reports the findings, where like the image experiments, we find ReDecNN has strong performance on OOD detection and calibration return for only a small drop in performance compared to standard neural network.
Figure 2: Decoding activations to im- ages for Fluent spectrograms.

---

7 Related Work
Although DecNNs are reminiscent of autoencoders, there are differences: while autoencoders only reconstruct the final hidden layer, DecNN has a reconstruction for every single hidden layer. This, and because DecNNs have a supervised objective (which autoencoders do not have) critically changes what information is captured in the hidden layers. Whereas the autoencoder seeks perfect reconstruction, the DecNN does not, as evidenced by loss of information for reconstructions later in the network (see Appendix A.3) Finally, we point out that the applications of autoencoders are very different from the DecNN. The former is not used for calibration or composition.
Autoencoders
Invertible and Ensemble Networks The DecNN is akin to invertible generative models 6 as DecNN is training generative model to "invert" the classifier up to hidden layer. However, unlike invertible flows, DecNN does not impose an explicit prior on the activations (i.e. latent variables), and further, DecNN has supervised objective. Our proposed recursive network, ReDecNN, has similarities to Bayesian neural networks [I 4 and ensemble networks 48 two baselines we compare against for evaluating model uncertainty. However, we find NeDecNN to be easier to train than doing inference over weights, and cheaper in parameter count than naive ensembles.
Probing Neural Networks A related line of work seeks to probe neural network post-training to understand the underlying computation []. Most relevant to our work is an approach that inverts
CNN activations through reconstruction (28). Unlike this approach, we are not proposing any additional computation post-hoc. Rather, we are interested optimizing neural networks such that their representations are more easily mapped back to the input domain.
Robustness and Protected Attributes
Out-of-distribution detection
26
24
52
40
20
selective classification 12
13
adversarial perturbation detection 49
T0
30
34
and neural
network calibration [6
51
50
32
38 each have a rich subfield of algorithms. While task-specific
methods likely outperform our results, we are excited about representations that enable uncertainty without task information or additional algorithms. Our approach also shares similarities where the authors minimize the mutual information between learned representations and a protected attribute. We view Equation as an approximation of this where we treat the pretrained classifier for the protected attribute as proxy for the mutual information.
8 Limitations and Future Work
To close, we discuss few limitations. First, approach is bottlenecked by the quality of the gen- erative model. Without good reconstruction, op- timization will be intractable. However, in light recent work [45] 44 5 21 43), this is becoming less of a problem as new generative models surface. Second, in the main text, we only explored feedfor- ward classifiers for simplicitly. Our approach ex- tends naturally residual and transformer blocks, and future research could explore this direction. As a start, we provide experiments for extensions to CNN architectures in Appendix A.6 Third, opti- mizing recursive networks, although parameter efficient, costs more compute as backpropagation more expensive. See Table 1O for timings of epoch in seconds. The proposed method does have significant computational overhead, averaging about 6x cost, while baseline methods like Dropout and MC-Dropout impose little to overhead. On the other hand, ReDecNN has comparable cost to DecNN despite requiring recursive gradients. Also, both DecNN and ReDecNN are both cheaper than naive ensembles, significantly in the CelebA case. Future work could explore weight sharing across layers (not just depth) to reduce compute.
Method
MNIST
Fashion
ColobA
Standard
13.6 (0.3)
13.5 (0.)
82.6 (6.3)
DecNN
80.5 (6.8)
99.2 (96)
486.2 (129)
ReDecNN
96.7
(6.6
(21.6
Dropout
13.9
(4.0
(9.5
MC-Dropout
10.9
14.0
79.5 (5.7)
Ensemble
10..6
132.6 (8.5)
659.4 (82)
Table 10: Cost (seconds) of 1 epoch on a Titan X GPU (averaged over 10 epochs).
In summary, we explored building neural networks with decodable representations, and leveraged this new property to compose models together. By re-purposing decoded activations as novel inputs, we were able to join neural networks together in useful ways for distriibitii detection and calibration, as well as influence what information a network retains in optimization. We are optimistic about promising results and look to future research for more applications of decodability.

---

Acknowledgments and Disclosure of Funding
We thank the Ermon group for their comments and suggestions. We thank the reviewers for their many iterations of feedback that helped this paper improve significantly. This research was supported in part by NSF (#1651565, #1522054, #1733686), ONR (N000141912145) ONR MURI (N000141612007 AFOSR (FA955019 ARO (W91 INF-21-1-0125), and Sloan Fellowship. MW is supported by the Stanford Interdisciplinary Graduate Fellowship as the Karr Family Fellow.
References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprini
2014.
[3] Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, 0tts/1807.03415 2018.
[4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[5] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233 2021.
[6] Laurent Dinh, Jascha Sohl- Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605v 2016.
[7] Piero Esposito. Blitz - bayesia layers in torch zoo (a bayesian deep learing library for torch). https://sg http://ggtttttt..ee/eee httpb///////ppppttoo///tttt--- http://tobbbttt-bbbabia-e-deeeee -ayessann-de---erniing 2020.
[8]
et al. Falcon, WA.
Pytorch lightning. 3, 2019.
GirHub.
Note:
[9] William Falcon and Kyunghyun Cho. A framework for contrastive self-supervised learning and designing new approach. arXiv preprint arXiv:2009.00104 2020.
[10] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint
000 2017.
[11] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[12] Yonatan Geifman and Ran EI- Fa-iv..v. Selective classification for deep neural networks. arXiv preprint arXiv: 1705.0850000 2017.
[13] Yonatan Geifman and Ran El- Yaniv.... Selectivenet: A deep neural network with an integrated reject option. International Conference on Machine Learning. pages 2151-2159, PMLR. 2019.
[14] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprins arXiv:1406.2660 2014.
[15] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv: 1412,6441226572 2014.
[16] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pages 1321-1330. PMLR, 2017.
10

---

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. arXiv preprint 1610.0: 36, 2016.
[19] Dan Hendryeks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: ww../......00.. 2.0666, 2018.
[20] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances Neural Information Processing Systems, pages 15663-15674, 2019.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[23] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1 1105, 2012.
[24] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[25] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of detection in neural networks. In International Conference on Learning Representations, 2018.
image
[26] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. arXiv preprint arXiv: 1 170.02269, 2017.
[27] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. arXiv preprint 10X4v:990400366 2019.
[28] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by invert- ing them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 51 3188-5196, 2015.
[29] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adver- sarial autoencoders. arXiv preprint 15Xiv:1511.05664 2015.
[30] Jan Hendrík Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint
2017.
[31] Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv:
2019.
[32] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[33] Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625-632, 2005.
[34] Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In NeurIPS, 2018.
[35] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019.

---

[36] George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. arXiv preprint arXiv:/705.070577 2017.
[37] Guim Peramau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355 2016.
[38] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
[39] Robin Rombach, Patrick Esser, and Bjorn Ommer. Network-to-netwonl translation with conditional invertible neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[40] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with in-distribution examples and gram matrices. arXiv preprint
2019.
[41] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
[42] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference Arlificial Intelligence and Statistics, pages 2164 2173. PMLR, 2019.
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[44] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint 190.v0190..05600 2019.
[45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020.
[46] llya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014.
[47] Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. arXiv preprint arXiv:2010.07432, 2020.
[48] Sean Tao. Deep neural network ensembles. In International Conference on Machine Learning, Optimization, and Data Science, pages 1-12. Springer, 2019.
[49] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint /7Xiv:/704.01155 155, 2017.
[50] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Ieml, volume 1, pages 609-611 616. Citeseer, 2001.
[51] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.
[52] Ev Zisselman and Aviv Tamar. Deep residual flow for of distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13994 14003, 2020.
12


Reviews:
[{"review_id": "jtXpeL4RUDc", "reviewer": "Reviewer_v4fL", "summary": "The paper proposes decodable neural networks (NN) where the idea is to add an extra term in the objective that focuses on reconstructing the input from the hidden activations thereby allowing for probing the NN, composing them ( along with a pretrained model), and constraining certain decodable computations. They show experiments on simple image datasets ( MNIST, FashionMNIST, CelebA) and highlight the usefulness of their formulation for out-of-distribution detection, adversarial robustness, calibration, and protection attribute prediction.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The idea of reconstructing the input from hidden activations seems old and used extensively in autoencoders. I didn't quite get the analogy to software 1.5. What does it mean to obtain the decodable representation of the same \"type\" as input? I can see that the space of representations can be the same but there is nothing in the objective that forces the regenerated input to be the same \"type\" as the original input, especially if you take the software analogy where types have specific meanings ( like keywords, identifiers). I felt that the analogy to software was unnecessary because the main idea doesn't bear any connection or even takes motivation from the software domain. I felt the invocation of software terms like *probing our “variables” (i.e. activations) for meaning, constraining model logic with “assertions”, and composing neural network modules together* (Line 60-61) was just a fancy way of motivating the work and somehow showing that they are doing something different from previous work. \n\n**Technique**: I appreciate that the authors have mentioned this in limitations that their technique has a high computational cost. I would some analysis of the same though. I have some questions about some of the choices that the authors made in the formulation. For example, in Equation (4), what is the need for reconstructing the input from *every* hidden layer and then taking a mean of that and adding to the objective? Essentially, we are directing the NN here to decode the input from every hidden layer in the NN, which places an additional burden on the NN. Imagine a big NN with multiple layers ( which will be used for slightly complex datasets like Imagenet), it will be extremely hard computationally to perform training. I understand that this gives the ability to probe every layer in the NN but at a huge computational cost thereby limiting the scalability of the approach to bigger models (especially if the recursive decodable NN in Equation 5 is used - which is the model that performs better in experiments).\n\n**Experiments and Results**: I like the use-cases where the paper has shown results. To me, Section 5.2 and Section 6 were quite interesting. In Section 5.2, though I fail to understand why using a ResNet-18 pretrained classifier was worse than both without using any pretrained model or using a linear classifier? Also, in Table 3, how were the attributes for OOD ( wearing a hat, long hair, bald ) chosen. Since in Table 2, MC is better than ReDecNN except in CelebA, choosing only CelebA for results in Table 3 seems unfair to other baselines. Similar question for attributes listed in Table 6. Were these results cherry-picked to show the superiority of ReDecNN over other techniques? As mentioned previously, I would also like to see an analysis of computational complexity of all the methods and the tradeoff with performance and interpretability. Anouther major concern I have related to the results is that the authors have chosen very simple datasets (MNIST, Fashion, CelebA). I feel like to evaluate the effectiveness of the proposed formulation, experiments on ImageNet or other datasets ( with higher image resolution and more complex) are needed.\n\n**Clarity**: The paper is clear and well written. I felt that there are too many results though that may divert the attention from the main claims (like Section 7) that could have been placed in the Appendix.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have done a fair job of listing the limitations and potential negative societal impact of their work. One suggestion in limitation can be to add that they have shown results on very simple datasets and model (only fully connected nets). So their technique might not be generalizable to other complicated datasets and models. The authors mention that extending to other models will be straightforward, but I think it is not fairly straightforward especially at the cost of severely increasing the computational complexity.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "UXXsODus3PW", "reviewer": "Reviewer_csH8", "summary": "This work proposes DecNN, and the main goal is to allow people to interpret the intermediate neural network activations. The base model is an 8-layer MLP. The core idea is to train another generative model that takes the MLP activation as the input, and the generative model aims to decode an image in the model input space. During training, the loss of the generative model aims to reconstruct the input image given the activations of all intermediate layers. They further design ReDecNN, which recursively composes DecNN by taking the output of the generative model as the input of DecNN. They demonstrate that the generative model produces interestingly interpretable outputs to explain why the classifier makes correct or wrong predictions. They also show some promising results for out-of-distribution detection, calibration, and fairness.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Model interpretability is an important topic, and the difficulty of debugging is indeed one weakness comparing neural networks to traditional programs. Therefore, in general this work is well-motivated, and it demonstrates some pretty interesting results on interpreting neural network activations.\n\nHowever, my main concern of this work is that simply applying this framework to an MLP model does not seem convincing. Due to the simplicity of the model architecture, the experiments are only conducted on relatively simple classification tasks, and the baseline performance is not great. What are the main difficulties of applying this framework to convolutional neural networks and/or recurrent neural networks? I wonder whether DecNN/ReDecNN still performs well when applied to more complicated neural network architectures.\n\nRelated to the fact that the implementation is based on a simplified model architecture, the authors did not empirically compare with recent OOD detection and calibration works, e.g., [20] [22] [23] [24]. Therefore, it is hard to tell whether ReDecNN indeed achieves a better trade-off between the classification accuracy and the OOD/calibration performance. The results in Table 5 show that even if the model simply utilizes a better model to improve its own performance, the OOD/calibration performance decreases notably, and in some cases the performance becomes worse than most or all baselines. This kind of indicates that the framework might not fit models that are able to better classification accuracies.\n\nThe results show that in general ReDecNN has lower classification accuracy than DecNN. I suppose the recursion depth D could be an important hyper-parameter for the performance. Do you have any ablation study that evaluates the performance of different D? Does increasing D generally improve the OOD detection/calibration results but decrease the classification accuracy?\n\nWhile the discussion in Section 6 about fairness is interesting, I think the training objective is not directly related to the ReDecNN architecture. As we have seen, ReDecNN itself does not improve fairness. Have you tried applying the new term in Equation (8) to standard neural networks? I suppose you can simply achieve similar fairness results in this way.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "9auBxGSxC-", "reviewer": "Reviewer_f5Sy", "summary": "This paper proposes to jointly train a generative model that maps activations back to the input space. The work demonstrates several benefits of this approach.\n\n* Probing: It provides a way to decode the activations which enables the users to make sense out of it since the decoded activations are in the input space. \n* Composing networks: The decoded activations allow to compose a DecNN model with another model with the same input type. \n* Data Augmentation: The DecNN model can be composed with itself. This can be viewed as data augmentation since the decoded activations are of the same \"type\" as the input and retain the features that are relevant to the prediction task.\n* Ensemble network. The DecNN model can be recursively composed with itself to get the “Recursively Decodable Neural Network” (ReDecNN) model. This model can be viewed as an ensemble network. The paper shows the usefulness of ReDecNN for the tasks of measuring uncertainty, OOD detection, and calibration.\n* Constrain the activations: The decodable activations makes it possible to specify the kind of information an activation is not supposed to capture. This provides assertion-like capability to the neural network.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe idea of invertible generative models has been explored in earlier works (invertible flows). However, this work is in a supervised setting where the activations are mapped to input space by training a generative model jointly with the classifier. Also in this work, no prior is imposed on the activations. Instead of relying on post-hoc computation, the proposed approach constrains the activations to be invertible. \n\nThe proposed recursive network ReDecNN presents a natural way for defining an ensemble network by randomly selecting activations (from the activation tree) to be fed to the network. This is especially appealing as the decoded activations retain only the task specific information.\n\n**Quality**\n* The paper is technically sound and the claims are supported by appropriate experiments\n* The analogy with composition in traditional software seems a bit far fetched. The proposed composition by using the decoded activation as conduit is very different from the conventional composition operation. For example, consider two hypothetical image to image networks f_flip and f_darken.   Now f_darken(f_flip(img)) will flip and darken the image. However, with the proposed approach f_darken(g_phi(h)) will only darken the image since the decoded activation will invert the flip. Admittedly, f_darken(f_flip(img)) is possible only when the input and output are of the same type. Comparing this conventional composition is confusing. Considering that “Compositionality” appears in the title, authors may want to clarify this aspect.\n* The authors have gone to great lengths to demonstrate the usefulness of the recursive self-composition. All the claims are well supported by appropriate experiments. To demonstrate that the methods are applicable to other domains, authors have also presented results on the utterance and action recognition tasks. \n\n**Clarity**\n\n* The paper is well written and easy to read. The experiment results are presented well.\n* Section 5. Line 143 talks about f1(f2(x)) but line 149 gives an example for f2(g_phi(h)). It will be better if the outer function is the same at both the places. It will also make sense to make it explicit that g_phi is a generative model trained along with f1 (and not f2). \n* Apart from the computational cost, are there other reasons for parameter sharing across layers?\n* Authors may want to emphasize the role of probing in error analysis.\n\n**Significance**\n\n* The idea of decoding the activations is novel. Since the decoded activations are in the input space, users can make sense out of them. The distinguishing feature of the approach is that no post-training computation is required to get the decoded activations.\n* In addition to providing a regularizing effect, the decoded activations can also be used for data augmentation. This is especially useful since the decoded activations retain only the features that are important for prediction. \n* The functionality of “constraining decodable computation” has a potential for significant impact. One can specify the protected information in input space and the method will take care that the protected information is not abused for solving the task.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It remains to be seen if the regularization effect of constraining the activations to the decodable space prevents it from learning complex tasks.\n", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "7PyiZF3zQwd", "reviewer": "Reviewer_7fvR", "summary": "This paper presents an approach to simultaneously train a prediction network along with a generative model to map the neural network activations back to inputs. The use cases of doing so include (1) interpreting the information encoded in an activation, (2) composing the network with itself and other pre-trained networks, which in turn provides a means to get uncertainties and to regularize the network, and (3) to assert that the network satisfies some constraints on what it can and cannot learn in the intermediate layers. The paper is evaluated on several image benchmarks and speech benchmarks and showcases all the above benefits. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, proposes a very new technique for training and analyzing the neural networks, and performs extensive evaluation with promising results. I thoroughly enjoyed the paper and definitely recommend it for acceptance. \n\nThe idea of learning a generative model to map the activations back to the inputs in-tandem is unique. In fact, I was expecting the performance to degrade, but surprisingly the performance is as good as a standard network. \n\nHowever, it is not clear to me how to evaluate the quality of the inversions (it can turn out that the generative modeling didn’t converge). Some of the images in figure 3 are promising and at the same time, some of them raise suspicions. For eg 3(b), in many of the intermediate images, it looks like the person has hair (instead of being bald). Can the authors comment on this?\n\nAdditionally, since ReDecNN performance is similar to DecNN, the expected “data augmentation” (line 162) is not really happening. \n\nNevertheless, the indirect evaluation through the uncertainties, OOD detection, and calibration is interesting. \n\nAlso, what is the rationale for using a deterministic generator model? What would happen with a non-deterministic generator model?\n\nOverall, the technique presented in this paper is novel and promising and is definitely interesting to the community. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision has been reviewed and the paper is officially accepted.

----

After extensive discussions among SACs, ACs, and the program chairs, we have decided to conditionally accept this paper. The primary concerns were around over-claiming, the interpretation of results, and the analogy to programming (which was viewed as a distraction from the main contributions of the paper).  There was some debate around whether these issues constitute a "fatal flaw" that should prevent the paper from being published at NeurIPS, but all agreed that these issues need to be addressed. We hope that the authors will take into account feedback from the reviewers and make the changes that were promised as part of the rebuttal.

Additionally, in order to be accepted, the following changes must be made in the revision:

Introduction:

L52: "This joint training optimizes the classifier not only to be highly performant, but constraints its activations to a manifold that the generative model is able to decode. Consequently, the networks entire computation is constrained to a 'decodable' vector space... " This is misleading and should be rephrased or removed; just because the activations at a given layer can decode the input doesn't mean that that information is being used by the downstream layers to actually make decisions.

L59: This paragraph should focus on what the paper actually demonstrates empirically; essentially remove the first sentence. 

Section 3: 

Highlight that equation (4) doesn't require that the reconstructed images $g_\phi(h_l)$ actually produce the same output y, so the reconstruction may not actually be highlighting the features that were important for f to make its decision in the first place.

Section 4: 

This section should be significantly rewritten to avoid making claims unsupported by the evidence. Specifically:

L116: "In practice, this theory leads us to expect 'useless' information..." This would be true if Equation (4) wasn't rewarding the network for keeping the useless information to help support the decoding task. 

L119: This paragraph is misleading and should be removed. While it is true that you can visualize the information at each layer, you cannot actually tell which of that information is actually being relied upon to make the decision. 

L130: This is the only paragraph in this section that states an empirical fact; the whole section should focus more on what this experiment objectively shows, but avoid making claims not supported by the observations made in this experiment.

L138: This paragraph again misrepresents what can really be inferred from the generated images. 

Section 6. 

The analogy with assertions is misleading as it implies a level of guarantees that is not supported by the evidence. The section should focus more closely on what the experiments actually demonstrate for this application. The attractiveness task should also be replaced with the "bald vs not bald", and "bearded vs not bearded" experiment presented in the rebuttal. 

Abstract: 

The abstract should be consistent with all the changes made to the rest of the paper. 

----

The reviewers found the idea of jointly training a generative model to additionally map neural network activations back to inputs quite interesting and novel. During discussions, a major concern about the approach that came up was regarding the computational cost of the approach and thereby its scalability for training more complex architectures and tasks. The additional results with CNN layers was helpful, but scalability still remains a challenge. However, there are enough interesting novel ideas and applications presented in the paper that could potentially open up many interesting future directions. It would be great if authors can incorporate the detailed feedback and clarifications from reviews and additional experiments in the final version of the paper.
Decision: Accept (Poster)