Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraint
Authors: Borui Zhang
Wenzhao Zheng
Jie Zhou
Jiwen Lu
Venue: ICLR 2023 poster (2023)
Keywords: Neural network
explainable AI
optimizer.

Abstract: Deep learning has revolutionized human society, yet the black-box nature of deep neural networks hinders further application to reliability-demanded industries. In the attempt to unpack them, many works observe or impact internal variables to improve the comprehensibility and invertibility of the black-box models. However, existing methods rely on intuitive assumptions and lack mathematical guarantees. To bridge this gap, we introduce Bort, an optimizer for improving model explainability with boundedness and orthogonality constraints on model parameters, derived from the sufficient conditions of model comprehensibility and invertibility. We perform reconstruction and backtracking on the model representations optimized by Bort and observe a clear improvement in model explainability. Based on Bort, we are able to synthesize explainable adversarial samples without additional parameters and training. Surprisingly, we find Bort constantly improves the classification accuracy of various architectures including ResNet and DeiT on MNIST, CIFAR-10, and ImageNet. Code: https://github.com/zbr17/Bort.

Published as a conference paper at ICLR 2023
BoRT: TOWARDS EXPLAINABLE NEURAL NETWORKS WITH BOUNDED ORTHOGONAL CONSTRAINT
Borui Zhang, Wenzhao Zheng, , Jie Zhou Jiwen Lu
Department of Automation, Tsinghua University, China
Beijing National Research Center for Information Science and Technology. China [zhang-br25 zhengwz1 8) @ma i 1s. tsi wingguuaa...... weighaa.edu.coom edd...n cn; {jzhou, lujiwen]@tsinginnn atsighua..nn edu.
ABSTRACT
Deep learning has revolutionized human society, yet the black-box nature of deep neural networks hinders further application to reliablity-demandingg indus- tries. In the attempt to unpack them, many works observe impact internal variables to improve the comprehensibility and invertibility of the black-bos models. However, existing methods rely intuitive assumptions and lack math- ematical guarantees. To bridge this gap, we introduce Bort, an optimizer for improving model explainability with Boundedness and orthogonality constraints on model parameters, derived from the sufficient conditions of model compre- hensibility and invertibility. We perform reconstruction and backtracking on the model representations optimized by Bort and observe clear improvement in model explainability. Based on Bort, we are able to synthesize explainable ad- versarial samples without additional parameters and training. Surprisingly, we find Bort constantly improves the classification accuracy of various architec- tures including ResNet and DeiT on MNIST, CIFAR-10, and ImageNet. Code: https: / //gittttiia com/ 2br17/Borz
INTRODUCTION
The success of deep neural networks (DNNs) has promoted almost every artificial intelligence ap- plication. However, the black-box nature of DNNs hinders humans from understanding how they complete complex analyses. Explainable models are especially desired for -emmanddng industries such as autonomous driving and quantitative finance. Complicated as DÑNs are, they work as mapping functions to connect the input data space and the latent variable spaces (Lu et al., 2017: Zhou, 2020). Therefore, we consider explainability in both mapping directions. (Forward) Comprehensibility: the ability to generate intuitive understanding of how each module trans- forms the inputs into the latent variables. (Backward) Invertibility: the ability to inverse the latent variables to the original space. We deem model explainable if it possesses comprehensibility and invertibility simultaneously. We provide the formal descriptions of the two properties in Section 3.
Input
SGD
Bort
Top-1 Accuracy (%) on ImageNet
Saliency Reconstruction
21:13
17.00
77.63
$285
y5.22
25
76.25
21.12
59.75
4
Adamw
Adanity
Adamii Adamw
VBG16
Reskett0
DeIT-5
Swit-5
(b)
Figure I: Bort improves explainability and performance simultancously. (a) Examples of reconstruc- tion and saliency analysis. (b) Top-1 accuracy with various networks and optimizers on ImageNet.
Existing literature on explainability can be mainly categorized into black-box and white-box ap- proaches based whether involving internal variables. Black-box explanations focus on the exter- nal behavior of the original complex model without considering the latent states (Zhou et al., 2016;
"orrrsponning author.

---

Published as a conference paper at ICLR 2023
Without Boundedness
With Boundedness
Without Orthogonality
With Orthogonality
X 85
K 93
1 Smilor
Inner product
Inner product
Reconstruction
Reconstruction
(5.6)-2.50 40
(6.53 -M
X
(b)
Figure 2: Motivations of the two constraints. (a) Boundedness ensures semantic similarity consis- tent with dot products. (b) Orthogonality minimizes the reconstruction error for linear projection.
Lundberg & Lee, 2017; Fong & Vedaldi, 2017). For example, some methods employ simple proxy models (Ribeiro et al., 2016) to mimic the input/output behavior of the target model. They tend to produce an intuitive and coarse description of external behavior rather than an in-depth analysis of the internal mechanism of the model. In contrast, white-box explanations delve into the model to observe or intervene for more thorough understanding. However, existing white-bot box explana- tions lack a rigorous mathematical guarantee, as shown in Figure 2. For comprchensibility, most methods (Simonyan et al., 2014; Zhou al., 2016; Zhang al., 2018: 8; Liang al., 2020) intuitively assume that the activation of feature maps is associated with the similarity between the input data and the corresponding kernel, but they provide no theoretical guarantee of the assumed "relation". For invertibility, conventional backtracking methods (Zeiler & Fergus, 2014; Springenberg et al., 2015) usually employ a linear combination of kernels layer by layer for feature reconstruction. However, they ignore the potential entanglement between kernels and thus lead to suboptimal reconstruction.
We find that almost all explainability literature is based on specific assumptions, which may be ob- jectively incorrect or have no causal connection to the actual mechanism of the model. To bridge this gap, we give formal definitions of comprehensibility and invertibility and derive their suffi- cient conditions as boundedness and orthogonality, respectively. We further introduce an optimizer with Bounded orthogonal constraint, Bort, as an effective and efficient instantiation of our method. Extensive experiments demonstrate the effectiveness of Bort in both model explainability and per- formance shown in Figure 1. We highlight our contributions as follows:
Mathematical interpretation of explainability. We further derive boundedness and or- thogonality as the sufficient conditions of explainability for neural networks.
A plug-and-play optimizer, Bort, to improve explainability. Bort can be generally ap- plied any feedforward neural networks such as MLPs. CNNs, and ViTs.
Clear improvement of model explainability. In addition to better reconstruction and backtracking results, we can synthesize explainable adversarial examples without training.
Consistent improvement of classification accuracy. Bort improves the performance of various deep models including CNNs and ViTs on MNIST, CIFARIO, and ImageNet.
2 RELATED WORK
The properties of trained neural network are highly affected by the opti- mization problem. The basic SGD optimizer updates the parameters along the stochastic gradient di- rection. The subsequent optimizers such as RMSProp (Tieleman et al., 2012) and Adam (Kingma & Ba, 2014) accelerate convergence by computing the adaptíve gradients with the second momentum estimation and moving average. Other works focus on improving the generalization performance with a flat loss landscape (Foret et al., 2020). Additionally, the optimization constraints also affect model properties. The widely used LI or L2 regularizations filter out redundant parameters for bet- ter generalization. AdamW (Loshchilov & Hutter, 2017) separates weight decay from the training objective to achieve this. Recent attempts adopt disentanglement constraints (Zhang et al., 2018; Shen et al., 2021; Liang et al., 2020) to improve the model explainability by forcing each filter to represent a specific data pattern. Transformation invariance constraints (Wang & Wang, 2021) later emerge to improve explainability robustness. However, these methods usually suffer from the trade- off between performance and explainability and cannot be generalized to different architectures. To break through this dilemma, we propose Bort, an optimizer with bounded orthogonal constraints, which improves both the model performance and explainability.
Optimization Problem.
2

---

Published as a conference paper at ICLR 2023
Model Explainability. The desire to understand deep neural networks promotes the development of explainable approaches over the past decade. We primarily categorize them into black- box and white-box explanations based on whether they consider the internal neural states. Black-box ex- planations focus on the external behaviors of a model. Saliency-based methods assign importance scores pixels that most influence the model predictions using activation maps (Zhou al., 2016), gradient maps (Selvaraju et al., 2017; Chattopadhay et al., 2018; Smilkov et al., 2017; Sundarara- jan et al., 2017: Kapishnikov et al., 2019), or perturbation maps (Petsiuk et al., 2018). Proxy- based methods approximate the input/output correlation by a simple proxy model, such as linear model (Ribeiro et al., 2016), Shapley value (Lundberg & Lee, 2017), and probabilistic model (Fong & Vedaldi, 2017; Zintgraf et al., 2017). Despite their promising results, the black-box nature pre- vents them from further understanding the internal mechanism of the model. Therefore, we advocate white-box methods to provide an in-depth understanding of a deep network. However, we find that existing white-box methods are usually based on ungrounded assumptions. Backtracking meth- ods (Simonyan et al., 2014; Zeiler & Fergus, 2014; Springenberg et al., 2015) assume that each filter represents a pattern and can reconstruct input features by a weighted sum; decomposition meth- ods (Bach al., 2015; Shrikumar al., 2017) believe that overall features can be expanded linearly near the reference point: bybrid-modellbassed methods rely on the coupled transparent rules (e.g., de- cision tree (Wan et al., 2020), additive model (Agarwal et al., 2021), and entropy rule (Barbiero al., 2022)) to help understanding the internal mechanism; other methods expect disentanglement (Zhang al., 2018; Shen et al.. 2021; Liang et al., 2020; Chen et al., 2020) and invariance (Wang & Wang, 2021) constraints to regularize the parameters for better explainability. In addition, some meth- ods (Li et al., 2018; Chen al., 2019) try to condense the prototypes inside the model to reveal the learned concepts. We notice that only few works (Marconato et al., 2022) try to formulate a mathe- matical definition of explainability, so the relationship between these assumptions and explainability lacks theoretical guarantees. To bridge this gap, we seek to define explainability mathematically for FNNs and derive its sufficient conditions to optimize an explainable network.
METHOD
In this section, we introduce the motivation and derivation of Bort in detail. Section 3.1 formulates an explainability framework including comprehensibility and invertibility properties for neural net- works. Section 3.2 further derives set of sufficient conditions (i.e., boundedness and orthogonality constraints). Finally, Section 3.3 introduces the efficient optimizer Bort and discuss its properties.
3.1 EXPLAINABILITY FRAMEWORK
Even though numerous efforts have explored how to define explainability descriptively (Zhang & Zhu, 2018; Gilpin et al., 2018: Bodria et al., 2021), it remains clusive to provide the mathemati- cal definition due to its high association with the specific model type. Therefore, in this work, we concentrate on feedforward neural networks (FNN for short) and attempt to investigate the corre- sponding formal explainability definition. FNNs cover a large number of mainstream models, such as CNN (LeCun al., 1995) and ViT (Dosovitskiy al., 2020). We find that all these models can be unified under one meta-structure, a multi-layer perceptron (MLP for short) with optional nonparametric operations. For example, the convolutional layer and the transformer layer addition- operation, respectively. Therefore, we focus on the
ally use folding/unfolding and the explainability of MLP which can be naturally generalized.
For an l-layer MLP f. we denote the dataset as X = {*EIgIENN E Rd) and the latent variables of each layer as z E IRd. The overall MLP can be regarded as a composite mapping f= f f o fi. Each layer fi is a fully-connected layer with an activation function as z' = = (W,z 1- + b,). where W; [wi w !T E Rd, and b E R44 are weight and bias parameters respectively, and a denotes the activation function. To understand the overall model, we start from each layer and consider both directions simultaneously.
Forward Projection. In this direction, information flows from input z to output z'_ To under- stand the internal mechanism, we first analyze each component's functionality. It is easy to know that the activation function like ReLU (Nair & Hinton, 2010) works as the switch and the bias b, acts as the threshold. These two components altogether filter out the unactivated neural nodes. However, we only roughly know that the weight W, behaves like an allocator, which brings the input data to activate the most related neural node. For explainable neural network, we argue that the row vector wi in W; should look similar to a semantic pattern, which we call comprehensibility. We provide the formal definition follows:
3

---

Published as a conference paper at ICLR 2023
Definition 1 (Comprchensibility) A weight wi in FNN is said to be comprehensible if there exists a semantic pattern & E Z similar to it, which means their elements are proportional, that is
32 Z, 3k
0, w] kz,
where Z represents the set of semantic data patterns.
Backward Reconstruction. This direction considers how the output z' backtracks the input as -;-1 g(=', ft), where y denotes the backtracking operation. If this backtracking operation can proceed layer by layer and ultimately reconstruct the original input data a with high precision, we call this property invertibility, which means that any editing of latent variables can be visually reflected by changes in the input data. The formal definition of invertibility is as follows.
Definition 2 (Invertibility). An FNN is said to be c-invertible if there exists a backtracking operation g which satisfies
Ee O, V2 s.t. g(=1, f.)l2 - se
3.2 BOUNDEDNESS AND ORTHIOGONALITY
Boundedness. Previous explainability approaches (Zhou et al., 2016; Simonyan et al., 2014; Zeiler & Fergus, 2014; Springenberg et al., 2015; Zhang et al., 2018; Shen et al., 2021) assume that the activation value 2) is natural indicator, which represents the possibility that the corresponding parameter w³ encodes the input pattern. However, parameter w with high activation value often dissimilar to the input pattern according to Definition 1. Considering as a monotone function, a higher activation value indicates a larger inner product, which is computed as
of = w) x'-1 |// ||||==~~|| cos x
This means that not only a high similarity but also a large amplitude may cause a prominent activa- tion, illustrated in Figure 2a. We need ensure that if wo encodes the input pattern z'-1, w/ elements should be proportional to zi-l's when training converges. To address this, we propose to restrict all w/ in a bounded closed hypersphere follows:
Vi |/ S Cav, where C is constant.
We denote ||2"~|||2 as Cz, so the inner product in Eq. (1) has an upper-bound as follows:
= m 2 < C..
According to Cauchy-Schwarz inequality, s) takes its maximum only when there exists non- negative & such that wi = kzzzr1, which happens to be the similarity in Definition 1. The bounded- ness constraint ensures that large activation value represents a high similarity between the corre- sponding weight and the input pattern, which sufficient condition of Comprehensibillty.
Orthogonality. In the FNN model, each weight w corresponds to a specific pattern. number of approaches (Zeiler et al., 2010; Zeiler & Fergus, 2014; Springenberg et al., 2015) believe that the linear combination of these weights can reconstruct the input as follows:
gi-1 = =gss,, W
wist W 8
(4)
k=1
where 8f represents the projection of - onto w) (i.e., inner product) and 2'-1 denotes the constructed input. We replace function in Definition with Eq. (4) and formulate an optimization problem to achieve the optimal reconstruction as follows:!
min Bx-ne || g( W 2)1l Ez-pp. W $112= Ez-pe ||
W Wz 212
(5)
where Pa is the distribution of z. We minimize Eq. (5) by letting VL 2m (zz (xz7)(WT) (WT I) = 0, seeing Appendix A.2.1 for details. The invertibility property is expected data-independent. Thus we remove the first term Ez-p, (zz") and get:
WTW=1
(6)
which we call the orthogonality constraint. 2 This constraint ensures optimal reconstruction by employing Eq. (4), thus being sufficient condition of Invertibility.
We omit the superscript for brevity.
2To ensure Eq. (6) solvable, W requires full row rank, which means the FNN should be wide enough. Besides, the term orthogonality here means that columns of W should be orthogonal, row wi..
4

---

Published as a conference paper at ICLR 2023
3.3 BORT OPTIMIZER
In this section, we introduce Bort, an optimizer with boundedness (Eq. (2)) and orthogonality (Eq. (6)) constraints for ensuring comprehensibility and invertibility simultaneously. Let L be the objective function. We first formulate the constrained optimization problem as follows:
min L W b¹, S 1
W. .b
S C 1
s.t. W W' I
As the orthogonality constraint implies the boundedness constraint, we simplify Eq. (7) as follows:
min (X; W b'; si W'
T W I, sist. s.t. .
Then we convert Eq. (8) into an unconstrained form by utilizing the Lagrangian multiplier:
min L, W b...1,11100 Σ A Wh
T W I|?:
(9)
1=1
where the second term in Eq. (9) is the penalty term denoted Ly, which is convex concerning wi T W By calculating the derivative (derived in Appendix A.2.2), we propose Bort as follows: (Wi)* <- - a(VI, VI,,) w aVI., ad (Wi (W*)""" W
(10)
where a is the learning rate and À is the constraint coefficient. Following Eq. (10), it is convenient to combine Bort with any other advanced gradient descent algorithm by adding an additional gradient term. Subsequently, we illustrate that the additional constraint does not limit the model capacity.
Proposition 1. Given two-layer linear model h(æ)) W''W with parameter v IR and W E Rmxn, model capacity is equivalent whether or not proposed constraints are imposed on W. Remark. We only consider the most simple without activation functions, proved in Ap- pendix A.2.3. Rigorous proof of keeping model capacity in general cases remains to be completed.
Early research (Huang et al., 2006) proves that two-layer network with random hidden nodes is universal approximator (Hornik et al., 1989), which means that scattering latent weights benefits the property of universal approximation. Moreover, we discover in Section 4.1 that the orthogonality can even improve model performance. addition, we design Salient Activation Tracking (SAT), naive interpreter take full advantage of boundedness and orthogonality (c.f. Appendix A.3).
4 EXPERIMENT
In this section, we evaluate the performance and explainability of Bort-optimized models. We con- duct classification experiments MNIST, CIFAR-10 10, and ImageNet, which shows that Bort boost the classification accuracy various models including VGG16 (Simonyan & Zisserman, 2014), ResNet50 (He et al., 2016), DeiT (Touvron et al., 2021). and Swin (Liu et al., 2021) in Section 4.1. We also present visualization results and compute the reconstruction error to demonstrate the ex- discover that only few binarized latent variables are enough to represent the primary features, whereby we can synthesize the adversarial samples without additional training and parameters. Bort can be incorporated any other optimiza- tion algorithms including SGD, AdamW (Loshchilov & Hutter, 2017), and LAMB (You et al., 2019). We denote the variant of Bort as Bort-X, where X is the first letter of the incorporated optimizer.
plainability endowed by Bort in Section 4.2. Moreover,
4.1 CLASSIFICATION EXPERIMENTS
4.1.1 RESULTS ON MNIST/CIFAR-10
To begin with, we test Bort MNIST (Deng, 2012) and CIFAR-10 (Krizhevsky et al., 2009). We hope to focus purely on fully-connecte layers and variants (c.g., convolution layers) by eliminating potential interference (e.g., pooling layers). Therefore, design a 5-layer all convolutional network (dubbed as ACNN-Small) by replacing all internal max-pooling layers with convolution layers with stride two (see Table 4 the appendix for detail) following All-CNN (Springenberg et al., 2015).
5

---

Published as a conference paper at ICLR 2023
Table 1: Top-1 accuracy (%) of ALNN-CCNSMAI and LeNet on MNIST and CIFAR-I0 datasets.
Setting
Dataset
Model
Optimizer
Epoch
Lr
Awd
A
MNIST
CIFAR-I0
SGD
40
0.01
0.01
79.01
57.35
LeNet
Bort-S
40
0.01
0.01
0.1
88.85 (+9.84)
62.24 (+4.89)
SGD
40
0.01
0.01
98.42
66.67
ACNN-Small
Bort-S
40
0.01
0.01
0.1
99.25 (+0.83)
72.75 (+6.08)
Experimental details. We optimize LeNet (LeCun et al., 1995) and ACNN-Small with SGD and Bort-S separately. The training recipe is quite simple. We set the learning rate to 0.01 without any learning rate adjustment schedule and train each model for 40 epochs with batch size fixed 256. No data augmentation strategy is utilized. The constraint coefficient is set to 0.1, and the weight decay is set to 0.01. All experiments are conducted on one NVIDIA 3090 card.
Result analysis. As shown in Table 1, ACNN-Small optimized by Bort-S perform significantly better than the counterpart model. We attribute this to the orthogonality constraint, which avoids redundant parameters for efficient representation. We further train a LeNet to assess the effect of other modules (e.g., pooling). We see Bort consistently boosts the classification accuracy of LeNet. This shows the internal distribution properties imposed by Bort are robust external interference (see ablation studies in Appendix A.8).
4.1.2 RESULTS ON IMAGENET
We evaluate Bort on the large-scale ImageNet (Deng et al., 2009) with both CNN models (i.e., VGG16 (Deng et al., 2009) and ResNet50 (He et al., 2016)) and ViT-type models (i.e., DeiT-S (Tou- vron et al., 2021) and Swin-S (Liu et al., 2021)) We also combine Bort with three widely used optimizers (i.e., SGD, AdamW (Loshchilov & Hutter, 2017), and LAMB (You et al., 2019)).
Table 2: Top-1 and Top-5 accuracy (%) ImageNet (Deng et al., 2009) dataset.
Model
Optimizer
Epoch
Lr
BS
Top-1
Top-5
SGD
300
0.05
1024
73.13
90.75
Bort-S
300
0.05
1024
73.36 (+0.23)
91.06 (+0.31)
VGG16
AdamW
0.001
1024
64.66
85.11
Bort-A
300
0.001
1024
69.75 (+5.09)
88.72 (+3.61)
SGD
300
0.05
1024
76.57
92.92
Bort-S
300
0.05
1024
77.60 (+1.03)
93.31 (+0.39)
AdamW
0.001
1024
76.91
93.33
ResNet50
Bort-A
300
0.001
1024
77.61 (+0.70)
93.53 (+0.20)
LAMB
300
0.00S
2048
79.72
94.53
Bort-L
300
0.005
2048
79.90 (+0.18)
94.37 (-0.16)
AdamW
0.0005
1024
79.79
94.72
DeiT-S
Bort-A
300
0.0005
1024
80.41 (+0.62)
95.24 (+0.52)
AdamW
300
0.0005
1024
82.63
96.02
Swin-S
Bort-A
300
0.0005
1024
82.71
(+0.08)
96.18 (+0.16)
Experimental details. In recent years, numerous approaches have improved the classification per- formance on ImageNet significantly. Two training recipes are involved. (1) For training CNN-type models (i.e., VGG16 and ResNet50), we follow the recipe in public codes (Wightman, 2019). We set the learning rate to 0.05 for SGD, 0.001 for A, and 0.005 for LAMB. We utilize 3-split data augmentation including RandAugment (Cubuk et al., 2020) and Random Erasing. We train the model for 300 epochs with the batch size set to 1024 for SGD and AdamW and 2048 for LAMB. For LAMB, weight decay is 0.002 and À coefficient to 0.00002: For SGD and A, we set weight decay to 0.00002 and À coefficient 0.0001. (2) For ViT-type models (i.e., DeiT-S and Swin-S), refer to the official descriptions (Touvron et al., 2021; Liu et al., 2021). We fix the batch size to 1024 and train models for 300 epochs with learning rate being 0.0005. We set weight decay 0.005 and A to 0.05. Data augmentation includes RandAugment, Random Erasing, CutMix (Yun et al., 2019), and Mixup (Zhang al., 2017). All experiments are conducted on A100 cards. For more detailed training settings, we refer readers to Table 6 and Table 7 in the appendix.
Result analysis. Table 2 presents the classification accuracy on ImageNet with various models and optimizers. Although Bort is optimizer designed specifically for explainability, it is not trapped

---

Published as a conference paper at ICLR 2023
in the trade-off between performance and explainability. The results demonstrate that Bort sig- nificantly improve the performance of various model types, especially with SGD and Adam We contribute this to Bort's constraint the parameter space, which filters out redundant parameters by orthogonality while maintaining the model capacity. In recent research, OSCN (Dai et al., 2022) has also discovered similar phenomenon that Gram-Schmidt orthogonalization improves the per- formance of the conventional SCN (Wang & Li, 2017).
4.2
EXPLAINABILITY EXPERIMENTS
4.2.1
VERIFICATION OF PROPERTIES
We conduct experiments to verify the existence of orthogonality and boundedness constraints. We first train ACNN-Small models with SGD and Bort-S on MNIST separately to see whether Bort can ensure the two constraints. Then, we compute the reconstruction ratio for each layer to show the contribution of the two constraints to invertibility.
Conv3
Conv
Conv5
Conv2
Comv3
6-14
Diagonal sott
14
10
Convd
Conv5
Non dinddiigonall
(b)
Figure 3: Distribution analysis and reconstruction ratio. (a) We monitor the distribution of diagonal and non-diagonal elements of Gram Matrix. (b) We compute the reconstruction ratio of each layer.
Distribution analysis. Figure 3a shows the distribution of Gram Matrix G = W W, where W denotes the convolution weight. We the non-diagonal ones to O while SGD with L2 regularization keeps squeezing all elements to O. This result demonstrates that our proposed Bort can effectively ensure the two constraints.
that Bort-S drives the diagonal elements closer to 1 and
Reconstruction ratio. Following the reconstruction protocol described in Eq. (4), we compute the =/-'|//||> |||/||='=1 II.- l//|- Figure 3b shows that layers optimized by Bort- S can consistently reconstruct with much higher precision than SGD, demonstrating that Bort-S is significantly superior to SGD in boosting invertibility.
reconstruction error ratio ||~--|- - ;²
4.2.2 QUALITATIVE VISUALIZATION
In this part, we conduct reconstruction experiments and saliency analysis on MNIST, CIFAR-10, and ImagenNet. Depending the dataset size, train the ACNN-Small (5 layers) MNIST and CIFAR-10 10 and the ACNN-Base (12 layers) on ImageNet, seeing Table 4 and Table 5 for details in the appendix. We generate the visualizations using feature maps the 5th layer and 8th layer of ACNN-Small and ACNN-Base respectively.
Reconstruction. After training the models, employ guided backpropagation (Springenberg et al., 2015) to reconstruct the input data (c.f. Appendix A.4). As shown in Figure 4, the model optimized by Bort-S can well preserve detailed information, such texture and edge, during recon- struction. contrast, the model optimized by SGD will clutter features. This phenomenon fully demonstrates that Bort can improve the invertibility of models.
Saliency Analysis. Exploiting boundedness and orthogonality, we design SAT algorithm to gen- erate saliency maps (see details in Appendix A.3). Figure displays the saliency map visualization results. Compared with conventional CAM (Zhou et al., 2016), our SAT approach renders more precise pattern localizations, thanks to the pixel-Ievel feature backtracking. Moreover, the saliency maps of the model optimized by Bort concentrate more on salient objects than baseline optimizers (i.e., SGD/AdamW), proving the advantage of Bort boosting the comprehensibility of models.
7

---

Published as a conference paper at ICLR 2023
MNIST
CIAR-100
ImageNet
4
5
Image
SGD
4
Bort-S
Figure 4: Reconstruction MNIST, CIFAR-IO 10, and ImageNet. We adopt guided backpropagation to reconstruct the input data, and our Bort achieves better reconstruction performance.
MNIST
CIFAR-10
ImageNet
3
9
Image
CAM (SGD)
SAT (SGD)
9
3
SAT (Bort-5)
Figure 5: Generating saliency maps by CAM (Zhou al., 2016) and our proposed SAT. We observe that SAT with Bort-S generates the best results and focuses mainly on the salient parts of objects. We set K 64 for MÑIST and CIFAR-10 10 and K 64 for ImageNet.
4.2.3 QUANTITATIVE ANALYSIS
Deletion/insertion metrics. We compute the deletion/insertio metrics (Petsiuk et al., 2018) on MNIST, CIFAR-10, and ImageNet. The deletion metric measures the performance drop as removing important pixels gradually, while the insertion metric does the opposite process. For deletion, the smaller the Area Under Curve (AUC) value, the better the explainability; for insertion, a larger AUC is expected. Baselines ImageNet and MNIST/CIFAR-IN are optimized by AdamW and SGD, respectively. As shown in Table 3, in most cases, for the common interpreters (i.e., CAM, IG, RISE, XRAI, and GuidedIG), the Deletion/Insertion metrics of models optimized by Bort significantly better than the baseline (optimized by SGD/AdamW), Besides, we also observed that when using naive SAT, the model optimized by Bort achieved consistent improvement in all cases. We think this is because SAT takes full advantage of the boundedness and orthogonality provided by Bort.
Table 3: Insertion and deletion metrics MNIST, CIFAR-10. 10, and ImageNet.
MNIST
CIFAR-10
ImageNet
Method
Optimizer
Deletion]
Insertion
Deletion
Insertion
Deletion
Insertion
SGD/AdamW
0.25
0.67
0.32
0.70
0.49
0.67
CAM
Bort
0.31 (+0.07)
0.63 (-0.05)
0.29 (-0.04)
0.76 (+0.06)
0.44 (-0.05)
0.77 (+0.10)
SGD/Adamw
0004
0.73
-0.37
0.81
0.07
0.79
IG
Bort
-0.07 (-0.03)
0.78 (+0.05)
-0.44 (-0.07)
0.84 (+0.03)
0.07 (+0.00)
0.88 (+0.09)
SGD/Adam
0.06
0.64
0.14
0.75
0.43
0.75
RISE
Bort
0.02 (-0.04)
0.72 (+0.08)
0.14 (+0.00)
0.78 (+0.03)
0.39 (-0.05)
0.82 (+0.06)
SGD/AdamW
0.73
0.24
0.76
0.39
0.78
XRAI
Bort-S
0.13 (+0.01)
0.79 (+0.06)
0.22 (-0.02)
0.79 (+0.03)
0.34 (-0.04)
0.84 (+0.06)
SGD/Adam W
0.04
0.71
-0.28
0.78
0.06
0.82
GuidedIG
Bort
-0.05 (-0.01)
0.78 (+0.06)
-0.26 (44.01)
0.82 (+0.04)
0.07 (+0.00)
0.88 (+0.06)
SGD/Adamw
0.26
0.61
0.31
0.76
0.35
0.78
SAT (Ours)
Bort
0.05 (-0.20)
0.80 (+0.20)
0.27 (-0.04)
0.81 (+0.05)
0.32 (-0.04)
0.84 (+0.07)
8

---

Published as a conference paper at ICLR 2023
4.2.4 FEATURE DECOMPOSITION AND ADVERSARIAL SAMPLES
In this part, we explore what Bort can provide for an in-depth understanding of networks through feature decomposition and sample synthesis (see details Appendix A.5). We examine an extreme case where we can reconstruct the input data with partial features. Most adversarial samples rely on additional training and parameters, and only few attempts focus on semantic adversarial sam- ple generation (Mao et al., 2022). Therefore, we investigate whether we can achieve this without additional expense after thoroughly understanding the internal mechanism of networks.
patterns
8 clusters
Image
7
Source
Ole core
Kmeaans
Target
SGD
7
7
Bort-S 7
7
Adversary
(a)
Figure 6: Feature decomposition and adversarial sample synthesis. (a) Networks optimized by Bort achieve precise reconstruction only with the 64 most salient features. (b) We visualize the top 64 features separately and run the K-Means algorithm their relations. (c) We synthesize semantic adversarial samples without any additional expense.
Feature decomposition. Given the feature map at the 5th layer of ACNN-Small trained on MNIST, we choose the 64 most salient channels out of 2592 channels according to the maximum activations each channel while setting all other activations to O. Finally, the original 2592 x 6 x variables. Figure 6a shows that even extremely sparse variables can re- construct the input data for Bort-optimized but SGD-optimized networks. We also reconstruct each variable and use the K-Means algorithm to cluster them as shown in Figure 6b. We observe pattern-red related clusters, showing the Bort-optimized CNN is compositional and understandable.
feature slice. Then, we set the maximum activation to at each chosen
only keep 64 binarized variables from
Adversarial sample synthesis. For image classification tasks, most networks predict class scores using a fully-connected layer following the spatial- -aggreaigggggg pooling layer. Therefore, we con- jecture that spatial information is not important for classification, and we can manipulate spatial features to synthesize adversarial samples. We first choose a source and target data pair and denote their feature map as Z and Z'. respectively. Different from decomposition, we select the top 64 channels of source Z according the target Z' and synthesize sparse binarized feature map z with them. Finally, we reconstruct Ztr to obtain the adversarial sample without additional pa- rameters and training. Interestingly, shown in Figure 6c, the obtained adversary is semantically explainable and casily fools the classifier.
CONCLUSION
In this work, provide formal definition of explainability with comprehensibility and invertibil- ity. We then derive two sufficient conditions (i.e., boundedness and orthogonality) and introduce the optimizer Bort optimize FNNs efficiently with constraints. Classification results demonstrate that by filtering out redundant parameters, Bort consistently boosts the performance of CNN-type and ViT-type models on MNIST, CIFAR-10. and ImageNet datasets. Visualization and saliency analysis qualitatively and quantitatively prove Bort's superiority in improving the explainability of networks. Surprisingly, we find that highly sparse binarized latent variables networks optimized by Bort can characterize primary sample features, whereby we can synthesize adversarial samples without additional expense. We expect our work to inspire more research for understanding deep networks. As derive Bort under the assumption that f is a sufficiently wide network. it would be an interesting direction to investigate the properties of Bort for narrow or extremely deep networks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, part by the National Natural Science Foundation of China under Grant 62125603, and in part by a grant from the Beijing Academy of Artificial Intelligence (BAAI).
REFERENCES
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural additive models: Interpretable machine learning with neural nets. In NeurlPS, pp. 4699 4711, 2021.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robers Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 1011), 2015.
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-based logic explanations of neural networks. In AAAI, pp. 6046-6054, 2022.
Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Sal- vatore Rinzivillo. Benchmarking and survey explanation methods for black box models, arXiv, abs/2102.13076 2021.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++:: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, pp. 839- 8499884744 2018.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. N/PS, 32, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence. 2(12):772-782, 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR (Workshop), pp. 702-703, 2020.
Wei Dai, Chuanfeng Ning, Shiyu Pei, Song Zhu, and Xuesong Wang. Orthogonal stochastic config- uration networks with adaptive construction parameter for data analytics. arXiv, abs/2205.13191, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hicrarchical image database. In CVPR, 248-255, 2009.
Imagenet: A large-scale
Li Deng. The mnist database of handwritten digit images for machine learning research. 1EEE Signal Processing Magazine, 29(6):141-142, 2012.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv, abs/2010.11929, 2020.
Lilian Edwards and Michael Veale. Slave to the algorithm: Why right to explanation is probably not the remedy you are looking for. Duke L. & Tech. Rev., 16:18, 18, 2017.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In ICCV, 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv, abs/2010.0141., 2020.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In DSAA, pp. 80-89, 2018.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359- -666, 1989.
Guang-Bin Huang, Lei Chen, Chee Kheong Siew, et al. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Networks, 17 (4):879-892 892, 2006.
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attributions through regions. In ICCV, pp. 4948-4957, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv,
abs/ (bbs/141..6900, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 1995.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In AAAI, volume 32, 2018.
Haoyu Liang, Zhihao Ouyang, Yuyuan Zeng, Hang Su, Zihao He, Shu-Tao Xia, Jun Zhu, and Bo Zhang. Training interpretable convolutional neural networks by differentiating class-specific filters. In ECCV, pp. 622-638, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, pp. 10012- 10022, 2021.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv, Abs/1711.00100/ 2017.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. NIPS, 30, 2017.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. NeurIPS, volume 30, 2017.
Xiaofeng Mao, Yuefeng Chen, Ranjie Duan, Yao Zhu, Gege Qi, Shaokai Ye, Xiaodan Li, Rong Zhang, and Hui Xue. Enhance the visual representation via discrete adversarial training. arXiv, abs/2209.07735. 2022.
Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretabile, leak-proof concept-based models. arXiv, abs/2205. 15512, 2022.
Grégoire Montavon, Wojeiech Samek, and Klaus-Robers Müller. Methods for interpreting and un- derstanding deep neural networks. Digital signal processing, 73:1-15, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In KDD, pp. 1135-1144, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In ICCV, pp. 618-626, 2017.
11

---

Published as a conference paper at ICLR 2023
Wen Shen, Zhihua Wei, Shikun Huang, Binbin Zhang, Jiaqi Fan, Ping Zhao, and Quanshi Zhang. Interpretable compositional convolutional neural networks. arXiv, 0tss/1107.0474,, 2021.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, pp. 3145-318 1153,,15,, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, abs//1409.n 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR (Workshop). 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv, abs//770.03825.222 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (Workshop), 2015.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pp. 3319-3328, 2017.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pp. 10347-10357, 2021.
Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: neural-backed decision trees. arXiv, abs/2004.00221,, 2020.
Dianhui Wang and Ming Li. Stochastic configuriinnn networks: Fundamentals and algorithms. IEEE transactions on cybernetics, 47(10))3466-3477 2017.
Yipei Wang and Xiaoqian Wang- Self-interpretable model with transformation equivariant interpre- tation. In NeurIPS, pp. 2359-2372, 2021.
Ross Wightman.
Pytorch image models. pytorch-imee wearchh-imager meal-iimmmemmmeee 1 s, 2019.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv, abs/1904.00962, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp- 6023-6032, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. pp. 818-833, 2014.
Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Robert Fergus. works. In CVPR, pp. 2528-2535, 2010.
Deconvolutional net-
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Paze mixup: Beyond empirical risk minimization. arXiv, abs/ 0bbs/1710.094/ 2017.
Quan-shi Zhang and Song-Chhun Zhu. Visual interpretability for deep learning: survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27-39, 2018.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In CVPR, pp. 8827-8836, 2018
12

---

Published as a conference paper at ICLR 2023
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pp. 2921-2929, 2016.
Ding-Xuad Xuan Zhou. Universality of deep convolutional neural networks. Applied and computational harmonic analysis, 48(2):787-794, 2020.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv, abs/1 2017.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 CLARIFICATION OF TERMS
Explainability and interpretability are often used interchangeably in many works of literature, although some papers actually point out subtle differences between them. In this paper, we refer to the definition in Montavon et al. (2018), where an "Interpretation" maps abstract concepts into an understandable domain and an "explanation" reveals the internal mechanism (e.g., how the internal features are calculated by the model). Our Bort optimizer does not focus on the input/output behavior of the model for mapping the output features back to an understandable format (eg, image and text), but aims revealing the internal mechanism of the black-box model by constraining the model parameters. Specifically, it includes: (I) aligning the inner product operation in FNN to the cosine similarity (comprehensibility) (2) allowing the internal features of the network recover the features of the previous layer to the greatest extent (trasparencyliveertiiig
We think that the property pursued by Bort is closer the "explainability" in Montavon et al. (2018) (also similar to "model-centric" in Edwards & Veale (2017)).
A.2 DERIVATION AND PROOF DETAILS
A.2.1 DERIVATION OF EQ. (5)
Let A WT and L = Ez=p= || A as follows:
A=112. We compute the first-order derivate of L with respect
dL = dB Tr (= - Az) Az)]
= dEgups
Tr zz (A2 -A2-24 I)
Tr [2E xmpps (zz (22")(A (A- I) dA]
VL 2Eg-ps (zz (A-I).
(11)
To minimize L, need let the derivate be zero Thus, we get VL 21 z0p (2=")(WT'
I)= = 0.
A.2.2 DERIVATION OF EQ. (10)
We denote the second term in Eq. (9) L, = - W -y'-1122
-1 Since | W -1
is convex with respect to Wi W boundedness and orthogonality will hold at convergence A large enough. Following the standard gradient descent algorithm, we compute the gradient of L, with respect to W as follows:
dL, = Xa Tr [((W+)
1) (W 1) 4X, Tr [((W")>''(WW)) (W')) (W")")
VL, 4x; (W")" W W w').
(12)
For simplicity, we let X; be the same, LL,- becomes 4A (Wi) W -W:). By substitute Eq. (12) into standard gradient descent algorithm, we propose Bort as follows: (Wi)* + W aVV, VL,)= W aVL, aA (W' W where a is the learning rate and À is the constraint coefficient.
(13)
A.2.3 PROOF OF PROPOSITION
Proof. We denote the model capacity as H., He for unconstrained/consraandd cases, respectively. (1) It is obvious that H., 2 He because H. might be squeezed by additional constraints. (2) We then demonstrate that H., C He. Given set of configuration (vu, Wo) E Ha, we have any data æ being projected to oi Woz. We can decompose Wo utilizing SVD as follows:
3US E Rmxmm V E RuXn E E mn s.t. U = II, VI. = I, = -UEV.
(14)
Therefore, if letting W1 VT and V1 = ETUTO
vo. we have vg Wa = of Wjx, which means the configuration (wy. Wn) and (w,, Wi) are equivalent. Thus (0, W.: W, WW = I) E H.. H. C H. is proved. Above all, H, H.
14

---

Published as a conference paper at ICLR 2023
Algorithm 1: The SAT algorithm.
Input:
The top feature map Z, the backtracking mapping y, number k, constant B, and threshold y.
Output: Saliency map A.
Reset set of tuples M = 0: Compute the vector y IR by passing Z through max- max-pooling layer;
Get the index set of largest elements of vy as 1I. = {i y E topk(y)): foreach i Iıg do
Initiate the zero-filled 2 with the same size of Z:
Get the ith slice of 2 as 2
Set the position in Z corresponding to the maximum in 2, to constant B; Recover the signal S as S g ):
Obtain the mask M' by binarizing S through given threshold
10
Update M + MU{(M) v:)}:
Calculate the saliency map as A W. M.
A.3 DETAILS SALIENT ACTIVATION TRACKING (SAT)
Motivation for SAT. We believe that mainstream interpretation methods are suboptimal for Bort because they do not take full advantage of boundedness and orthogonality, which results in Dele- tion/Insertion metrics not being significantly improved in few cases, shown in Table 3. There- fore, we germinated the idea of building a saliency map generation algorithm (SAT) for visual tasks exploiting boundedness and orthogonality.
Implementation SAT. Due boundedness and orthogonality, the model optimized by Bort hibits the properties of Principal Component Analysis (PCA) to some extent. Therefore, analogous the PCA reconstruction process. SAT selects the k most salient channels of the top feature map Z E Rexhxw for back-pre propagation. Note that if we do backpropagation directly. we will get fea- tures/signals instead of attribution/saliencyy. because saliency more similar to masks than signals. To address this, we convert the features into masks by binarizing the reconstructed features of each channel, and calculate the final saliency map by weighted average of those masks. This design idea also appeared in RISE (Petsiuk et al., 2018). The difference is that RISE randomly samples the mask, and we calculate the mask of the salient channel by back backkpropagation. We present the SAT algorithm as follows:
A.4 DETAILS OF GUIDED-BACKPROPAGATII
We follow the standard algorithm of Guided-BP (Springenberg et al., 2015) for recovering the sig- nals layer by layer. During the forward phase, we denote the input a, and the ReLU layer com- putes the output as
(eLU 0, az,
5
(15)
if a >
We need to store the positions where a; O. During the back-propagation phase, given the feature & from the upper layer, the Guided-BB BP defines the backpropagation rule as
81: (0,
if a, > 0 and &; > 0
à Guided
(16)
otherwise
Other convolution layers can perform backpropagation according to Eq. (4).
A.5 DETAILS OF DECOMPOSITION AND SYNTHESIS
Given input image X, we first calculate the top feature map Z = f(X) E gex and get the vector y Re by passing Z into max-pooling layer.
15

---

Published as a conference paper at ICLR 2023
Table 4: Architecture of ACNN-Small for MNIST and CIFAR-IO.
ACNN-Small
Layer
for MNIST
for CIFAR-IO
Input 28 x 28 gray image
Input 32 x 32 RGB image
conv
5 x 5, 8, padding + ReLU
5 5, 24 channel ReLU
conv2
2 x 2, 24, stride 2
2 x 2, 64, stride 2
conv3
x 4, 288, padding I + ReL.U
4 x 4, 512, padding + ReL.U
conv4
2 x 2, 864, stride 2
1536, stride
conv5 pool
x 3, 2592, padding + ReLU 3 x 3, 4608, padding + ReLU
adaptive max pool 10-way softmax
softmax
Table 5: Architecture of ACNN-Base for ImageNet.
ACNN-Base
Layer
ImageNet
Input 224 x 224 RGB mmgge
conv1
10 x 10, 96, stride 3, padding 4 ReLU
conv2
1 1, 96, stride
ReLU
conv3
3 x 3, 96, stride 2 + ReLU
conv4
3 x 3, 256, stride 1 ReLU
conv5
1 1 1, 256, stride 1 + ReLU
conv6
3 256, stride 2 ReLU
conv7
3 3, 384, stride + ReLU
conv8
1 1, 384, stride 1 ReLU
conv9
3 3 3, 384, stride 2 + ReLU
conv 10
x 1024, stride ReLU
conv 11
1x 1024, stride 1 ReLU
conv 12 pool
1x 1, 1000, stride + ReLU
adaptive max pool
Decomposition. Analogous to PCA algorithm, we obtain the index set II of i largest elements of y. For any index i sI, we initiate a zero-filled 2o with the same size of Z. Then we set the position in slice Z corresponding to the maximum in Z, to constant value B. Finally, we perform backpropagation to get g)Z)). Repeating the above procedure & times, we can obtain the set of recovered signals S (S' II,, which is top-k decomposition of X.
Synthsis. Given the target feature map Z' === and the source feature map Z f(X*). we first construct the index set Is of & largest elements of y'. Then we initiate zero-filled "r Subsequently, for each index i It. we set the position in slice Z corresponding the maximum in Z to a constant value B. Obviously, n possesses the salient channels of z and the spatial information of Z simultaneously. Finally, we get the adversarial sample as Xadv g( (Z*),, which may have the outlook of X but be classified the same X
A.6 ARCHITECTURE DETAILS
In this work, we mainly focus on fully-connected layers and the variants. Previous research (Sprin- genberg et al., 2015) has discovered that networks only with convolution layers achieve compet- itive performance as conventional CNN. Therefore, we replace each internal max-pooling layer with convolution layer (stride 2). According to the different image sizes of datasets, design two networks (i.e., ACNN-Small and ACNN-Base with different perceptive fields following All- CNN (Springenberg al., 2015). The detailed architectures displayed in Table 4 and Table 5.
16

---

Published as a conference paper at ICLR 2023
Table 6: Recipes for optimization setting on ImageNet.
Model
Optimizer
Awd
Epoch
DropPath
Momen.
BS
LR
Sched.
Warmup
SGD
0.00005
300
0.9
1024
0.05
Cos.
Bort-S
0.00002
0.001
300
0.9
1024
0.05
Cos.
VGG16
AdamW
0,00002
300
1024
0.001
Cos.
5
Bort-A
0.00002
0.0001
300
1024
0.001
Cos.
5
SGD
0.00002
300
0.9
1024
0.05
Cos.
Bort-S
0.00002
0.0001
0.9
1024
0.05
Cos.
AdamW
0,000022
300
1024
0.001
Cos.
5
ResNet50
Bort-A
0.00002
0.0001
300
1024
0.001
Cos.
5
LAMB
0.02
300
2048
0.005
Cos.
Bort-L
0.002
0.00002
300
2048
0.005
Cos.
AdamW
0.05
300
0.1
1024
0.0005
Cos.
5
DeiT-S
Bort-A
0.005
0.05
300
0.1
1024
0.0005
Cos.
5
AdamW
0.05
300
0.3
1024
0.0005
Cos.
Swin-S
Bort-A
0.005
0.05
300
0.3
1024
0.0005
Cos.
Table 7: Recipes for loss and data setting on ImageNet.
Model
Optimizer
AA
Mixup
CutMix
Brase
Color
AugSplit
JSD
BCD
SGD
m9-mstd0.5
0.6
VGG16
Bort-S
m9-mstd0.5
0.6
3
AdamW
m9-mstd0.5
0.6
3
Bort-A
m9-mstd0,
0.6
3
SGD
m9-mstd0.5
0.6
Bort-S
m9-mstd0.5
0.6
ResNet50
AdamW
m9-mstd0.5
0.6
Bort-A
m9-mstd0.5
0.6
LAMB
m7-mstd0.5
0.1
0
Bort-L
m7-mstd0.5
0.1
0
AdamW
m9-mstd0.5
0.25
DeiT-S
Bort-A
m9-mstd0.5
0.8
0.25
0.3
AdamW
m9-mstd0.5
0.8
0.25
0.4
Swin-S
Bort-A
m9-mstd0.5
0.8
0.25
0.4
A.7 TRAINING RECIPES ON IMAGENET
Numerous attempts have explored effective techniques boost the classification performance on ImageNet in recent years. To compare with other optimizers under fair settings, we employ two mainstream training recipes. For CNN-type networks (i.e., VGG16 and ResNet50), we follow the setting in the popular open-source library tinun (Wightman, 2019): For ViT-type networks (i.e., DeiT-S and Swin-S), we employ the official setting described in the original papers. Detailed settings are shown in Table 6 for optimization and Table for data augmentations and loss functions.
Top-1 Accuracy (%) CIFAR-10
Top-1 Accuracy
60
Dort
30
And
1.0x10
1.0
1.0
1.0x10
1.0
Figure 7: Ablation study about Avd and A on CIFAR-10.
17

---

Published as a conference paper at ICLR 2023
MNIST
4
5
SGD
4
5
CIFAR-10
Image
SGD
(b)
ImageNet
Image
Bort-S
Figure 8: More reconstruction results on MNIST, CIFAR-10, and ImageNet datasets.
A.8 ABLATION STUDY ON CIFAR-10
In this part, explore the influence of hyper- papem-parmmeters (i.c., weight decay Apd and constraint coefficient )ort) on CIFAR-10 dataset. Figure 7 shows that our Bort is more stable under different Bort In contrast, large And tends to collapse networks. We think this is because the constraints of Bort limit weights on the hyper-sphere instead of forcing them to move towards the original point.
A.9 MORE QUALITATIVE RESULTS
We provide more visualization results in this part. To ensure the fairness of visualization, we ran- domly select candidates for visualization. For reconstruction results shown in Figure 8, Bort con- sistently boosts the reconstruction accuracy for all three datasets. Optimized by Bort, networks become invertible and easily recover most of the detailed information, such as edges and textures. For saliency maps shown in Figure 9, networks optimized by Bort better focus on the salient objects than SGD, especially for the Small model on MNIST and CIFAR-10. We also discover that for the larger ACNN-Bar Base model on ImageNet not all results are distinctly improved when opti- mized by Bort. We think this is because ACNN-Base is not wide enough to ensure perfect feature
18

---

Published as a conference paper at ICLR 2023
MNIST
3
9
4
3
5
Image
f
O
CAM (SGD)
SAT (SGD)
9
3
9
3
S
SAT (Bort-S)
CIFAR-10
Image
CAM (SGD)
SAT (SGD)
SAT (Bort-S)
ImageNet
Image
CAM (SGD)
SAT (SGD)
SAT (Bort-S)
Figure 9: More saliency map results on MNIST, CIFAR-10, and ImageNet datasets.
backtracking according to the orthogonality condition (i.e., Eq. (6) is not solvable). To address this, modifying the architecture with more channels for each layer may be one possible solution, which we will investigate in the future.
19


Reviews:
[{"review_id": "CislBJN4_8", "reviewer": "Reviewer_eZA8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors develop notions of comprehensibility and transparency as indicator of explainability, and develop formal notions to introduce these as constraints during model training. The authors also propose a new technique for generating saliency maps. Subsequently, they show that inclusion of these constraints improves accuracy as well as the insertion/deletion metric for quantifying the quality of saliency maps.", "summary_of_the_review": "Overall, while I think the formalization does reduce to straightforward constraints, the results are (surprisingly) good, and the community could benefit from wider dissemination. I would recommend the paper to be accepted at the venue based on the performance but I would have much appreciated any deeper insights.", "strength_and_weaknesses": "Baking explainability into models by somehow making them inherently interpretable is a relevant problem. The approach proposed in the paper seems sensible and the results look convincing. Some more thoughts:\n\n1. I am not sure if the notions of comprehensibility and transparency are well-established. With that in mind, the presentation of these appears somewhat ad-hoc.\n\n2. While the development appears logical, the end result is quite simple — instead of penalizing the L_p norm of the parameters in the loss function, penalize deviation from orthogonality instead. I wouldn’t be very surprised if a little more algebra reveals some relationship to L_p norm. However, simplicity itself is not something to criticize.\n\n3. While I wasn’t very convinced of the value of the proposal during the development, the results look quite good. Specifically, the improvement in accuracy, improvement in explainability metrics, feature composability, and generation of adversarial examples are all well documented. The only sub-section that seems unsurprising is the verification of properties — orthogonal constraints lead to orthogonality, and the inherent reconstruction constraint leads to good reconstruction.\n\n4. I am not quite sure if I can clearly argue why the technique produces good results. The constraints were developed for one set of objectives yet they somehow improve on another (better established) set as well. A discussion of that and authors’ thoughts on the possible explanation would improve the paper.\n\n5. Minor errors: “possesses comprehensible and transparent simultaneously”, wrong double quotes in “relation”, “only and if only”, “equivalent whether to exert”, “benefits of the two constraints benefit of transparency”, add x-axis label to Fig 3a, “assumption f enough”.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CImyyUemevx", "reviewer": "Reviewer_dhB8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new optimizer called Bort to improve model explainability (comprehensibility and transparency) with Boundedness and orthogonality constraints on model parameters.", "summary_of_the_review": "The approach of mathematical definition and analysis of explainability proposed in this paper is novel and plausible. However, the paper is over-ambitious and experiment is not adequate.", "strength_and_weaknesses": "Strengths\n+ Explainability is a critical and challenging field in ML, and the mathematical attempt proposed in this paper is encouraging.\n\nWeaknesses\n- Boundedness constraint seems similar to cosine similarity. Any difference?\n- The discussion to treat CNN and Transformer as MLP is quite sketchy, e.g. \"can be unified under one meta-structure, a multi-layer perceptron (MLP for short) with optional nonparametric operations. For example, the convolutional layer and the transformer layer additionally use folding/unfolding and the self-attention operation, respectively\". Will parameter sharing and attention introduce extra complexity and inconsistency to the definition and analysis of comprehensibility and transparency?\n-\"the row vector wij in Wi\". wij is a weight, right? How can a weight \"look similar to a specific data pattern\"? What does \"look similar\" mean? How can weights look similar to an input data pattern after many layers of non-linear transformation?\n- The definition of comprehensibility is confusing. Is a semantic data pattern z a vector or scalar value? How will z be proportional to a weight wij? especially in later layers?\n- The definitions of comprehensibility and transparency are debatable, and it is also a question whether comprehensibility and transparency are sufficient for complete explainability.\n- Both boundedness and orthogonality constraints seem to limit the expressivibility of FNNs, any insight why they actually improve accuracy? some regularization effects?\n- Although  a two-layer network is a universal approximator, proposition 1 is on  a two-layer linear network.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ACjmDzq3Vv", "reviewer": "Reviewer_t9ra", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces two methodological components: (1) BORT, a regularization for feed-forward neural networks term that encourages the various layers to implement a bounded orthogonal projection, (2) SAT, a saliency algorithm specifically designed for BORT-regularized networks.  The empirical evaluation highlights how BORT can improve accuracy of several models on MNIST and CIFAR10, and that BORT+SAT produces \"clear\" saliency maps with good deletion and insertion scores (which measure sensitivity of model predictions to pixels identified as relevant by the saliency maps).  In addition, the BORT+SAT combo facilitates acquiring features with input reconstruction capabilities and generation of adversarial examples.", "summary_of_the_review": "Explainability-enhancing regularizer based on per-layer orthonormalization + ad-hoc saliency map algorithm.  Alas, the latter is poorly described, complicating evaluation.", "strength_and_weaknesses": "PROS\n- Proposed idea is is based on a solid intuition.\n- Formalization is clean and looks correct.\n- No performance drop upon applying BORT - actually, quite the contrary.\n- Text is mostly well written.\n\nCONS\n- Key element (SAT) is not well described, complicating evaluation of the results.\n- Other aspects of the experiments are not well explained.\n- Some terminological choices are a bit confusing.\n- Relationship to existing approaches is not entirely clear.\n- A subset of the results rely on guided backpropagation, which is known to be flawed.\n- Only two smaller image classification data sets (not a huge deal).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster