C 3-LRP: Visual Explanation Generation based on Layer-Wise
Relevance Propagation for ResNet
Félix Doublet

Seitaro Otsuki

Iida Tsumugi

Komei Sugiura

Keio University
In this paper, we focus on the task of visualizing important regions in an image as high-quality visual explanationsof the model’s decisions with a clear theoretical background. We introduce a novel calculation method for Layerwise Relevance Propagation (LRP) speciﬁcally tailored to models featuring skip connections such as ResNet. Thismethod’s strength lies in its adaptability, as the backpropagation technique is distinctly deﬁned for each layer,enhancing its extensibility. To validate our method, we conduct an experiment on the CUB-200-2011 dataset.
The proposed method successfully generates appropriate explanations and, based on the Insertion-Deletion score,outperforms the baseline methods.

1.

For models based on Convolutional Neural Networks(CNNs), a multitude of studies have put forward methodologies for generating visual explanations, with notable examples including GradCAM [Selvaraju 17], LIME [Ribeiro
16], and RISE [Petsiuk 18]. These techniques predominantly rely on predeﬁned computational approaches to formulate explanations. While these methods are generally
agnostic to the speciﬁc architecture of the model, they encounter limitations in generating tailored explanations for
ResNet architectures, often resulting in attending irrelevant
areas, as shown in Section 6.
Another notable method is LRP [Bach 15], which employs
backpropagation from the output for explanation generation. Nevertheless, this method has only been applied tomodels without residual connections, and its extension tomodels incorporating such connections such as ResNet remains an area yet to be explored.
In this study, we extend the conventional method ofgenerating explanations for ResNet models by integrating
LRP. LRP is distinguished by its well-established theoretical framework and transparent computational processes.
Thus, using rule-based explainers, we are aiming to generate high-quality explanations. Our approach leverages theclarity and transparency inherent in LRP to elucidate thedecision-making processes within ResNet models, thereby
contributing to a deeper understanding and improved interpretability of these neural networks.
Our study distinguishes itself from existing researchthrough two primary innovations.

Introduction

The widespread adoption of neural networks underscores
the critical importance of explainability of these models[Shrikumar 17] [Ribeiro 16]. The European Parliament haseven proclaimed that AI systems must be safe and transparent in its AI Act [Madiaga 23] promulgated in December
2023. This strengthens the needs of accurate and meaningful explanations in neural network models.
However, current methods often lack transparency, leading the interpretation of the results to be a non-trivial task
[Molnar 20]. Additionally, the black-box essence of neuralnetwork architectures tends to veil the underlying logic of
their decision-making processes. This lack of transparency
poses signiﬁcant challenges in verifying the validity of themodels’ classiﬁcations, necessitating a rigorous assessment
to determine if they are based on pertinent or irrelevant factors. To address this issue, research in eXplainable Artiﬁcial
Intelligence (XAI) is aiming to delve into the inner workingsof neural networks and to develop trustworthy models.
The generation of visual explanations within neural networks poses a signiﬁcant challenge, necessitating the precise extraction of critical areas [Jacovi 23]. For example,
the current Layer-wise Relevance Propagation (LRP) implementation encounters notable challenges when applied
to ResNet architectures. ResNet’s residual connections create non-linear and multiple relevance pathways, which are
not handled by the typical relevance attribution process of
LRP. Another diﬃculty in this process is the frequent absence of a clear and deﬁnitive ground truth, which serves asa benchmark for validating these explanations. This task
demands a meticulous balance in identifying important areas, ensuring that the focus is neither excessive nor insufﬁcient. Consequently, the explanations generated by LRP
for ResNet models often do not show reliable or insightful results, as shown in Section 6. Therefore, while LRP
oﬀers a framework for relevance attribution in neural networks, its limitations, particularly in handling architectures
like ResNet, presents a challenge, underscoring the pressing
need for advanced methodologies in this domain.

• We introduce a novel calculation method for LRP,
speciﬁcally tailored to models featuring skip connections. This novel approach is designed to adapt LRP’s
methodology to the unique architectural characteristics of such models, enhancing its applicability andeﬀectiveness.
• We implement the Contour Component Choice (C 3 )mechanism. This mechanism signiﬁcantly elevates thequality of the generated explanations by strategicallyselecting the most pertinent area.

Contact: 3-14-1, Hiyoshi, Kohoku Ward, Yokohama, Kanagawa, felixdoublet@keio.jp

1

Figure 2: Schematic diagram of the calculation method forRelevance R(x) for ResNet50. The black path represents
the forward propagation, where p(ŷ) is computed based on
the input image x. The purple path illustrates how Relevance R(x) is calculated by back-propagating R(ŷ) using
our method.
Figure 1: Example of an input image (left) and the visualexplanation (right).

2.

Problem Statement

In this paper, we focus on the task of visualizing important regions in an image as a visual explanation of
the model’s decisions. The pixels that contributed to themodel’s prediction should be attended.
Figure 1 shows an example of an CUB-200-2011 image.
The left and right ﬁgures show the input image and thevisual explanation, respectively.
The input is an image x ∈ Rc×w×h , where c, w, and hdenote the number of channels, width and height of the input image, respectively. The output p(ŷ) ∈ Rc denotes thepredicted probability for each class, where C is the number of classes. Additionally, the importance of each pixel
is obtained as a heatmap α ∈ Rw×h which is used as a visual explanation. In this paper, we assume that the modelis based on a ResNet architecture. The Insertion-Deletion
score [Petsiuk 18] is used as an evaluation metric for explanation generation.

3.

Proposed Method

3.1

Relevance backpropagation

Figure 3: Schematic diagram of a Bottleneck layer fromResNet model (left) and how it is considered when computing the Relevance.
method for Relevance R(x) for ResNet50. The input is animage x ∈ R3×h×w where h and w denote the height andwidth of the image, respectively.
Figure 3 shows a schematic of the calculation method
for the Relevance R(L) of the layer L. For computing R(L)based on R(L+1) , we consider a Bottleneck block as a single
layer with z as an input and z ′ as an output.
This allows the backpropagation to be calculated in
the same manner as for other layers. We consider z ∈RcI ×hI ×wI and z ′ ∈ RC×U ×V where C, U , and V represent
the number of output channels, height, and width, respectively.
The backpropagation of R(L+1) is represented as follows,with ε ensuring numerical stability by avoiding a zero division :We extend Layer-wise Relevance Propagation (LRP)[Bach 15] for use in models with residual connections, specifically developing LRP for ResNet [He 15] models. The extension implemented in our method deﬁnes the calculation
method of LRP for models with residual structures. Consequently, this approach is generally applicable to models
possessing residual blocks.
We propose a calculation method for LRP in models featuring skip connections. We also introduce C 3 -LRP (Contour and Component Choice), that take the most signiﬁcant regions from the generated attention areas based on
the greatest contour.
ResNet50 consists of Convolution layers, Batch Normalization layers, Max Pooling layers, 16 Bottleneck layers, aGlobal Average Pooling (GAP) layer, and a Linear layer.
Each Bottleneck consists of three layers – a 1 × 1 convolution layer for dimensionality reduction, a 3 × 3 convolution
for processing, and another 1 × 1 convolution to restore dimensions. We treat each bottleneck block as a single dense
layer with its unique LRP rule. The Relevance score canthen be computed via backpropagation by applying an LRP
rule for each encountered layer.
Figure 2 shows a schematic diagram of the calculation∂z ′R (zij ) =
L

uv
U ∑
V z
∑
ij ∂z
ij

u

v

′ +ε
zuv

′
RL+1 (zuv)(1)

For Convolution layers, Batch Normalization layers, Max
Pooling layers and the GAP, the same formula is applied.
For the Linear layer, the Relevance of a unit j in this layer
L is computed as follows:∑
wkj hj∑RL (hj ) =RL+1 (h′k )(2)w′
kj ′ hj ′ + εjkwhere hj , h′k and wkj denote activation of the j-th unit, activation of the k-th unit and the weight connecting unit j in
the Linear layer to the unit k of the layer L+1, respectively.

3.2

Obtaining the ﬁnal heatmap with C 3

To prevent the relevance map to focus on inappropriate areas such as the background, we introduce C 3 to extract the most noteworthy area in order to obtain the ﬁnale
heatmap αC 3 .
First, R(x) is reduced to a 28 × 28 αCR to remove ﬁnenoise and unnecessary information. Then, based on thelargest contour, a binary mask αC1 is created. As with C1C

2

Table 1: Quantitative comparison results between methods.
MethodAcc ↑Insertion ↑Deletion ↓ID score ↑RISE [Petsiuk 18]
GradCAM [Selvaraju 17]
LRP [Bach 15]
ABN [Fukui 19]
Ours

0.815 ± 0.001
0.815 ± 0.001
0.815 ± 0.001
0.642 ± 0.009
0.815 ± 0.001

0.371 ± 0.015
0.466 ± 0.019
0.063 ± 0.007
0.282 ± 0.052
0.685 ± 0.015

0.043 ± 0.004
0.156 ± 0.008
0.051 ± 0.006
0.075 ± 0.011
0.017 ± 0.001

0.328 ± 0.004
0.310 ± 0.020
0.011 ± 0.001
0.207 ± 0.054
0.668 ± 0.015

(a)

(b)

(c)
(d)
GradCAM
RISE
LRP[Sel[Bach
Original [Petsiuk
varaju
18]
15]
17]

(e)

(f)

ABN
[Fukui19]

Ours

images for data augmentation. The training, validation,and test sets were used for parameter training, hyperparameter validation, and evaluation, respectively. We used
the SGD optimizer with a learning rate of 1.0 × 10−3 anda batch size of 32.
The number of parameters and the number of multiplyaccumulate operations for the proposed method were 23.9M
and 65.4G, respectively. For training, a GeForce3090 GPU
and an Intel Core i9 processor were used. It took approximately 2 hours to train a ResNet50 model on the CUB-2002011 dataset. The inference time was approximately 0.1s.
We stopped the training if the loss on the validation set did
not improve for six consecutive epochs.

4.3[Iida 23], the largest connected component is calculated andanother binary mask αC2 is created. These two masks arethen combined by taking the bitwise AND operation of thetwo masks to obtain αC3 :αC 3 = αC1 ∧ αC2(3)

The ﬁnal mask αC 3 is obtained by multiplying the originalmask with the combined mask converted to a boolean array to keep the original pixel values where both the largest
contour or the largest connected component are present:
αC 3 = αC R ⊙ αC 3

(4)Finally, αC 3 is expanded to w × h to obtain the ﬁnal relevance map α.

4.

Experiment

4.1

DatasetThe Caltech-UCSD Birds-200-2011 (CUB-200-2011)
dataset [Wah 11] was used for experimental evaluation. According to [Wah 11], a list of 278 bird species was compiled
from an online ﬁeld guide. Next, all images were downloaded from the corresponding Wikipedia page or fed toFlickr as query terms, and up to 40 images were downloaded for each species. We used the CUB-200-2011 dataset
because it is a standard dataset for visual explanation generation tasks.
It contains 11,788 images of 200 subcategories belonging
to birds. According to the standard setting [Wah 11], thedataset was divided with 5,794 images for training, 200 for
validation and 5,794 for testing.

4.2

Quantitative results

Table 1 presents the quantitative results of the comparison between the baseline methods and the proposed
method. Five experiments were conducted for each method.
Additionally, the bold values in Table 1 represent the bestvalues. As the baseline methods, we used RISE [Petsiuk18], GradCAM [Selvaraju 17], Layer-wise Relevance Propagation (LRP) [Bach 15], and Attention Branch Network(ABN) [Fukui 19]. The reason for choosing ABN as abaseline method is because it is a standard method thatuses ResNet as the backbone network. On the other hand,
RISE, GradCAM, and LRP were chosen as baseline methods because they are standard among methods applicableto generic models.
In this experiment, the evaluation metrics used were
Accuracy, Insertion score, Deletion score, and InsertionDeletion score (ID score). Additionally, as it is the moststandard metric, we used ID score as the primary evaluation metric. The Insertion score and Deletion score arecalculated as the Area Under Curve (AUC) of the Insertion and Deletion curves, respectively. Additionally, theID score is deﬁned as the diﬀerence between the Insertion
score and Deletion score. The Insertion and Deletion curves
represent the changes in prediction when important regions
based on the ﬁnal relevance map α are inserted or deleted,
respectively. The details are deﬁned as follows. First, sortthe elements of α in descending order as αi1 ,j1 , αi2 ,j2 , …
, αiw ,ih , and deﬁne the sets An , in , dn as follows:

Figure 4: Qualitative Results.

An = {(ik , jk ) | k ≤ n}{(xij , 0) if (i, j) ∈ An
(in , dn ) =(0, xij ) if (i, j) ∈
/ An

(5)
(6)

Here, n represents the number of pixels to insert or delete.
When in and dn are input into the model, we denote theoutputs as y (ins,n) and y (del,n) , respectively. The curves(ins,n)(del,n)plotted for n, yC
and yCare the Insertion and Dele-

Experiment settings

The input images were resized to 224×224. During training, we ﬂipped, rotated, cropped and changed brightness to3

Table 2: Quantitative results of the Ablation Study.
Model

MethodInsertion ↑Deletion ↓ID score ↑

(i)
(ii)
(iii)

None
C1C [Iida 23]
C30.250 ± 0.017
0.642 ± 0.009
0.685 ± 0.015

0.018 ± 0.001
0.042 ± 0.004
0.017 ± 0.001

0.232 ± 0.017
0.600 ± 0.006
0.668 ± 0.015• We introduced the C 3 -LRP, which improves the quality of explanations by selecting the most noteworthy
area based on the generated attention regions.

tion curves, respectively, where C represents the class towhich x belongs.
From Table 1, in the primary metric, the ID score, RISE,
GradCAM, LRP, ABN, and the proposed method scored0.328, 0.310, 0.011, 0.140, and 0.668 respectively. The proposed method exceeded the highest among baselines, RISE,
by 0.340 and achieved both the best performances for Insertion and Deletion scores. The performance diﬀerence inthe primary metric, the ID score, and the Insertion score
was statistically signiﬁcant (p < 0.05).

In the standard evaluation metrics for this task, such as
the Insertion score and ID score, the proposed method outperformed the baseline methods.

4.4

References

Acknowledgment
This work was partially supported by JSPS KAKENHI Grant
Number 23H03478, JST CREST, and NEDO.

Qualitative results

Figure 4 shows the qualitative results. Column (a) displays the original images, while columns (b)-(e) show the
explanations generated by the baseline methods overlaid on
the original images, and column (f) represents the results
generated by the proposed method. From column (b) in
Figure 4, the explanations generated by RISE strongly focus on speciﬁc parts like the bird’s eyes and feathers, but
fail to focus on the bird as a whole. Furthermore, column(c) shows that the explanations generated by GradCAM
have attention regions that encompass the whole bird, butalso focus on the background surrounding the bird. Next,
from column (d), the explanations generated by LRP are
inappropriate, as most regions have equal attention. Column (e) shows that the explanations generated by ABN are
spotty and of low quality. On the other hand, column (f)shows that the proposed method focuses in detail on theentire bird, especially on the eyes, and has a low degree ofattention to the background, thus generating appropriate
explanations.

4.5[Bach 15] Bach, S., et al.: On Pixel-Wise Explanations for NonLinear Classiﬁer Decisions by Layer-Wise Relevance Propagation, PLOS ONE, Vol. 10, No. 7, pp. 1–46 (2015)[Fukui 19] Fukui, H., Hirakawa, T., et al.: Attention Branch
Network: Learning of Attention Mechanism for Visual Explanation, in CVPR, pp. 10705–10714 (2019)[He 15] He, K., Zhang, X., Ren, S., and Sun, J.: Deep Residual
Learning for Image Recognition, in CVPR, pp. 770–778 (2015)[Iida 23] Iida, T., Otsuki, S., et al.: Visual Explanation Generation for Road Damage Classiﬁcation by Using Layer-wise Relevance Propagation for Branch Networks, in SIG-AM (2023)[Jacovi 23] Jacovi, A., Schuﬀ, H., et al.: Neighboring Words
Aﬀect Human Interpretation of Saliency Explanations, arXiv
preprint arXiv:2305.02679 (2023)[Madiaga 23] Madiaga, : Artiﬁcial intelligence act (2023)[Molnar 20] Molnar, C., Casalicchio, G., et al.: Interpretable
Machine Learning - A Brief History, State-of-the-Art, andChallenges, in ECML PKDD, pp. 417–431 (2020)[Petsiuk 18] Petsiuk, V., Das, A., and Saenko, K.: RISE: Randomized Input Sampling for Explanation of Black-box Models,in BMVC, pp. 151–164 (2018)

Ablation study

We investigated the contribution of C 3 by removing it(Model i) or replacing it with C1C [Iida 23] (Model ii).
From Table 2, the ID score in model (i) was 0.600, which
is a decrease of 0.436 compared to model (iii). Also, theID score in model (ii) was 0.600, which is a decrease of
0.068 compared to model (iii). This suggests that C 3 was
eﬀective to exclude the background and areas not directly
related to classiﬁcation, thereby promoting the generationof high-quality explanations.

5.

[Ribeiro 16] Ribeiro, M., Singh, S., et al.: ”Why Should I Trust
You?”: Explaining the Predictions of Any Classiﬁer, in KDD,
pp. 1135–1144 (2016)[Selvaraju 17] Selvaraju, R., et al.: Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization, in
ICCV, pp. 618–626 (2017)[Shrikumar 17] Shrikumar, A., et al.: Learning Important Features Through Propagating Activation Diﬀerences, in PMLR,
Vol. 70, pp. 3145–3153 (2017)[Wah 11] Wah, C., Branson, S., Welinder, P., and others., :The Caltech-UCSD Birds-200-2011 Dataset, Technical Report
CNS-TR-2011-001, California Institute of Technology (2011)

Conclusion

This study dealt with the generation of visual explanations for the rationale in the multi-class classiﬁcation problem of predicting the bird species from an image. The contributions of this research are as follows:• We proposed a method for calculating Layer-wise Relevance Propagation (LRP) in models with residual connections, taking the example of ResNet.

4