<figure><img src="image_1.png" /></figure>

HAL
open science

# AttentNet: Fully Convolutional 3D Attention for Lung Nodule Detection

Majedaldein Almahasneh, Xianghua Xie, Adeline Paiement

► To cite this version:

Majedaldein Almahasneh, Xianghua Xie, Adeline Paiement. AttentNet: Fully Convolutional 3D Attention for Lung Nodule Detection. SN Computer Science, 2025, 6 (3), pp.292. 10.1007/s42979-025-03799-4. hal-05319437

HAL Id: hal-05319437

https://hal.science/hal-05319437v1

Submitted on 17 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

# AttentNet: Fully Convolutional 3D Attention for Lung Nodule Detection

Majedaldein Almahasneh<fnref n="1" /> Ⓔ · Xianghua Xie<fnref n="2" /> Ⓔ · Adeline Paiement<fnref n="3" /> Ⓔ

Received: 16 July 2024 / Accepted: 12 February 2025

© The Author(s) 2025

## Abstract

Motivated by the increasing popularity and success of attention mechanisms, we observe that popular convolutional attention models like Squeeze-and-Excite (SE) and Convolutional Block Attention Module (CBAM) rely on expensive multi-layer perceptron (MLP) layers. These MLP layers significantly increase computational complexity, making such models less applicable to 3D image contexts, where the data dimensionality and computational costs are inherently higher. In 3D medical imaging, such as pulmonary CT scans, efficient processing is crucial due to the large volume of data. Traditional 2D attention mechanisms, when generalized to 3D, increase the computational load, creating a demand for more efficient attention mechanisms that can operate effectively within a 3D tasks. In this work, we investigate the possibility of incorporating fully convolutional attention within 3D context. Particularly, we present two 3D fully convolutional attention blocks in which we demonstrate effectiveness within the 3D context. We demonstrate effectiveness of the proposed method using the pulmonary CT scans for 3D lung nodule detection. Building on existing 3D nodule detection methods, we present AttentNet, an automated lung nodule detection framework from CT images that performs detection as an ensemble of two stages, candidate proposal stage (Region Proposal Network), in which a high number of candidates is produced, and a false positive reduction stage to reduce the number of false alarms. Throughout our work, we compare the proposed 3D convolutional attention blocks to a number of popular 2D convolutional attention mechanisms by generalizing them to 3D modules, including SE units and CBAM channel attention and spatial attention units. We also compare these to Transformer self-attention units. For the False positive reduction stage, we incorporate a joint analysis approach that considers the variable nodule morphology by aggregating spatial information from different contextual levels. We use LUNA-16 lung nodule detection dataset to demonstrate the benefits of using the proposed fully convolutional attention blocks comparing to baseline popular lung nodule detection methods where no attention is used. It is worth noting that our work does not aim at achieving state-of-the-art results in the lung nodule detection task, rather to demonstrate the benefits of incorporating fully convolutional attention within a 3D context.

**Keywords** Convolutional attention · 3D image analysis · Pulmonary nodule detection · False positive reduction · Joint analysis

## Introduction

The strength of CNNs manifests in their rich representational power and ability in embedding spatial and cross-channel information. However, due to their inherent locality, CNNs suffer in modelling long-range relations. Moreover, performance of CNNs may be prone to network factors, such as depth and width [18, 24, 58], and data characteristics such as target structures with high morphological variance (e.g. texture, shape, and size). To combat these limitations and enhance the performance of CNNs, research efforts has been directed towards attention mechanisms [11, 19, 56].

<footnote>Majedaldein Almahasneh
m.almahasneh@lboro.ac.uk</footnote>

<footnote n="1">¹ Department of Computer Science, Loughborough University, Loughborough, UK</footnote>

<footnote n="2">² Department of Computer Science, Swansea University, Swansea, UK</footnote>

<footnote n="3">³ Université de Toulon, Aix Marseille Univ, CNRS, LIS, Marseille, France</footnote>
Attention simulates the cognitive process of selective focus on features with high relevance to a task while excluding others. Incorporating such methods can elevate CNNs performance by automatically learning to focus on important structures associated with a given objective.

The introduction of data collections (e.g. LUNA16 dataset [43]) attracted more interest to the pulmonary nodule detection problem. It also made the integration of DL tools possible, however, training robust DNNs requires huge amounts of labelled samples, which remains a challenge when dealing with pulmonary images. Indeed, preparing such datasets is very resource intensive due to the 3D nature of the CT images and the high complexity of the task. Moreover, available DL methods are designed for generic object detection tasks and typically aim at analysing 2D and RGB images. They are not suited to be directly applied to the medical imaging domain (e.g. 3D CT images) due to the different nature of the data and the complexity of the task. Additionally, unlike common computer vision problems, the sparse nature of lung nodules within the lung region, along with the prevailing class imbalance in the pulmonary data, make the detection task more challenging. Therefore, this scenario requires designing a specialised DL approach.

Computer aided detection (CAD) are systems designed to assist radiologists in detecting lung nodules. Incorporating such systems has been clinically proven to decrease observational oversights (i.e. false negative rate) while significantly reducing the reading time required per scan and retaining a consistent quality [2, 33]. Indeed, a number of studies have demonstrated that CAD systems were able to detect nodules that were originally missed by experienced radiologists [2, 23].

Generally, CAD systems consists of two consecutive stages, a candidate proposal stage, in which a region proposal network (RPN) is used to propose candidate locations at a high sensitivity and typically on the account of high false positive rates, and a false positive reduction stage to minimise the number of the false alarms and produce the final set of predictions [43].

Several deep neural networks (DNN) has been proposed for generic object detection (e.g. YOLO, SSD, R-FCN, Corr-Net, and Faster RCNN). These may be divided into two groups: two stage detectors, in which objects are detected in two steps, region proposal and classification, and single stage methods, where a DNN is trained to regress object locations and classes simultaneously. In general, two stage detectors (e.g. Faster RCNN) can achieve higher accuracy in contrast to single stage detectors [21, 48].

In this work, we investigate the possibilities of incorporating DL methods to solve the pulmonary detection problem. Specifically, we utilise a two stage (candidate proposal and false positive reduction) detector based on Faster RCNN [39]. We adopt a 3D encoder-decoder structure for

the backbone network of our candidate proposal stage. Moreover, motivated by the high relevance of nodule morphology and the sparse nature of nodule locations, as well as the success of attention mechanisms in medical imaging related tasks (e.g. detection [29, 35, 42, 49] and segmentation [6, 54, 64]), we evaluate different state-of-the-art attention mechanisms and propose two fully convolutional attention blocks that demonstrate potential within the pulmonary nodule detection task. We also utilise a cross-sectional augmentation approach to battle the limited availability of annotated samples. Additionally, in line with the objective of the candidate proposal stage, we exploit a testing time augmentation strategy that further enhances the sensitivity of the trained model. To tackle the class imbalance problem present in the lung data, we adopt focal loss [28] and exploit an online hard example mining technique [46].

For our False positive reduction stage, in addition to exploiting attention techniques, and inspired by the variant nodule sizes (See Fig. 1), we adopt a joint analysis based approach in which information from different contextual levels is aggregated and analysed simultaneously. We also propose a zoom-in convolutional path to assist the network in capturing multi-scale spatial embeddings.

More formally, our contributions may be summarised as follows:

1. We propose two fully convolutional attention blocks in which we incorporate 3D features to infer cross-channel and cross-sectional spatial correlations. We demonstrate the potential of our attention approach within the lung nodule detection task. We also carry out extensive experiments using different state-of-the-art attention mechanisms and compare their performance against our proposed methods. We evaluate our attention blocks within a 3D lung nodule detection framework that

<figure><img src="image_5.png" /><figcaption>**Fig. 1** Distribution of nodule diameters in LUNA16 [43] dataset. The average nodule diameter is 8.32 mm</figcaption></figure>
detects nodules in two stages, candidate proposal and false positive reduction. We use on the publicly available dataset, LUNA16, and compare the performance against existing lung nodule detection baseline methods where no attention is used.

2. We employ a joint analysis based approach that elevates the performance of the false positive reduction stage by exploiting different levels of spatial contextual information simultaneously. We present a zoom-in convolutional block that allows the network learning information from different scales and therefore enhance the final prediction.

In the remainder of this article, Sect. Related work discusses related literature. Section Methodology presents the details of our proposed framework, Sect. Experiments discusses datasets, pre-processing, experiments and results. Last, Sect. Conclusion concludes our work.

# Related Work

## Attention-Mechanisms

Attention in DL may be split into three main categories, channel-wise, spatial-wise, and global attention. The objective of channel-wise attention is refine cross-channel embeddings by directly modelling the correlations between the different channels. Hu et al. proposed squeeze and excitation units, multi-layer perceptron (MLP) networks that generate attention maps using a dimensionality reduced descriptor in which cross-channel spatial information is aggregated. The resulting attention maps are then used to recalibrate the channels of the convolutional feature [19].

$$A_c = \sigma(MLP(AvgPool(F)))^{\mathbb{R} \in C_{x1x1}} \quad (1)$$

where $A_c$ is the resulting channel attention map. F represents an intermediate convolutional feature map of size C x H x W. AvgPool represents adaptive average pooling. MLP is a multi-layer perception network and $\sigma(\cdot)$ is a sigmoid activation function. Subsequently, a refined feature map $F'$ is computed using element-wise multiplication such that $F' = F \otimes A_c$.

Thereafter, Woo et al. proposed convolutional block attention module (CBAM), in which they extend the concept of squeeze and excitation by adding a subsequent spatial attention gate [56]. Spatial attention utilises the inter-spatial relationship within the convolutional features to assist the CNN learning where to attend within a feature map. This was achieved by projecting the channels of the convolutional feature into a 2D map embedding using channel-wise pooling operators, the resulting map was then passed through a

convolutional layer followed by a sigmoid function to generate the final spatial attention map. Moreover, in contrast to squeeze and excitation units, [56] demonstrate that incorporating max pooling in addition to average pooling can provide complementary clues that can enhance the overall attention performance. Accordingly, cross-channel and spatial attention can be described as follows:

$$A_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))^{\mathbb{R} \in C_{x1x1}} \quad (2)$$

$$A_s = \sigma(Conv2D_{3x3}(AvgPool(F) + MaxPool(F)))^{\mathbb{R} \in C_{xHxW}} \quad (3)$$

where $A_c$ and $A_s$ are the channel and spatial attention maps, respectively. F represents an intermediate convolutional feature map of size C x H x W. AvgPool and MaxPool are adaptive average and max pooling layers. MLP is a multi-layer perception network. Conv2D$_{3x3}$ is a convolutional layer with kernel size of 3 x 3. $\sigma(\cdot)$ is a sigmoid activation function. Note that in the case of spatial attention, pooling operations are performed along the channel axis. Consequently, a refined feature map $F'$ is computed using element-wise multiplication of the intermediate feature map F and the resulting attention map M, where $M \in \{A_c, A_s\}$. Moreover, [56] shows that combining both attention approaches by applying them in a consecutive order (i.e. channel-wise followed by spatial attention) can further enhance the overall performance in contrast to using either of the attention approaches individually, or using other combination schemes (e.g. spatial followed by channel-wise attention, or by applying both approaches in parallel). Incorporating channel and spatial attention in CBAM can improve the representational capabilities of CNNs, and therefore enhances their performance [56]. This is in line with the findings of Sun et al. [49] (pulmonary nodule classification in CT images), Lu et al. [29] (pulmonary tuberculosis detection in CT images), Nawshad et al. [35] (COVID-19 detection in X-ray images), Sangeroki et al. [42] (thoracic disease detection in chest X-rays), and Park et al. [36] (generic object detection). In this work, we evaluate channel and spatial attention mechanisms for both the candidate proposal and the false positive reduction stage. We also present two fully convolutional attention blocks based on channel and spatial attention and demonstrate their effectiveness for the pulmonary nodule detection task in contrast to state-of-the-art attention approaches.

Global attention on the other hand, aims to model long-range correlations and inter-dependencies between arbitrary positions. Vaswani et al. proposed Transformer networks based on multi-headed self-attention for sequence-to-sequence tasks. An input vector is used in three ways, query, key, and value. Accordingly, attention is expressed as a mapping between a query and a set of key and value pairs, to an output vector, by finding the weighted sum of the values
using weights that are found by a compatibility function of the query and the corresponding key [52]. More formally, the overall attention process is described as follows:

$$ \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_i)W^O \\ \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) \quad (4) $$

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

where Q, K, and V represent queries, keys, and values, and their correspondent learnable parameters W^Q, W^K, and W^V, respectively. W^O represents a learnable linear projection process.

Dosovitskiy et al. generalised this concept to computer vision tasks by splitting an image into a sequence of vectorised patches that can then be managed by a pure Transformer [11]. While Transformers demonstrate a great potential in computer vision tasks, they rely on heavy pre-training and are difficult to scale to large inputs due to their computational cost. [6, 11, 54]. Moreover, vision transformers suffer when modelling local structures due to the tokenisation of input images [61]. To tackle these limitations, a number of studies proposed using a hybrid architecture that combines both, CNNs for their spatial representation power and their relatively low computational cost, along with Transformers for their ability in modelling long-range dependencies. A CNN encoder was used to extract embeddings from 3D MRI (magnetic resonance imaging) images, the down-sampled features were then fed into a pure Transformer unit in which output was passed into a decoder CNN to perform object segmentation [54]. Other works used a similar approach to perform segmentation in 2D images [6, 64]. In this work, due to the recent success and the increasing interest in vision transformers, we evaluate Transformer [54] in the pulmonary nodule detection task and compare it against our proposed attention approach.

## Pulmonary Nodule Detection

The increasing amounts of pulmonary data collections along with the advances in convolutional neural networks have attracted research interest to the pulmonary nodule localization and classification problem. Generally, most existing works deploy two stage CNN based detectors and investigate different levels of feature dimensionality, e.g, 2D, pseudo 3D (cross-sectional 2D planes), and 3D [40, 43]. Berens et al. [4] proposed a two stage detector based on a 2D U-Net [41] for region proposal, followed by a false alarm reduction CNN that takes three orthogonal 2D slices as an input. They evaluate their approach on LUNA16 dataset ([43]), and conclude that incorporating 3D information may be a good direction to improve on the performance of their 2D

approach. Indeed, this was demonstrated in [45], where the authors compare 2D, pseudo 3D, and 3D CNNs for the lung nodule detection task and find that using 3D kernels significantly improves the performance. Similarly, Riquelme et al. demonstrate in their extensive survey on DL for lung nodule detection [40], that the best performing methods are the ones that incorporate 3D information in their approaches. This shows that 2D based approaches fail to fully exploit the inherent 3D nature of the nodule structure.

Liao et al. proposed a 3D region proposal network based on Faster RCNN [39] for the candidate proposal stage. The top five suspicious proposals are passed into a subsequent CNN in which a modified or-gate [37] is employed to predict the final score [27]. Zhu et al. proposed a similar 3D CNN based on dual path networks (DPN) [8] for nodule detection along with a gradient boosting machine (GBM) [13] for the final classification stage [65]. They deploy an encoder-decoder design for their detection network to allow learning nodule features from different semantic levels. They show that by using grouped convolutions [58] and dense convolutional connections [20] in their DPN, the network was able to detect more nodules while decreasing the computational overhead.

Building on [27] and [65, 26] adopted squeeze and excitation paths ([19]) to their detection CNN to assist the network learn inter-dependencies within the extracted features. This encoder-decoder design was also exploited by [7] and [51], and have demonstrated great potential within the lung nodule detection task. Hence, we adopt this 3D encoder-decoder structure for our candidate proposal stage. Additionally, both [7] and [26] adopt focal loss [28] to tackle the class imbalance in the pulmonary images and have empirically demonstrated an enhanced performance in the nodule detection task. We therefore adopt focal loss in our framework.

Moreover, both 3DDUALRPN [59] and APNDCTDCNN [10] used RPN (Region Proposal Network) as the foundation of their methods. [10] incorporated a 2D Faster RCNN detector to propose candidates from axial slices in the first stage, and employed a 3D CNN for the false positive reduction. While 3DDUALRPN [59] proposed a dual path network consisting two U-shaped sub-networks, ResNet50 based RPN and DenseNet based detector, candidates from the two networks are combined using a label fusion method based on non maximum suppression (NMS) to produce the final detection. Both methods demonstrate excellent results on the LUNA16 dataset; however, they are relatively complex and require substantial computational resources as we demonstrate in Sect. Experiments.

More recently, several studies utilized attention mechanisms for lung nodule and other medical imaging based detection and segmentation tasks. [31] introduced an improved V-Net with pixel threshold based background filtering and convolutional spatial attention for lung nodule
segmentation. [55] proposed incorporating multi-headed attention modules in combination with skip-connections for lung cancer segmentation, aiming to enhance feature fusion and improving the overall performance. Both methods utilize attention to refine the extracted feature maps, improving segmentation performance. In the same line, both [30] and [5] proposed using multi-headed attention blocks at the bottleneck of a UNet-like CNNs for medical image segmentation, aiming at capturing long-range dependencies at reduced feature map resolution, lowering the computational requirement associated with self-attention. These methods and others, e.g., [1], demonstrate promising results when employing attention methods within their respective applications. Generally, these works target segmentation applications or generalise 2D attention mechanisms into 3D to cope with the volumetric nature of the lung images. Driven by these observations, in this study, we assess the role of self-attention in the context of nodule detection. We also propose and examine fully convolutional blocks tailored for 3D characteristics of the data, while retaining a relatively low computational cost and preserving the 3D spatial information.

For the false positive reduction stage, [38] proposed using an ensemble of five CNNs to analyse nodules using different levels of context. Each CNN was trained using a unique crop size (i.e. level of context) and the final score was defined as a function of all the predictions. In the same line, [49] proposed a more efficient approach by jointly analysing nodules on different contextual levels using a single CNN. Both experiments demonstrate that incorporating contextual information leads to an enhanced prediction. This is expected since pulmonary nodules have highly variable morphology and size (see Fig. 1). Thus, we employ a joint analysis approach in our false positive reduction stage that aggregates multi contextual level features.

## Methodology

Our proposed framework, AttentNet, detects pulmonary nodules in two stages: 1) candidate proposal, in which we adopt a 3D encoder-decoder CNN to propose suspicious locations of nodules at high sensitivity, 2) and a subsequent false positive reduction stage to reduce the number of false alarms. For the candidate proposal stage, we incorporate fully convolutional attention blocks to efficiently assist the network in focusing on informative features along both, inter-channel and spatial axes. For the false positive reduction stage, we exploit nodule morphology by jointly analysing inputs of different contextual levels. Additionally, we use a zoom-in convolutional paths to allow the network pick fine details from different feature scales. During inference time, detections are aggregated from different transformations of the input image. The final classification score is found by

ensembling the models of both stages. The two detection stages are optimised separately, each according to its correspondent objective. An overview of our proposed framework is presented in Fig. 2.

## Candidate Proposal Stage (Region Proposal Network)

We incorporate an encoder-decoder region proposal network (RPN) as the backbone of our candidate proposal stage. An input image is first processed by two consecutive 3D convolutional layers of 32 channels each, using 3 x 3 x 3 kernels, followed by a 3D max pooling layer to reduce the spatial dimension of the resulting feature map by a factor of 2. The resulting volumetric feature is then passed into a sequence of four residual blocks [18] that consist of 2, 3, 3 and 3 residual units, respectively. Each of the residual blocks is followed by a max pooling layer. The resulting embeddings are then up-sampled by two subsequent deconvolutional layers using 2 x 2 x 2 kernels and strides of 2. Each up-sampling layer is followed by a residual block (consisting of 3 residual units each). To avoid overfitting, we regularise the network using 3 dropout layers in the encoder part and one dropout layer in the last decoder layer. We use 0.3 dropout rate for all dropout layers.

It is worth noting that at an effective image resolution, even a single image can exhaust the GPU memory. To address this constraint and allow processing the 3D inputs at an effective resolution and a representative training batch size, we utilise grouped convolutions [58] within the residual units. Grouped convolution allows efficient model parallelism by splitting both, the feature channels, and the convolutional kernels into a number of groups. Each group of filters processes their correspondent feature channels and produce a part of the output channels. The resulting feature maps are then projected into a new linear space using a subsequent 1 x 1 x 1 convolution. This allows reducing the computational overhead and the number of parameters while retaining the number of features and a consistent performance [24, 58]. Grouped convolution have repeatedly demonstrated effectiveness within the lung nodule detection task [15, 62, 65]. Accordingly, a single residual unit consists of three convolutional layers of 1 x 1 x 1, 3 x 3 x 3, and 1 x 1 x 1 kernel sizes, respectively, with the second layer utilising grouped convolutions of 32 groups [24, 58]. An overview of our candidate proposal backbone network is presented in Fig. 2.

Inspired by the success demonstrated by attention mechanisms in medical imaging related tasks (e.g. [29, 35, 42, 49]), as well as the sparsity of pulmonary nodule locations, and the high relevance of nodule morphology for the nodule detection task, we utilise two attention mechanisms within the building blocks of our backbone network to assist the network in focusing at meaningful and effective embeddings.
<figure><img src="image_6.png" /><figcaption>**Fig. 2** The framework of AttentNet. AttentNet performs pulmonary nodule detection in two stages, candidate proposal, in which we exploit a 3D encoder-decoder network to predict suspicious nodule locations, and a false positive reduction stage in which a 3D CNN is</figcaption></figure>

Specifically, we propose 3D fully convolutional cross-channel and spatial attention blocks for the candidate proposal stage.

Given an intermediate 3D convolutional feature map $F \in \mathbb{R}^{CxD_F \times H_F \times W_F}$, cross channel attention is computed as follows: First, spatial information is aggregated using adaptive average pooling, resulting an embedding $E \in \mathbb{R}^{CxD_E \times H_E \times W_E}$. Here, $C$ represents the channels of the feature map, $D_F, H_F, W_F$ and $D_E, H_E, W_E$ represent the depth, height, and width of $F$ and $E$, respectively. Note that in our preliminary experiments, we test other types of spatial feature aggregation (e.g. adaptive max pooling) and observe no particular benefit within our nodule detection task, therefore we continue using average pooling. Consequently, $E$ is passed into a convolutional layer with 3D kernels of size $D_E \times H_E \times W_E$, and no padding, producing a $C \times 1 \times 1 \times 1$ descriptor. A sigmoid function is then applied to produce the final channel attention map $A_c$, in which channel-wise importance scores

used to extract deep features from the proposed nodules and produce the final prediction. We augment the building blocks of our network with attention units to assist the network in focusing on effective nodule features and therefore produce a more robust detections

are predicted. The feature map $F$ is then recalibrated using element-wise multiplication with the attention map $A$. For our experiment, we set $D_E, H_E, W_E$ to 3, therefore, the overall attention map computation can be summarised as follows:

$$ 
\begin{aligned}
A_c &= \sigma(W(E))^{\mathbb{R}\in C\times1\times1} \\
&= \sigma(Conv3D_{3\times3\times3}(AvgPool3D(F)^{\mathbb{R}\in C\times3\times3\times3}))
\end{aligned} 
\quad (5) 
$$

where $AvgPool3D$ represents adaptive 3D average pooling. $Conv3D_{3\times3\times3}$ is a 3D convolutional layer, the subscript represents the size of the kernels used in the convolutional layer. $\sigma(\cdot)$ is a sigmoid activation function. Note that we continue using these notations for the reminder of the article. Accordingly, the refined feature map is computed as follows:

$$ F' = A_c \otimes F \quad (6) $$

where $\otimes$ denotes element-wise multiplication. Our cross-channel attention block is visualised in Fig. 3
<figure><img src="image_7.png" /><figcaption>Fig. 3 An overview of our proposed 3D fully convolutional cross-channel attention unit within a residual block. As illustrated, our channel attention exploits 3D adaptive pooling to embed spatial information from an intermediate convolutional feature map F, these are passed into a 3D convolutional layer in which output is an attention map A_c of size C x 1 x 1 x 1. This will then be used to adaptively refine the intermediate feature maps inferring channel importance and inter-channel correlations (F'). Here, ⊗ and ⊕ represent element-wise multiplication and addition, respectively. Note that the addition operation represents the residual path in the residual block. The parameter k represents the kernel size used in the convolutional layers</figcaption></figure>

Our channel-wise attention strategy aims to assist the network in effectively learning to focus on informative feature and ignore (down-weight) irrelevant or less informative ones. Other works that incorporate multi-layer perceptrons rely on heavy dimensionality reduction to decrease the computation overhead of the attention network. Particularly, [19, 56] use an adaptive pooling operation to create a C x 1 x 1 x 1 descriptor in which spatial information of the intermediate features is embedded. This is then passed to a multi-layer perceptron where the feature's dimensionality is further reduced by a pre-defined factor (see Eqs. 1 and 2). Note that the size of the spatial descriptor is less proportional to the intermediate embedding when dealing with 3D feature maps (e.g. pulmonary nodules) in contrast to that in 2D based analysis (e.g. [19, 56]), leading to a limited spatial information. We avoid this by replacing the multi-layer perceptron by a fully convolutional network, allowing an efficient use of rich spatial descriptors with higher dimensionality (i.e. C x 3 x 3 x 3).

To perform spatial attention, we adopt a joint analysis approach that integrates spatial information from different image cross-sections (i.e. axial, coronal, and sagittal). First, an intermediate 3D feature map F ∈ ℝ^CxD_FxH_FxW_F is linearly transformed into E ∈ ℝ^{1xD_FxH_FxW_F} using a convolutional layer with kernels of size 1 x 1 x 1:

$$E = \text{Conv}3D_{1 \times 1 \times 1}(F^{\mathbb{R} \in C_x D_F \times H_F \times W_F})^{\mathbb{R} \in 1 \times D_F \times H_F \times W_F} \quad (7)$$

Note that the resulting feature map E is of dimensions 1 x D_F x H_F x W_F, therefore, by reducing the first axis (i.e. D_F x H_F x W_F), E can be processed using 2D convolutions along the D_F axis (this is equivalent to having a 2D

input of D_F channels). The feature map E is now projected into the axial, coronal and sagittal planes. Each of these feature maps is then processed by a unique 2D convolutional network, producing three 2D convolutional feature maps {E_axial, E_coronal, E_sagittal} ∈ ℝ^D_FxH_FxW_F:

$$ 
\begin{align} 
E_{\text{axial}} &= \text{Conv}2D_{3 \times 3}(\text{Axial}(E))^{\mathbb{R} \in D_F \times H_F \times W_F} \nonumber \\
E_{\text{coral}} &= \text{Conv}2D_{3 \times 3}(\text{Coronal}(E))^{\mathbb{R} \in D_F \times H_F \times W_F} \tag{8} \\
E_{\text{sag}} &= \text{Conv}2D_{3 \times 3}(\text{Sagittal}(E))^{\mathbb{R} \in D_F \times H_F \times W_F} \nonumber
\end{align} 
$$

where Axial(·), Coronal(·) and Sagittal(·) are transformation functions that project inputs from axial plane into coronal and sagittal planes, respectively. This is equivalent to applying 2D convolution with different absolute orientations in the 3D space. The resulting embeddings are then spatially aligned, and are combined (concatenated) along a new axis to form a cross-sectional (3D) feature map E_{cs} ∈ ℝ^{3xD_FxH_FxW_F}:

$$E_{cs} = \text{Concat}(E_{\text{axial}}, E_{\text{coral}}, E_{\text{sag}})^{\mathbb{R} \in 3 \times D_F \times H_F \times W_F} \quad (9)$$

Last, E_{cs} is transformed back into ℝ^CxD_FxH_FxW_F using 1 x 1 x 1 convolution. A sigmoid function is then applied to produce the final spatial attention map A_s, in which spatial importance scores are predicted:

$$A_s = \sigma(\text{Conv}3D_{1 \times 1 \times 1}(E_{cs})^{\mathbb{R} \in C_F \times D_F \times H_F \times W_F}) \quad (10)$$

The attention map A_s is used to refine the intermediate feature map using element-wise multiplication:

$$F' = A_s \otimes F \quad (11)$$

The overall proposed spatial attention process is visualised in Fig. 4.

While spatial attention proposed in [56] focuses on capturing spatial correlations within the convolutional features, we argue that the use of element-wise pooling operations along the channel axis, to aggregate spatial information limits the volumetric information when dealing with 3D data. Note that spatial attention in [56] was originally designed for 2D based analysis, see Eq. 3.

Our cross-sectional spatial attention approach not only leverages 2D inter-spatial relations, but also captures 3D information by jointly analysing the three orthogonal planes of the 3D feature map (axial, coronal, and sagittal). This particularly important when managing volumetric images, where target structures may have different visual appearance when viewed in different cross-sectional planes, e.g., pulmonary nodules. See Fig. 5.

Our attention blocks (cross-channel, or spatial attention) can be straightforwardly integrated and jointly trained with any 3D CNN architecture. In line with [56] and [19], we place our proposed attention block prior to
<figure><img src="image_8.png" /><figcaption>**Fig. 4** An overview of our proposed 3D fully convolutional inter-spatial attention unit within a residual block. Our spatial attention takes as input an intermediate 3D feature map F of C channels, projects it into a 1 channel feature map (using 1 x 1 x 1 convolutions) that is then transformed into three orthogonal planes (axial, coronal, and sagittal). Each of the resulting features is then processed by a unique 2D convolutional layer to learn cross-sectional spatial representations. The</figcaption></figure>

resulting feature maps are then spatially aligned, aggregated by concatenation, and are linearly projected back to C 3D channels in which we use to infer cross-sectional spatial attention $A_s$. Intermediate feature maps are adaptively refined (F') using element-wise multiplication (⊗ in the figure). Here, ⊕ represent element-wise addition used in the residual path of the residual block. The parameters k and s represent the kernel size and the stride used in the convolutional layers

<figure><img src="image_9.png" /><figcaption>**Fig. 5** Pulmonary nodules viewed in different cross-sectional planes: axial (top), coronal (middle), and sagittal (bottom)</figcaption></figure>

the residual path of a residual unit. In our experiment, we individually evaluate the benefits of both, channel and spatial attention, within the pulmonary nodule detection task. We further evaluate the impact of incorporating both attention techniques in combination. More details are presented in Sect. Experiments.

To perform the final detection, the output of the feature extraction network is passed into two parallel convolutional layers, a classification and a regression layer, to predict classes and nodule coordinates for each position in the feature map, respectively. In line with [26, 51, 65], we train our network with 3 reference anchor sizes, 5, 10, and 20, set based on the nodule size distribution (see Fig. 1). An anchor is considered to be positive if it has an intersection over union (IoU) ≥ 0.5, and negative if IoU < 0.2. The network is trained according to the combined loss function:

$$L = \lambda L_{cls} + p^* L_{reg} \qquad (12)$$

where $L_{cls}$ and $L_{reg}$ are the classification and location regression losses, respectively. $\lambda$ is a balancing operator and is set to 1 in our experiment. $p^* \in \{1, 0\}$ denoting positive and negative anchors, respectively. Similar to [7, 26], we adopt a focal cross-entropy loss [28] for the nodule classification task:

$$L_{cls}(p_t) = -\alpha(1 - p_t)^\gamma \log(p_t) \qquad (13)$$

here, considering an anchor's predicted probability $p$, $p_t = p$ if the ground-truth label is positive, and $p_t = (1 - p)$ otherwise. $\gamma$ is a modulating parameter. Well classified samples (i.e. $p_t \to 1$) cause the modulating term to approach 0, down-weighting their loss values, and vice versa for hard examples. Note that $\alpha$ acts as a class balancing parameter. We find that setting $\gamma$ and $\alpha$ to 2 and 0.5, respectively, provides a favourable balanced performance. In line with the essence of attention, focal loss assists the network in focusing on more informative samples and therefore, structures. This is particularly important when managing class imbalanced data, in this case, pulmonary nodule images.

Moreover, we use smooth L1 loss [39] for location regression task:

$$L_{reg}(t^*, t) = \begin{cases} |t^* - t| & \text{if } |t^* - t| > 1 \\ (t^* - t)^2 & \text{otherwise} \end{cases} \qquad (14)$$
given that $t$ and $t^*$ are vectors representing the relative coordinates of a nodule location in, respectively, the prediction space and ground-truth:

$$t_i = \left( \frac{x - x_a}{d_a}, \frac{y - y_a}{d_a}, \frac{z - z_a}{d_a}, \log \frac{d}{d_a} \right) \qquad (15)$$

where $(x, y, z, d)$ and $(x_a, y_a, z_a, d_a)$ are the predicted nodule centre and diameter, and the coordinates of an anchor i, respectively. Similarly, the ground-truth relative coordinates are defined as $t_i^*$ using the original nodule coordinates $(x^*, y^*, z^*, d^*)$.

To further reduce the class imbalance impact, we adopt an online hard negative mining strategy [46]. During training, $N$ negative samples with the highest nodule probability are selected to be used in the computation of the loss. Remaining samples are ignored and do not contribute to the loss. Here, we set $N$ to 2. Hard negative mining has repeatedly demonstrated usefulness in the nodule detection task [7, 26, 27, 51, 65].

For the network's activation function, we use a modified rectified linear unit (ReLU). ReLU [14] functions have gained popularity due to their simplicity and robustness. This manifests in their ability of preserving the gradient flow in the positive input range. Since the linear portion of ReLU doesn't saturate (i.e. unbounded), it allows gradients on active neurons to remain proportional to their activation. Moreover, in the negative input range, ReLU promotes network sparsity by setting the activation value to zero [14]. However, this becomes a problem when many inputs have negative values, leading to a degraded gradient flow in the backpropagation process, and therefore, no learning. To avoid this problem, we simply modify ReLU such that it generates small negative outputs when the input lies in the negative range. Particularly, we incorporate a hyperbolic tangent function for the negative range of ReLU. Given a ReLU function $f(x) = max(0, x)$, our modified ReLU can be formulated as follows:

$$g(x) = \begin{cases} x & \text{if } x > 0 \\ \tanh(x) & \text{otherwise} \end{cases} \qquad (16)$$

where $x$ is the input of the activation function. The idea of this modification is to preserve the linear characteristic of ReLU for positive input values, while improving the gradient flow for negative inputs. Moreover, unlike pure ReLU, the modified ReLU is smooth around the origin ($x=0$), promoting a faster learning process. A visual comparison of both activations is presented in Fig. 6. In Sect. Experiments, we empirically demonstrate the benefits of our proposed activation and compare it to pure ReLU activation. We also compare the performance of the proposed activation function against commonly used ReLU alternatives, Leaky ReLU [32] and ELU [9], in which the negative gradient flow is

<figure><img src="image_10.png" /><figcaption>Fig. 6 Visual comparison of ReLU, Leaky ReLU, ELU, and our proposed modified ReLU activation (top), and their correspondent derivatives (bottom). The smooth gradient flow observed in the modified ReLU activation promotes faster learning comparing to other activation functions under comparison. The modified ReLU activation allows small outputs for inputs in the negative range, in a bounded manner, promoting parameter regularization, reducing the risk of overfitting</figcaption></figure>

enhanced by allowing outputs for input values that lie within the negative range of ReLU. It is worth noting that Leaky ReLU has been widely used within the pulmonary nodule detection task to reduce the risk of the dying neurons [16, 22, 50, 53, 60], we demonstrate in our experiment that our modified activation can outperform ReLU, Leaky ReLU, and ELU activations in the nodule detection task.

Furthermore, we employ a testing time augmentation (TTA) strategy in which an input image is orientated along axial, coronal, and sagittal cross-sections. Each of the resulting images is then processed by the network to predict candidate locations. Results from all cross-sections are then aggregated and used to form the final set of predictions. It is worth noting that these augmentations are similar to the ones used during training. Testing time augmentation is a simple and an effective way to enhance the performance of DNNs
[34, 47, 54]. We demonstrate this in our experiment for the nodule detection task.

## False Positive Reduction Stage

Pulmonary nodules are highly variable in shape, size, and density, they have similar morphological characteristics to neighbouring organs and non-nodule structures, e.g., blood vessels and airways (see Figs 1 and 7). This high morphological variability increases the complexity of the detection task leading to high rates of false positive detections [25, 43, 44, 57, 63]. To address this issue, we deploy a false positive reduction stage, in which we utilise a joint analysis based approach that incorporates nodule morphology and spatial context information to perform the final detection. We further utilise cross-channel attention to assist the network in focusing on important information by modelling correlations between the embedded features. An overview of our false positive reduction stage is presented in Fig. 2.

First, a suspected region (i.e. candidate locations proposed by the candidate proposal stage) is extracted in three

<figure><img src="image_11.png" /><figcaption>**Fig. 7** A visual comparison between pulmonary nodule (top) and non-nodule (bottom) regions. The two classes share similar appearance and morphology, including variability in shape, size, and density, which closely resemble neighbouring structures like blood vessels and airways. This morphological similarity increases the complexity of the classification task, suggesting the need for advanced techniques, such as spatial context analysis to effectively distinguish nodules and reduce false positive rates in detection</figcaption></figure>

different scales, such that each patch spans a different level of spatial context (area) around the centre of the suspected region. Here, we use small (15 x 15 x 15), medium (25 x 25 x 25), and large (40 x 40 x 40) patches, representing short, medium, and long range spatial context, respectively. Note that these are picked with respect to nodule size distribution. The extracted patches are then resized (individually) into 20 x 20 x 20 using bicubic interpolation, and are fed into three parallel convolutional layers in which each layer is specialised in a particular type of patches (i.e. small, medium, or large). The resulting feature maps are then individually down-sampled using a max pooling operation, aggregated (concatenated), and are jointly analysed by a sequence of four residual blocks [18], in which 2, 3, 3, and 3 residual units are incorporated, respectively. Moreover, the first residual block is followed by subsequent max pooling layer to further reduce the dimensionality of the 3D feature maps. In line with the concept of attention, our design aims at assisting the network in learning contextual information by independently analysing nodules at different extents of spatial information (scales), while modelling correlations between the different contexts by jointly analysing the aggregated feature maps. Figure 8 shows examples of nodule images of different contextual levels and a random selection of their feature maps. We notice that by jointly exploiting inputs of different scales, the network was able to integrate information from multiple levels of spatial context (MLSC). In our experiment (Sect. Experiments), we demonstrate the benefit of the joint analysis and compare it against the performance when single scale inputs are used.

As observed in Fig. 1, most nodules are of small size, making the classification task more challenging [25, 44, 57, 63]. Moreover, smaller structures are prone to shrinkage due to the use of down-sampling layers, this contributes negatively to the quality of the acquired information, and therefore the accuracy of the detection. Thus, we augment the residual units of our backbone network by

<figure><img src="image_12.png" /><figcaption>**Fig. 8** Examples of random pulmonary nodules (odd columns) and their correspondent feature maps (even columns) from different levels of spatial context: long range context (top), medium range context (middle), and short range context (bottom). Note that the feature</figcaption></figure>

maps are selected randomly from the second convolutional layer in the false positive reduction network. We notice that exploiting information from different input scales (levels of context) assists the network in integrating spatial contextual information of different levels
zoom-in convolutional paths to assist the network in picking fine details from different feature scales. An intermediate feature map F is first processed by a deconvolutional layer using kernels of size 2 x 2 x 2 and stride of 2, the feature map is therefore up-sampled by a factor of 2. The resulting embedding is then regularised using a batch normalisation layer, followed by a non-linear activation layer, and is finally transformed back into its original dimensions using a 3D max pooling operation. The output of the zoom-in layer is incorporated as a skip connection similar to a residual path using element-wise addition. See Fig. 9. The intuition behind this strategy is to promote the network into learning to emphasise (magnify) small structures using the up-sampling convolution, such that they are less prone to diminishing due to repeated down-sampling operations and the increasing receptive field. In our experiment (Sect. Experiments), we demonstrate the effectiveness of our proposed zoom-in path within the false positive reduction task, and evaluate its performance within the candidate proposal task.

Surprisingly, unlike our finding in the candidate proposal stage, we observe no significant improvement when integrating our proposed spatial attention (Eqs. 7 to 11) within the false positive reduction task. A similar pattern was found when evaluating spatial attention from [56]. Nonetheless, when evaluating channel attention within the false positive reduction task, we observe an enhanced performance using the channel attention approach proposed in [56], while no significant yield was found by integrating our proposed technique (Eqs. 5 to 6). This may be due to the different complexity of the two tasks. In Sect. Experiments, we provide an extensive analysis in which we compare different types and combinations of attention techniques. Accordingly, we adopt the cross-channel attention approach from [56] into the building blocks of our false positive reduction network. The overall channel attention process is described in Eq. 2. Similar to [56], channel attention is placed prior to the residual path in a residual unit.

The output of the last residual block is then passed into a fully connected network of 3 subsequent layers in which the final layer predicts nodule probabilities using a sigmoid function. Similar to the candidate proposal stage, we evaluate focal cross entropy [28] within our false positive reduction task. We observe that setting the modulation parameter $\gamma$ (Eq. 13) to 0 produces the best overall performance. While using higher values provoke lower false positive rates on the account of lower sensitivity in the nodule class, setting $\gamma$ to 0 induces a favourable balance in the overall performance. Note that when $\gamma$ is set to 0, the loss computation is equivalent to using pure cross entropy.

## Integration of Detection Stages

Our framework performs the detection in two stages, candidate proposal and false positive reduction. In line with [43] and [51], we find that ensembling the two stages such that the final prediction is a function of the two models, achieves the best detection performance. First, for each tested image, predictions are then sorted in a descending order with respect to their nodule probability, locations with the highest 300 nodule scores are used as an initial pool of candidates. These are then evaluated against a threshold $t$, where $t$ is set to 0.3. To eliminate any overlapping predictions, candidates are then processed by a non maximum suppression (NMS) operation with an intersection over union threshold of 0.1.

The remaining nodule candidates are then passed into the false positive reduction stage, where candidate regions are extracted and are processed by the false positive reduction network. For each detection, the final score is defined as the average probability of both stages. Last, detections with probabilities $\ge$ 0.3 are used to form the final set of detection.

Our candidate proposal network leverages 3D attention to improve the detection performance by prompting the network to focus on the most relevant information. The fully convolutional nature of the proposed channel and spatial attention enables the network to focus on informative features without the computational burden associated with MLP layers used in common attention mechanisms (e.g., Squeeze-and-Excite and CBAM). Inspired by variable morphology and sizes of lung nodules, in the false positive reduction stage, we employ a joint analysis that considers spatial information at different contextual levels (short, medium, and long range spatial contexts). This is further augmented by zoom-in paths that emphasize smaller structures (e.g., small nodules), improving the overall detection accuracy. In line with the candidate proposal stage, we utilise spatial attention blocks to assist the network focus on the most relevant features. We also employ a modified ReLU activation function to accelerate convergence and enhance the overall performance.

Combining these components, our approach reduces the computational costs associated with the 3D pulmonary nodule detection while improving detection performance, showing that efficient attention mechanisms can be effectively utilised within 3D imaging contexts. Listing 1 presents an algorithmic overview of the proposed framework.
```python
# TTA: Augment input (3D CT image)
axial, coronal, sagittal = Augment(CT)

# Stage 1: Candidate Proposal
for x in [axial, coronal, sagittal]:
    for ResBlock in RPN:
        x = Attnet(ResBlock(x))
        det = Detect(x) # 3D coord.
        proposal.append det

# Aggregate using NMS
proposal = NMS(proposal)

# Stage 2: FP Reduction
scales = [15, 25, 40] # spatial context
size = 20
x1, x2, x3 = Crop(proposal, scales)
x1 = 3DConv(Resize(x1, size))
x2 = 3DConv(Resize(x2, size))
x3 = 3DConv(Resize(x3, size))
x = Concatenate(x1, x2, x3)
for ResBlock in Cls:
    x = Attnet(ZoomIn(ResBlock(x)))
    pred = Classify(x) # final class
```

**Listing 1** An algorithmic overview of the proposed framework. AttentNet performs the detection in two stages, candidate proposal, in which we exploit a 3D CNN to predict suspicious nodule locations at a high rate, and a false positive reduction stage in which a 3D CNN is used to reduce the number of false alarms. We augment RPN network with attention units to assist the network in focusing on effective nodule features and therefore produce a more robust detection. Moreover, to assist the network cope with the complex nodule morphology, we incorporate multi-scale spatial contextual information using a joint analysis approach and a zoom-in convolutional path in the false positive reduction stage. To further improve the detection performance, we utilize a simple test time augmentation technique in which detections are aggregated across different image augmentations, enhancing the overall detection accuracy.

## Experiments

All experiments were implemented using PyTorch DL library with an NVIDIA V100 16GB GPU. Both the candidate proposal and false positive reduction stages were trained (separately) for 250 and 10 epochs (~2 and ~0.5 days), respectively, using a batch size of 7 and 64, respectively. For both stages, we use Stochastic Gradient Descent (SGD) optimisation with an initial learning rate of 0.01. For the candidate proposal stage, the learning rate is decreased to 0.001, 0.0005, and 0.0001 after 50, 100, and 150 epochs, respectively, while it is conserved at 0.01 for the false

positive reduction stage. Network parameters are initialised using He et al. [17].

We train and evaluate both stages following 10-fold cross validation as suggested by LUNA16 [43]. Performance is evaluated using Free Response Receiver Operating Characteristic (FROC) curve, in which sensitivity is computed at 7 predefined average false positive rates per scan (i.e. 0.125, 0.25, 0.5, 1, 2, 4, and 8), as well as Competition Performance Metric (CPM), the LUNA16's official metric, defined as the average sensitivity across all predefined false positive rates [43]. Most clinical setups define their effective threshold between 1 and 4 false positives per scan. Including lower false positive rates in the evaluation metric makes the task more challenging [43].

In the remainder of this section, we present data and pre-processing in Sects. Data and Pre-processing, respectively. Furthermore, to evaluate the contribution of the individual components of our approach, we first perform an ablation study for each of the detection stages (candidate proposal and false positive reduction stage) in Sect. Ablation study, then we evaluate the performance of the fully integrated system in Sect. Integrated system performance.

## Data

We use the LUNA16¹ (Lung Nodule Analysis 2016) [43] dataset, a subset of LIDC/IDRI dataset [3], to carry out our experiments. The LIDC/IDRI dataset was collected in two stages, a blinded annotation stage, where 4 radiologists were asked to independently mark suspicious locations, and an unblinded stage, where results of all radiologists where anonymised and provided to each of the radiologists to assist them and re-evaluate their initial annotations.

LUNA16 excludes scans with slice thickness > 2.5 mm. Scans with impaired slices or inconsistent slice spacing are also excluded, resulting a total of 888 CT scans. Moreover, only nodules that are ≥ 3 mm and are accepted by a minimum of 3 out of 4 radiologists are included in LUNA16, summing up to 1186 nodule labels. Following the lung cancer screening protocols, any remaining annotations are flagged as irrelevant findings.

Furthermore, for the false positive reduction task, LUNA16 provides 754,975 candidate locations acquired using multiple existing methods. These include 1,166 nodules that match the radiologists annotations and 753,809 of non-nodule locations.

<footnote n="1">¹ Available at https://luna16.grand-challenge.org/.</footnote>
## Pre-Processing

Similar to [26] and [65], we pre-process LUNA16 images by clipping their intensity values between -1200 and 600 Hounsfield units (HU) followed by rescaling them between 0 and 1. We use the lung masks provided by LUNA16 to isolate the informative lung region and mask out any surrounding organs. Due to GPU limitations, for the candidate proposal stage, images were split into patches of size 128 x 128 x 128 during the training stage. We follow the same approach for the testing stage allowing an overlap of 32 pixels between the cropped patches.

Additionally, we follow a cross-sectional augmentation strategy, where scans, along with their annotations, are transformed from axial plane (original plane), into sagittal and coronal planes. Resulting in a 3x increase in the dataset size for the Candidate Proposal stage, increasing the original 1186 nodule annotations to 3558 augmented nodules. For the false positive reduction stage, patches are extracted using annotation coordinates provided by LUNA16 for the false positive reduction task. Each patch is cropped in 3 different sizes (levels of spatial context), 15 x 15 x 15, 25 x 25 x 25, and 40 x 40 x 40. This ensures the coverage of 99% of the nodules [12]. All patches are then resized to 20 x 20 x 20 using bicubic interpolation. To further augment the nodule class during training, we apply cross-sectional transformations similar to that in the candidate proposal stage, and perform 1-pixel shifts along the x, y, and z axes. The shifts are applied in both directions for each axis resulting in 6 additional variations per patch. Combined with the cross-sectional transformations, this leads to an 18x increase in the number of nodule annotations, increasing the 1166 nodules to 20,988 augmented nodules. Non-nodule patches are not augmented as their count is already sufficient (753,809). These augmentations aim to improve the model generalizability by introducing spatial variations and leveraging multi-plane perspectives. Table 3 presents a summary of the label number before and after applying augmentation.

## Ablation Study

### Candidate Proposal Stage

For the candidate proposal task, we compare the impact of different attention mechanisms against the performance of our backbone CNN when no attention is used. Results are presented in Table 1, and Fig. 10. Performing 10-folds cross validation requires extensive amounts of time (e.g., ~2 days per fold), therefore, for the purpose of evaluating the candidate proposal stage, training and testing were performed using a randomly selected keep-out fold. Nonetheless, in Sect. Integrated system performance, we evaluate our fully integrated system using 10-folds cross validation as suggested in LUNA16 [43]. Note that all our experiments are trained using ReLU activation function unless stated otherwise.

In our first experiment, we evaluate the performance of our candidate proposal network when no attention is incorporated. This will serve as a baseline for our candidate proposal stage. We find that the network has successfully detected lung nodules at a recall (sensitivity) of 0.878 at 4 false positives per scan, and a CPM score (i.e. average sensitivity over 7 false positive thresholds, 0.125, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0) of 0.772, prior to any false positive reduction. Note that most clinical setups define their effective threshold between 1 and 4 false positives per scan [43]. This demonstrates a compelling potential of CNNs in the pulmonary nodule detection task.

To evaluate the impact of different attention mechanisms, we compare cross-channel and spatial-wise attention when applied individually, and when applied in combination (i.e. subsequently) within the building blocks of the detection network.

We find that all channel attention approaches (i.e. the proposed approach and the ones from [56] and [19]) contribute positively to the overall detection performance, with our proposed strategy producing the highest CPM score of

<table><caption>**Table 1** Candidate proposal ablation study: FROC at different numbers of false positives per scan obtained by different methods under comparison on a randomly selected keep-out fold from LUNA16 dataset</caption><thead><tr><th>FROC</th><th>Mean</th><th>0.125</th><th>0.25</th><th>0.5</th><th>1.0</th><th>2.0</th><th>4.0</th><th>8.0</th></tr></thead><tbody><tr><td>RPN</td><td>0.772</td><td>0.576</td><td>0.669</td><td>0.754</td><td>0.790</td><td>0.846</td><td>0.878</td><td>0.890</td></tr><tr><td>RPN+Transformer [54]</td><td>0.746</td><td>0.537</td><td>0.619</td><td>0.706</td><td>0.764</td><td>0.826</td><td>0.872</td><td>0.897</td></tr><tr><td>RPN+CA [19]</td><td>0.813</td><td>0.638</td><td>0.697</td><td>0.792</td><td><strong>0.856</strong></td><td>0.877</td><td><strong>0.911</strong></td><td>0.919</td></tr><tr><td>RPN+SA+CA [56]</td><td>0.743</td><td>0.576</td><td>0.631</td><td>0.687</td><td>0.745</td><td>0.816</td><td>0.862</td><td>0.886</td></tr><tr><td>RPN+SA [56]</td><td>0.719</td><td>0.507</td><td>0.602</td><td>0.688</td><td>0.725</td><td>0.801</td><td>0.843</td><td>0.869</td></tr><tr><td>RPN+CA [56]</td><td>0.779</td><td>0.584</td><td>0.679</td><td>0.753</td><td>0.807</td><td>0.847</td><td>0.884</td><td>0.902</td></tr><tr><td>RPN+proposed CA+SA</td><td>0.782</td><td>0.544</td><td>0.689</td><td>0.765</td><td>0.810</td><td>0.864</td><td>0.886</td><td>0.912</td></tr><tr><td>RPN+proposed SA</td><td>0.784</td><td>0.575</td><td>0.670</td><td>0.750</td><td>0.813</td><td>0.860</td><td>0.900</td><td>0.923</td></tr><tr><td>RPN+proposed CA</td><td><strong>0.826</strong></td><td><strong>0.657</strong></td><td><strong>0.761</strong></td><td><strong>0.808</strong></td><td>0.834</td><td><strong>0.887</strong></td><td>0.909</td><td><strong>0.929</strong></td></tr></tbody></table>

Here, CA and SA stand for channel attention and spatial attention, respectively. The highest scores are highlighted in bold
0.826, in contrast to 0.779 and 0.813 using the attention approaches from [56] and [19], respectively. This demonstrates the advantage of our fully convolutional design; by replacing the MLP units (i.e. channel attention in [56] and [19]) with convolutional operations, we avoid the need for heavy dimensionality reduction (see Sect. Candidate proposal stage (RegionProposal Network)), and attention can therefore be efficiently performed using spatial embeddings of higher dimensions, leading to an enhanced performance. As demonstrated by the results, this is particularly useful when managing 3D data of high spatial resolution.

Moreover, when evaluating the impact of spatial attention, we find that our proposed spatial attention strategy was able to successfully enhance the network's performance with a CPM score of 0.784, showing an increase of 1.2% comparing to the baseline network when no attention is used. On the other hand, we find that spatial approach from [56] leads to a worse performance comparing to the baseline network. This shows that, in contrast to [56], where spatial descriptors are aggregated using element-wise pooling operations along the channel axis, neglecting the volumetric aspect of the data, the incorporation of cross-sectional spatial information in our attention strategy can successfully assist the network in learning correlations between the different image cross-sections, and consequently capturing important 3D information within the images.

Furthermore, we experiment with multi-headed attention (Transformer) [11, 52] to evaluate the impact of incorporating long-range correlations and global context on the pulmonary nodule detection task. For this experiment, we adopt a hybrid architecture that takes advantage of the spatial representational power of CNNs and the ability in modelling long-range dependencies in Transformers [54]. Particularly, we use our 3D encoder CNN to extract and down-sample pulmonary nodule feature maps in which a Transformer takes as an input to analyse and perform self-attention Subsequently, the resulting feature maps are passed into the decoder part of the CNN where the final detection is performed. To the best of our knowledge, we are the first to evaluate Transformer networks within the pulmonary nodule detection task. We find that Transformer based detector produces an increased sensitivity of 0.897 at 8 false positives per scan comparing to 0.890 using the baseline network when no attention is used. However, Transformer shows a decrease of 2.6% in the overall CPM score in contrast to the baseline network. Unlike CNNs, the tokenisation process of input images in Transformer leads to a limited ability in modelling local structures, and consequently, a limited translation equivariance [11]. This may indicate that the pulmonary data benefits more from localised attention (e.g. convolution based attention) than long range global attention. In [11], the authors suggest that training Transformers using larger datasets (e.g. 14 M-300 M images) may help in

overcoming this issue, however, due to the lack of a suitable dataset (e.g. a dataset of similar configurations and nature to the lung images) of this size, this remains a challenge when dealing with pulmonary images.

Generally, while both our spatial and cross-channel attention demonstrate a positive impact within the pulmonary nodule detection task, we observe that the network yields a better performance using channel-wise attention in contrast spatial attention. A similar trend is observed when evaluating channel and spatial attention from [56] and [19]. Moreover, when combining both our channel and spatial attention by applying them in subsequently (as suggested in [56]), we notice an increase in the CPM score comparing to the baseline network when no attention is used, however, we find that the network benefits the most when cross-channel and spatial attention are incorporated individually, with channel attention being the best performing amongst all configurations. In the same line, when evaluating the attention methods from [56], we find that their channel attention produces better results in contrast to both, their spatial attention, and spatial and channel attention when applied in combination. These observations indicate a high importance of inter-channel dependencies in the pulmonary nodule detection task. In fact, we argue that channel attention not only assists the network in modelling inter-channel correlations, but also inherently infers spatial attention by assisting the network in focusing on important feature maps (channels) in which informative spatial features are embedded. Accordingly, and due to the improved performance demonstrated in our experiment, we continue using our proposed channel attention as the method of choice for the candidate proposal task.

In our preliminary experiments, we evaluate our zoom-in path within the candidate proposal stage. We observe comparable performance to the network when no zoom-in paths are used. This is expected since the candidate proposal network benefits from an encoder-decoder design in which spatial features are extracted from multiple spatial scales. However, the zoom-in path demonstrates benefits when incorporated within the false positive reduction task, where an encoder-decoder design is not straightforwardly applicable due to the relatively narrow input dimensions (details in Sect. False positive reduction stage).

Finally, we evaluate the impact of our proposed modified ReLU activation function within the nodule detection task by applying it to the best performing network configuration amongst all tested detectors (i.e., RPN with the proposed cross-channel attention), and compare its performance to the network with pure ReLU, Leaky ReLU, and ELU activations. Results are presented in Table 2. We find that the modified ReLU activation produces an increased FROC score of 0.833 comparing to 0.826, 0.822, and 0.830 when using pure ReLU, Leaky ReLU, or ELU activations, respectively. More particularly, when comparing to pure ReLU
<table><caption>**Table 2** FROC at different numbers of false positives per scan obtained by the best performing candidate proposal network (proposed RPN with cross-channel attention) on a randomly selected keep-out fold from LUNA16 dataset using different activation functions</caption><thead><tr><th>FROC</th><th>Mean</th><th>0.125</th><th>0.25</th><th>0.5</th><th>1.0</th><th>2.0</th><th>4.0</th><th>8.0</th></tr></thead><tbody><tr><td>ReLU</td><td>0.826</td><td><strong>0.657</strong></td><td><strong>0.761</strong></td><td>0.808</td><td>0.834</td><td>0.887</td><td>0.909</td><td>0.929</td></tr><tr><td>Leaky ReLU</td><td>0.822</td><td>0.602</td><td>0.732</td><td><strong>0.826</strong></td><td>0.868</td><td>0.897</td><td>0.910</td><td>0.916</td></tr><tr><td>ELU</td><td>0.830</td><td>0.641</td><td>0.743</td><td>0.802</td><td>0.861</td><td>0.895</td><td><strong>0.923</strong></td><td><strong>0.943</strong></td></tr><tr><td>prop. mod. ReLU</td><td><strong>0.833</strong></td><td>0.642</td><td>0.733</td><td>0.808</td><td><strong>0.882</strong></td><td><strong>0.909</strong></td><td>0.922</td><td>0.938</td></tr></tbody></table>

The highest scores are highlighted in bold

<figure><img src="image_13.png" /><figcaption>**Fig. 9** An overview of our proposed 3D fully convolutional zoom-in path within a residual block. Our zoom-in path takes intermediate feature maps of size C x D x H x W as an input. These features are up-sampled by a transposed convolutional operation resulting a feature map of size C x D' x H' x W' to assist the network in capturing finer details from different spatial scales. The extracted features are then normalised using a batch normalisation layer, mapped into a non-linear space, and are pooled using a max pooling operator in which the resulting embeddings are of the same dimensions as the original input. These are then aggregated along with the features from the residual block using an element-wise addition process (⊕ in the figure). The parameters k and s represent the kernel size and the stride used in the convolutional and the max pooling layers</figcaption></figure>

and Leaky ReLU, a significant increase is observed in the sensitivity at false positive per scan thresholds from 1.0 to 8.0, in which an effective clinical threshold (i.e., 1.0 - 4.0) is defined within [43]. In the same line, the proposed modified ReLU activation achieves the highest sensitivity at 1 and 2 false positives per scan. ELU on the other hand obtains higher sensitivity at 4 and 8 false positives per scan, and a comparable average FROC score to the proposed activation. This is expected since ELU and the proposed activation have similar characteristics over the negative segment of the activation function. The performance gain using the proposed activation may be explained by the smoother gradient flow in contrast to that in ELU as demonstrated in Figs. 6 and 11. Unlike unbounded activations (e.g., Leaky ReLU), our modified ReLU activation, as well as ELU activation, are bounded for negative inputs, promoting network regularization and reducing the risk of overfitting. These observations demonstrate that allowing small outputs for inputs in the negative range of ReLU can improve the gradient flow and therefore enhances the learning process, see Fig. 11. Accordingly, we adopt this modified ReLU activation for our detection network and continue using it for the remainder of our experiment.

## False Positive Reduction Stage

For the false positive reduction task, we first evaluate the impact of using different input scales (levels of spatial context) on the detection, and compare it against our proposed joint analysis approach in which we incorporate inputs of

<table><caption>**Table 3** Summary of dataset sizes and transformations applied during the Candidate Proposal and False Positive Reduction stages</caption><thead><tr><th>Stage</th><th># Labels (before)</th><th>Transformations</th><th># Labels (After)</th><th>Increase</th></tr></thead><tbody><tr><td>Candidate proposal</td><td>1186 nodules</td><td>Cross-sectional transformations (axial, sagittal, coronal)</td><td>3558</td><td>3x</td></tr><tr><td>FP Reduction</td><td>1,166 nodules</td><td>Cross-sectional transformations (3x) xyz shifts (6x)</td><td>20,988</td><td>18x</td></tr></tbody></table>

The table shows the number of labels before and after transformations, along with the types of transformations applied and the corresponding increase in data size
<table><caption>**Table 4** False positive reduction ablation study: FROC at different numbers of false positives per scan obtained by different methods under comparison on a randomly selected keep-out fold from LUNA16 dataset</caption><thead><tr><th>FROC</th><th>Mean</th><th>0.125</th><th>0.25</th><th>0.5</th><th>1.0</th><th>2.0</th><th>4.0</th><th>8.0</th></tr></thead><tbody><tr><td>Short range spatial context</td><td>0.623</td><td>0.207</td><td>0.354</td><td>0.538</td><td>0.681</td><td>0.763</td><td>0.877</td><td>0.936</td></tr><tr><td>Medium range spatial context</td><td>0.769</td><td>0.503</td><td>0.646</td><td>0.738</td><td>0.822</td><td>0.861</td><td>0.902</td><td>0.910</td></tr><tr><td>Long range spatial context</td><td>0.715</td><td>0.487</td><td>0.578</td><td>0.681</td><td>0.740</td><td>0.814</td><td>0.854</td><td>0.855</td></tr><tr><td>Multi-level spatial context (MLSC)</td><td>0.792</td><td>0.529</td><td>0.626</td><td>0.738</td><td><strong>0.831</strong></td><td><strong>0.915</strong></td><td><strong>0.946</strong></td><td><strong>0.963</strong></td></tr><tr><td>MLSC+zoom-in (MLSC-Z)</td><td><strong>0.813</strong></td><td><strong>0.661</strong></td><td><strong>0.730</strong></td><td><strong>0.783</strong></td><td>0.822</td><td>0.846</td><td>0.912</td><td>0.938</td></tr><tr><td>MLSC-Z+Transformer [54]</td><td>0.672</td><td>0.260</td><td>0.453</td><td>0.631</td><td>0.714</td><td>0.840</td><td>0.902</td><td>0.904</td></tr><tr><td>MLSC-Z+proposed CA+SA</td><td>0.678</td><td>0.455</td><td>0.500</td><td>0.575</td><td>0.715</td><td>0.792</td><td>0.844</td><td>0.866</td></tr><tr><td>MLSC-Z+proposed SA</td><td>0.805</td><td>0.564</td><td>0.661</td><td>0.790</td><td>0.828</td><td>0.881</td><td>0.939</td><td><strong>0.972</strong></td></tr><tr><td>MLSC-Z+proposed CA</td><td>0.775</td><td>0.564</td><td>0.640</td><td>0.740</td><td>0.790</td><td>0.849</td><td>0.899</td><td>0.946</td></tr><tr><td>MLSC-Z+CA+SA [56]</td><td>0.740</td><td>0.433</td><td>0.566</td><td>0.713</td><td>0.801</td><td>0.849</td><td>0.907</td><td>0.909</td></tr><tr><td>MLSC-Z+SA [56]</td><td>0.778</td><td>0.505</td><td>0.634</td><td>0.744</td><td>0.821</td><td>0.903</td><td>0.919</td><td>0.919</td></tr><tr><td>MLSC-Z+CA [19]</td><td>0.833</td><td>0.593</td><td>0.711</td><td>0.812</td><td><strong>0.883</strong></td><td><strong>0.926</strong></td><td><strong>0.950</strong></td><td>0.957</td></tr><tr><td>MLSC-Z+CA [56]</td><td><strong>0.848</strong></td><td><strong>0.702</strong></td><td><strong>0.745</strong></td><td><strong>0.815</strong></td><td>0.862</td><td>0.906</td><td>0.940</td><td>0.963</td></tr></tbody></table>

The top section of the table compares results of the false positive reduction network when incorporating spatial information from different contextual levels. The bottom section presents a comparison of different attention mechanisms. Here, CA and SA stand for channel attention and spatial attention, respectively. The highest scores are highlighted in bold

<figure><img src="image_14.png" /><figcaption>**Fig. 10** FROC of the proposed region proposal network using different attention mechanisms under comparison on a randomly selected keep-out fold from LUNA16 dataset. Legend indicates CPM score for each tested detector. Generally, in contrast to the baseline RPN (dashed line), we observe an enhanced performance when incorporating channel attention (CA) mechanisms, e.g., proposed CA and CA Squeeze and Excite [19]. We also note the highest gains when using our proposed CA approach, demonstrating the importance of directly incorporating the 3D aspect of the data when performing attention</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**Fig. 11** Validation loss of the proposed lung nodule detection network using different activation functions. Our proposed modified ReLU activation, as well as ELU activations, are smooth around the origin, promoting faster learning in contrast to, pure ReLU and leaky ReLU activations</figcaption></figure>

multi-level spatial context (MLSC). We then evaluate the performance of our zoom-in path, as well as different attention mechanisms, against the performance of our backbone CNN when no attention is used. Results are presented in

Table 4. Similar to the candidate proposal stage, we evaluate our false positive reduction network using a randomly selected keep-out fold.

When comparing different input scales for the false positive reduction network, we find that medium range spatial context (25 x 25 x 25) produces the best performance, in contrast to short (15 x 15 x 15) and long (40 x 40 x 40) range spatial context. However, we find that jointly analysing MLSC inputs enhances the overall detection comparing
to using any of the input scales individually, with a CPM score of 0.792 comparing 0.769 produced using medium range spatial context inputs. This demonstrates the importance incorporating spatial information from different levels when managing objects with the high size variability such as the pulmonary nodules (see Fig. 1). Furthermore, Fig. 8 shows that our MLSC network was able to integrate spatial contextual information of different levels using our joint analysis approach. We also find that augmenting our multi-level spatial context network with the proposed zoom-in path (MLSC-Z) enhances the performance even further, with a CPM score of 0.813, showing an increase of 2.1% comparing to the performance when the zoom-in path is not used. Note that unlike the candidate proposal stage where the network benefits from an encoder-decoder design in which feature maps are extracted from multiple spatial scales, due to the nature of the false positive reduction task and the relatively narrow input dimensions, an encoder-decoder approach is not straightforwardly applicable. Thus, we design and incorporate our zoom-in path within the building blocks of the false positive reduction network.

When comparing different attention approaches, similar to our observation in the candidate proposal stage (Sect. Candidate proposal stage), channel attention demonstrates the best performance comparing to both, spatial and global (i.e. self-attention) attention. Indeed, the essence of cross-channel attention is in line with the concept of our joint analysis approach, in which we aim to assist the network in capturing correlations between inputs of different levels of spatial context, this suggests that both, channel attention and the joint analysis approach provide complementary information to one another. Furthermore, contrary to our finding in the candidate proposal task, we notice that channel attention approaches form [56] and [19] show better performance comparing to our proposed channel attention within the false positive reduction task, with the channel attention from [56] producing the highest results, showing an increase of 3.5% in the overall CPM score in contrast to the performance when no attention is used. This may be due

the different complexity between the two tasks, this may indicate that the MLP based attention (e.g. [56] and [19]) is more suitable for the false positive reduction task in contrast to our fully convolutional channel-wise attention strategy. Accordingly, we continue using channel attention from [56] as our approach of choice in the false positive reduction stage. On the other hand, we find that our spatial attention produces the highest sensitivity at 8 false positives per scan, and a higher overall performance in contrast to the spatial attention approach from [56]. This demonstrates the benefits of incorporating cross-sectional spatial information in our spatial attention comparing to the channel-wise pooling approach used in [56].

Overall, our results demonstrate the importance of exploiting morphological information when dealing with pulmonary nodules. Combining nodule morphology with attention mechanisms further enhances the network ability in learning effective embeddings and consequently produce more robust predictions.

## Integrated System Performance

In this section, we evaluate the performance of the proposed nodule detection system by integrating both detection stages (candidate proposal and false positive reduction stage) in an ensemble model. To allow comparison with existing methods, all experiments in this section are done following 10-folds cross validation, as suggested in LUNA16 [43]. Results are presented in Table 5.

First, we evaluate the performance of our candidate proposal network (RPN) when no false positive reduction stage is incorporated. We find that that our network produces a better overall performance in contrast to the two stage detector [4] and higher or comparable results to the single stage detectors, [27, 65] and [26]. Additionally, when comparing our network to baselines at false positives per scan ≥ 1, in which an effective clinical threshold is defined [43], we find that our network outperforms [4, 27], and [65] at 1.0, 2.0, 4.0 and 8.0 false positives per scan, as well as [26] at 4.0 and 8.0

<table><caption>**Table 5** FROC at different numbers of false positives per scan obtained by our fully integrated two stage pulmonary nodule detection system, AttentNet, in contrast to baseline methods using 10-folds cross validation on LUNA16 dataset</caption><thead><tr><th>FROC</th><th>Mean</th><th>0.125</th><th>0.25</th><th>0.5</th><th>1.0</th><th>2.0</th><th>4.0</th><th>8.0</th></tr></thead><tbody><tr><td>ZNET [4]</td><td>0.811</td><td>0.661</td><td>0.724</td><td>0.779</td><td>0.831</td><td>0.872</td><td>0.892</td><td>0.915</td></tr><tr><td>3D RCNN [27]</td><td>0.834</td><td>0.662</td><td>0.746</td><td>0.815</td><td>0.864</td><td>0.902</td><td>0.918</td><td>0.932</td></tr><tr><td>DeepLung [65]</td><td>0.842</td><td>0.692</td><td>0.769</td><td>0.824</td><td>0.865</td><td>0.893</td><td>0.917</td><td>0.933</td></tr><tr><td>DeepSeed [26]</td><td>0.862</td><td>0.739</td><td>0.803</td><td>0.858</td><td>0.888</td><td>0.907</td><td>0.916</td><td>0.920</td></tr><tr><td>AttentNet (RPN) (Ours)</td><td>0.842</td><td>0.656</td><td>0.774</td><td>0.831</td><td>0.874</td><td>0.903</td><td>0.923</td><td>0.936</td></tr><tr><td>AttentNet (Ours)</td><td>0.871</td><td><strong>0.752</strong></td><td><strong>0.817</strong></td><td><strong>0.857</strong></td><td>0.885</td><td><strong>0.920</strong></td><td>0.933</td><td>0.933</td></tr><tr><td>AttentNet+TTA (Ours)</td><td><strong>0.874</strong></td><td>0.748</td><td>0.812</td><td>0.856</td><td><strong>0.893</strong></td><td>0.919</td><td><strong>0.942</strong></td><td><strong>0.945</strong></td></tr><tr><td>APNDCTDCNN [10]</td><td>0.891</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>3DDUALRPN [59]</td><td><strong>0.922</strong></td><td><strong>0.801</strong></td><td><strong>0.860</strong></td><td><strong>0.919</strong></td><td><strong>0.952</strong></td><td><strong>0.968</strong></td><td><strong>0.977</strong></td><td><strong>0.977</strong></td></tr></tbody></table>

Here, TTA indicates the use of testing time augmentation. The highest scores are highlighted in bold
false positives per scan with a comparable performance at 1.0 and 2.0 false positives per scan. Note that our proposed network contains 3.1M trainable parameters, in contrast to 17 M, 5.4M, 1.4M, and 5.4M parameters in [4, 27, 65], and [26], respectively, making it one of the most compact networks amongst the methods under comparison. This demonstrates the positive impact of our proposed 3D attention strategy, by enabling the network in focusing on important features, our network was able efficiently to produce robust detections, prior to any false positive reduction.

Furthermore, we find that by integrating both detection stages as an ensemble, our network was able to achieve higher performance score comparing to 3D detector methods [27, 65], and [26], with a CPM score of 0.871, showing a significant increase of 2.9% in contrast to our network prior to the false positive reduction step, and 0.9% comparing to [26], in which MLP based SE attention blocks were utilised. This indicates the importance of the false positive reduction step, particularly when managing a critical task such as pulmonary nodule detection. By analysing suspected nodule regions and extracting deeper semantic features, the network was able to produce more robust predictions.

On the other hand, both APNDCTDCNN [10] and 3DDUALRPN [59] demonstrate higher performance on the LUNA16 dataset compared to our attention-based approach, achieving CPM scores of 0.891 and 0.922, respectively. However, it is important to note that [10] relies on Faster R-CNN [39] for their candidate proposal task, which involves 41 M trainable parameters, in contrast to our candidate proposal stage, which consists of only 3.1M parameters. Meanwhile, 3DDUALRPN [59] uses a dual-path network, consisting of a ResNet50 based RPN as well as a DenseNet path, in an ensemble manner for their candidate detection stage, resulting in an optimising time of 1 week on an NVIDIA K80 12GB GPU (4992 CUDA cores) for each fold of cross-validation. The exact number of trainable parameters for [59] was not reported. In contrast, our model was trained in just 2 days per fold on an NVIDIA V100 16GB GPU (5120 CUDA cores), making it significantly more efficient in terms of training time and resource usage. While this is an indirect comparison, it effectively highlights the efficiency of our approach.

Moreover, to evaluate the impact of testing time augmentation (TTA), we compare the performance of our network with TTA against the network when no TTA is used. We find that TTA produces an enhanced CPM score of 0.874, showing an increase of 0.3% in contrast to the network when no TTA is used. This is in line with the finding in [34, 47, 54], in which TTA was demonstrated to be a simple and effective way to boost the performance of DL models. Figure 12 compares pulmonary nodules as detected by our network against their correspondent ground-truth locations.

<figure><img src="image_16.png" /><figcaption>Fig. 12 Pulmonary nodules detected by AttentNet (green) and their correspondent ground-truth boxes (red)</figcaption></figure>

Overall, our proposed detector and attention methods demonstrate promising results within the pulmonary nodule detection task. AttentNet can efficiently achieve competitive performance in contrast to existing detectors, a total sensitivity of 95%. See Fig. 13. AttentNet significantly outperforms [4, 26, 27, 65] at 1.0, 2.0, and 4.0 false positives per scan, in which an effective clinical threshold is defined [43], and produces the highest sensitivity at the more challenging lower false positive per scan thresholds (i.e. 0.125, 0.25, and 0.5),
<figure><img src="image_17.png" /><figcaption>**Fig. 13** FROC of all systems under comparison using 10-folds cross-validation over LUNA16 dataset. Our proposed system produces the highest CPM score (average sensitivity over all predefined false positive thresholds) of 0.874 with a total sensitivity of 95%</figcaption></figure>

while maintaining a compact network design with a relatively smaller number of learnable parameters.

Future work could also explore the proposed convolutional attention methods across different domains and applications, or using datasets with structural similarity to that in LUNA16. This provides closer insights into their performance and broader applicability. For instance, we observed that the performance of different types of attention (spatial and channel wise attention) may vary depending on the target task (e.g., detection versus classification). In the same line, combining both attention can sometimes diminish the benefits of using them individually. Further investigation can help better understand these observations and how to optimize attention based models.

Another key challenge is the limited availability of annotated datasets. While LUNA16 is a widely used dataset and provides a valuable foundation for experimentation, it may not fully capture the variability and the diverse lung nodule morphology seen in real-world clinical settings, limiting the generalizability of the models. This constraint can also impact the use of Vision Transformers, which typically require large datasets and extensive pretraining to be effective.

Finally, while we demonstrated empirically that the proposed modified ReLU activation improves detection performance, further experiments are required to fully understand its broader impact. Large-scale analyses across various domains could provide closer insights into the activation function's advantages and limitations, helping to assess its general applicability and effectiveness.

## Limitations and Future Work

While the proposed framework demonstrates potential by incorporating convolutional attention mechanisms into the 3D lung nodule detection task, there are several limitations. The computational complexity associated with 3D nature of data presents scalability challenges, which can be further complicated by the high memory requirements of the attention mechanisms and large input sizes. For instance, while we experimented with Vision Transformers [11], they were applied at embeddings of reduced resolutions due to computational limitations. Future work could explore more efficient ways to integrate Transformers at larger resolutions to fully leverage their potential in 3D imaging tasks.

Additionally, future work could highlight the systematic benchmarking of computational complexity, such as measuring floating-point operations per second (FLOPs) second, to gain insights into the efficiency of the proposed framework. Currently, such analysis is challenging due to the lack of this data for many baseline models. This helps in establishing a more comprehensive understanding of performance trade-offs, particularly in resource-constrained environments and when dealing with high-dimensional image data.

## Conclusion

In this work, we explore attention mechanisms to assist CNNs focus on essential features, thereby producing more robust predictions. We propose a 3D cross-channel and a cross-sectional spatial attention unit in which we demonstrate effectiveness within the 3D lung nodule detection task, using a two stage detector (candidate proposal and false positive reduction). We show that by incorporating a fully convolutional networks, attention can be efficiently performed using richer descriptors of higher spatial dimensionality, improving the overall performance in contrast to popular multi-layer perceptron based attention networks while retaining model compactness. Additionally, for the false positive reduction task, we show that by jointly analysing inputs of different spatial scales attention and using our zoom-in paths along with cross-channel, the network was able to aggregate information of different contextual levels and produce enhanced predictions. We carry out an extensive analysis on LUNA16 dataset and show the potential of incorporating fully convolutional attention within the 3D lung nodule detection context comparing to baseline detectors where no attention is used achieving a comparable CPM
score of 0.874 and a total sensitivity of 0.95 while retaining a relatively small number of trainable parameters.

**Author Contributions** Majedaldein Almahasneh: carried out body of work and experiments, research, and writing up. Xianghua Xie: first project supervisor, research, and writing and reviewing. Adeline Paie-ment: second project supervisor, research and reviewing.

**Funding** Not Applicable.

**Data Availability Statement** Not Applicable.

## Declarations

**Conflict of interest** Not Applicable.

**Research Involving Human and/or Animals** Not Applicable.

**Informed Consent** Not Applicable.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

## References

1. Aishwarya K, Asuntha A. A survey on comparative study of lung nodules applying machine learning and deep learning techniques. Multimed Tools Appl. 2024;1-55.

2. Al Mohammad B, Brennan PC, Mello-Thoms C. A review of lung cancer screening and the role of computer-aided detection. Clini Radiol. 2017;72(6):433-42.

3. Armato III SG, McLennan G, Bidaut L, McNitt-Gray MF, Meyer CR, Reeves AP, Zhao B, Aberle DR, Henschke CI, Hoffman EA, et al. The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Med Phys. 2011.

4. Berens M, van der Zugten R, de Kaste M, Manders J, Zuidhof G. Znet-lung nodule detection. 2016.

5. Chen B, Liu Y, Zhang Z, Lu G, Kong AWK. Transattunet: multi-level attention-guided u-net with transformer for medical image segmentation. IEEE Trans Emerg Top Comput Intell. 2023.

6. Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille AL, Zhou Y. Transunet: Transformers make strong encoders for medical image segmentation. 2021. arXiv preprint arXiv:2102.04306.

7. Chen Y, Cao P, Dou L, Yang J. An end-to-end framework for pulmonary nodule detection and false positive reduction from ct images. In: The Fourth International Symposium on Image Computing and Digital Medicine. 2020;156-162.

8. Chen Y, Li J, Xiao H, Jin X, Yan S, Feng J. Dual path networks. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, editors. Advances in neural information processing systems, vol. 30. New York: Curran Associates Inc; 2017.

9. Clevert D-A, Unterthiner T, Hochreiter S. Fast and accurate deep network learning by exponential linear units (ELUs). In: International Conference on Learning Representations. 2016.

10. Ding J, Li A, Hu Z, Wang L. Accurate pulmonary nodule detection in computed tomography images using deep convolutional neural networks. In: Descoteaux M, Maier-Hein L, Franz A, Janin P, Collins DL, Duchesne S, editors. Medical Image Computing and Computer Assisted Intervention - MICCAI 2017. Berlin: Springer International Publishing; 2017.

11. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations. 2020.

12. Dou Q, Chen H, Yu L, Qin J, Heng P-A. Multilevel contextual 3-d cnns for false positive reduction in pulmonary nodule detection. IEEE Trans Biomed Eng. 2016;64(7):1558-67.

13. Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat 2001;1189-1232.

14. Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings. 2011.

15. Gu J, Wang F, Qi Y, Sun Z, Tian Z, Zhang Y. Retrieval method of pulmonary nodule images based on multi-scale convolution feature fusion. J Comput Appl. 2020;40(2):561-5.

16. Haibo L, Shanli T, Shuang S, Haoran L. An improved yolov3 algorithm for pulmonary nodule detection. In: 2021 IEEE 4th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), volume 4, pages 1068-1072. IEEE. 2021.

17. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. 2015;1026-1034.

18. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016;770-778.

19. Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018;7132-7141.

20. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017;4700-4708.

21. Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

22. Huang W, Hu L. Using a noisy u-net for detecting lung nodule candidates. IEEE Access. 2019;7:67905-15.

23. Jacobs C, van Rikxoort EM, Murphy K, Prokop M, Schaefer-Prokop CM, van Ginneken B. Computer-aided detection of pulmonary nodules: a comparative study using the public lide/idri database. Eur Radiol. 2016;26(7):2139-47.

24. Kopuklu O, Kose N, Gunduz A, Rigoll G. Resource efficient 3d convolutional neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019.

25. Larici AR, Farchione A, Franchi P, Ciliberto M, Cicchetti G, Calandriello L, Del Ciello A, Bonomo L. Lung nodules: size still matters. Eur Respirat Rev. 2017;26(146).

26. Li Y, Fan Y. Deepseed: 3d squeeze-and-excitation encoder-decoder convolutional neural networks for pulmonary nodule processing systems, vol. 30. New York: Curran Associates Inc; 2017.
detection. In: IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1866–1869. IEEE, 2020.

27. Liao F, Liang M, Li Z, Hu X, Song S. Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky noisy-or network. IEEE Trans Neural Netw Learn Syst. 2019;30(11):3484–95.

28. Lin T-Y, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. 2017;2980–2988.

29. Lu X, Chang EY, Hsu C-n, Du J, Gentili A. Multi-classification study of the tuberculosis with 3d cbam-resnet and efficientnet. In CLEF2021 Working Notes. CEUR Workshop Proceedings, Bucharest, Romania, CEUR-WS. 2021. http://ceur-ws.org. (September 21-24 2021).

30. Ma-Jiajun, Yuan G, Guo C, Gang X, Zheng M. Sw-unet: a u-net fusing sliding window transformer block with cnn for segmentation of lung nodules. Front Med. 2023;10:1273441.

31. Ma-Xiaopu, Song H, Jia X, Wang Z. An improved v-net lung nodule segmentation model based on pixel threshold separation and attention mechanism. Sci Rep. 2024;14(1):4743.

32. Maas AL, Hannun AY, Ng AY, et al. Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, page 3. Citeseer. 2013.

33. Matsumoto S, Ohno Y, Aoki T, Yamagata H, Nogami M, Matsumoto K, Yamashita Y, Sugimura K. Computer-aided detection of lung nodules on multidetector ct in concurrent-reader and second-reader modes: a comparative study. Eur J Radiol. 2013;82(8):1332–7.

34. Moshkov N, Mathe B, Kertesz-Farkas A, Hollandi R, Horvath P. Test-time augmentation for deep learning-based cell segmentation on microscopy images. Sci Rep. 2020;10(1):1–7.

35. Nawshad MA, Shami UA, Sajid S, Fraz MM. Attention based residual network for effective detection of covid-19 and viral pneumonia. In: 2021 International Conference on Digital Futures and Transformative Technologies (ICoDT2), pages 1–7. IEEE. 2021.

36. Park J, Woo S, Lee J-Y, Kweon I-S. Bam: Bottleneck attention module. In: British Machine Vision Conference (BMVC). British Machine Vision Association (BMVA). 2018.

37. Pearl J. Probabilistic reasoning in intelligent systems: networks of plausible inference. Amsterdam: Elsevier; 2014.

38. Polat G, Halici U, Dogrusoz YS. False positive reduction in lung computed tomography images using convolutional neural networks. 2018. arXiv preprint arXiv:1811.01424.

39. Ren S, He K, Girshick R, Sun J. Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inform Process Syst. 2015;28:91–9.

40. Riquelme D, Akhloufi MA. Deep learning for lung cancer nodules detection and classification in ct scans. AI. 2020;1(1):28–67.

41. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. 2015; 234–241. Springer.

42. Sangeroki BA, Cenggoro TW. A fast and accurate model of thoracic disease detection by integrating attention mechanism to a lightweight convolutional neural network. Proc Comput Sci. 2021;179:112–8.

43. Setio AAA, Traverso A, De Bel T, Berens MS, Van Den Bogaard C, Cerello P, Chen H, Dou Q, Fantacci ME, Geurts B, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge. Med Image Anal. 2017.

44. Shen L-H, Wang X-H, Gao M-X, Li B. Classification of benign-malignant pulmonary nodules based on multi-view improved dense network. In: International Conference on Intelligent Computing. 2021; 582–593. Springer.

45. Shi J. Lung nodule detection using convolutional neural networks. Electrical Engineering and Computer Sciences. Berkeley: University of California at Berkeley; 2018.

46. Shrivastava A, Gupta A, Girshick R. Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016;761–769.

47. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. arXiv preprint arXiv: 1409.1556.

48. Soviany P, Ionescu R. Optimizing the trade-off between single-stage and two-stage deep object detectors using image difficulty prediction. In: SYNASC: IEEE. 2018.

49. Sun L, Wang Z, Pu H, Yuan G, Guo L, Pu T, Peng Z. Attention-embedded complementary-stream cnn for false positive reduction in pulmonary nodule detection. Comput Biol Med. 2021;133:104357.

50. Tan J, Huo Y, Liang Z, Li L. A fast automatic juxta-pleural lung nodule detection framework using convolutional neural networks and vote algorithm. In: International Workshop on Patch-based Techniques in Medical Imaging. 2018;85–92. Springer.

51. Tang H, Kim DR, Xie X. Automated pulmonary nodule detection using 3d deep convolutional neural networks. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 2018;523–526. IEEE.

52. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Advances in neural information processing systems. 2017;5998–6008.

53. Wang Q, Shen F, Shen L, Huang J, Sheng W. Lung nodule detection in ct images using a raw patch-based convolutional neural network. J Digit Imaging. 2019;32(6):971–9.

54. Wang W, Chen C, Ding M, Yu H, Zha S, Li J. Transbts: multi-modal brain tumor segmentation using transformer. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021;109–119. Springer.

55. Wei C, Yang F, Zhang X, Xu X, Qiao X. Mau-net: multiple attention 3d u-net for lung cancer segmentation on ct images. Proc Comput Sci. 2021;192:543–52.

56. Woo S, Park J, Lee J-Y, Kweon IS. Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), 2018;3–19.

57. Woźniak M, Połap D, Capizzi G, Sciuto GL, Kośmider L, Frankiewicz K. Small lung nodules detection based on local variance analysis and probabilistic neural network. Comput Methods Prog Biomed. 2018;161:173–80.

58. Xie S, Girshick R, Dollár P, Tu Z, He K. Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017;1492–1500.

59. Xie Z. 3d region proposal u-net with dense and residual learning for lung nodule detection. Accessed: 2024-07-11 2017.

60. Yu M, Cai L, Gao L, Gao J. Amplification method of lung nodule data based on dcgan generation algorithm. In: International Conference of Pioneering Computer Scientists, Engineers and Educators. 2020; 563–576. Springer.

61. Yuan L, Chen Y, Wang T, Yu W, Shi Y, Jiang Z, Tay FE, Feng J, Yan S. Tokens-to-token vit: Training vision transformers from scratch on imagenet. 2021. arXiv preprint arXiv:2101.11986.

62. Zhang G, Yang Z, Gong L, Jiang S, Wang L. Classification of benign and malignant lung nodules from ct images based on hybrid features. Phys Med Biol. 2019;64(12): 125011.

63. Zhao B, Gamsu G, Ginsberg MS, Jiang L, Schwartz LH. Automatic detection of small lung nodules on ct utilizing
a local density maximum algorithm. J Appl Clin Med Phys. 2003;4(3):248-60.

64. Zheng S, Lu J, Zhao H, Zhu X, Luo Z, Wang Y, Fu Y, Feng J, Xiang T, Torr PH, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021;6881-6890.

65. Zhu W, Liu C, Fan W, Xie X. Deeplung: deep 3d dual path nets for automated pulmonary nodule detection and classification. In

IEEE Winter Conference on Applications of Computer Vision (WACV). 2018; 673-681.

**Publisher's Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.