Learning Dictionary for Visual Attention
Authors: Yingjie Liu
Xuan Liu
Hui Yu
Xuan Tang
Xian Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: dictionary learning
attention
transformer
computer vision
point cloud

Abstract: Recently, the attention mechanism has shown outstanding competence in capturing global structure information and long-range relationships within data, thus enhancing the performance of deep vision models on various computer vision tasks. In this work, we propose a novel dictionary learning-based attention (\textit{Dic-Attn}) module, which models this issue as a decomposition and reconstruction problem with the sparsity prior, inspired by sparse coding in the human visual perception system. The proposed \textit{Dic-Attn} module decomposes the input into a dictionary and corresponding sparse representations, allowing for the disentanglement of underlying nonlinear structural information in visual data and the reconstruction of an attention embedding. By applying transformation operations in the spatial and channel domains, the module dynamically selects the dictionary's atoms and sparse representations. Finally, the updated dictionary and sparse representations capture the global contextual information and reconstruct the attention maps. The proposed \textit{Dic-Attn} module is designed with plug-and-play compatibility, allowing for integration into deep attention encoders. Our approach offers an intuitive and elegant means to exploit the discriminative information from data, promoting visual attention construction. Extensive experimental results on various computer vision tasks, e.g., image and point cloud classification, validate that our method achieves promising performance, and shows a strong competitive comparison with state-of-the-art attention methods.

Learning Dictionary for Visual Attention
Yingjie Liu'
Xuan Liu?
Hui Yu?
Xuan Tang'
Xian Wei
East China Normal University
FJJRSM, Chinese Academy of Sciences
Abstract
Recently, the attention mechanism has shown outstanding competence in capturing global structure information and long-range relationships within data, thus enhanc- ing the performance of deep vision models on various computer vision tasks. In this work, we propose a novel alternative dictionary learning-based attention (Dic-Attn) module, which models this issue as decomposition and reconstruction problem with the sparsity prior, inspired by sparse coding the human visual perception system. The proposed Dic-Artn module decomposes the input into a dictionary and corresponding sparse representations, allowing for the disentanglement of underlying nonlinear structural information in visual data and the reconstruction of an attention embedding. By applying transformation operations in the spatial and channel domains, the module dynamically selects the dictionary's 's atoms and sparse representations. Finally, the updated dictionary and sparse representations capture the global contextual information and reconstruct the attention maps. The proposed Dic-Attn module is designed with plug-and-pla, compatibility and fa- cilitates integration into deep attention encoders. Our approach offers an intuitive and elegant means to exploit the discriminative information from data, promoting visual attention construction. Extensive experimental results on various computer vision tasks, e.g., image and point cloud classification, validate that our method achieves promising performance, and shows strong competitive comparison with state-of- tfate-of-the-an ft-thharrt art attention methods.
Introduction
Visual perception plays a critical role in obtaining external information. However, processing massive visual information in the real world is challenging for both human brains and computers. Modern psychology and cognitive neuroscience find that the visual attention mechanism is one of the essential keys a human's good cognitive ability 43 enabling selective and sparse activation of neurons in response to input. In the field of computer vision (CV), various attention methods have also been developed, with successful applications in lots of Convolutional Neural Networks and Transformers.
48 46 15]. CV attention methods utilize various calculation functions to generate attention maps, but they can be seen as a general form (refer to Sec. 3...3.T)
revealing great potential in visual tasks 4 32 S
[1 41 10). Researchers have recognized several advantages of the attention mechanism, e.g., capturing global long-term information and extracting hierarchical data structures (6). Moreover, recent research has also highlighted the potential of leveraging sparsity to develop attention algorithms
31 62 23). However, there are still pending questions regarding the efficiency and effectiveness of current attention methods when dealing with regular image features and non-Euclidean point cloud data [57 35 9. Considering the significant memory costs and time complexity [42) associated with
current attention methods, it is crucial to explore possibilities of optimizing their internal operations or re-evaluating the current form, especially in the view of sparsity.
"Corresponding author: xwei @seisei....o ecnu. edu.cn en
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this context, recalling the previous works and considering the original motivation behind in- troducing attention methods remind us of the dictionary learning/sparse coding algorithm, which holds importance but is often overlooked in present discussions. The dictionary learning algorithm decomposes visual data into a dictionary basis and their corresponding sparse representationsscodee, Under different conditions, the sparsity of sparse code can be adapted: code with higher sparsity (more zero elements) mostly corresponds to features smaller scale, higher roughness, or higher hierarchy. Furthermore, this process echoes the nonlinear effects in complex cells and the sparse coding strategies for sensory information the primary visual cortex and advanced visual region 19 56]. It has been observed that the structure in sparse domain could make the hidden patterns more prominent and easier to be captured [29 52 55 60). All these motivate us to develop an alternative novel attention form, i.c. Dictionary learning- based Attention (Dic-Artm), that offers a visual focus for deep neural networks (DNNs). The Dic-Astn module can be conveniently used to replace existing attention modules, and compatibly explain their working principles.
34
The Dic-Attm module employs two selection matrices to exploit channel and spatial attention from dictionary atoms. By selecting the appropriate basis and re-locating the coordinates, can uncover hidden visual attention information by reconstructing the attention map using the updated basis (dictionary) and coordinates (corresponding sparse representations), Note that, the sparse represen- tations in our proposed module originate from the non-linear decomposition of visual data and are subsequently reconstructed into the attention matrix. The sparsity property of the attention matrix is induced naturally by the sparse prior [52] 550 60) and is related to the visual attention area within the images. The main contributions of this paper summarized follows:
1. We propose a novel visual attention module, namely, the Dictionary learning-basse based At- tention (Dic-Art) module, which is based on dictionary learning and provides better interpretability of providing contextual information.
2. We demonstrate that the proposed module has great potential in disentangling non-linear structural information by employing a masked dictionary and transformed sparse repre- sentations. We construct deep attention stage composed of the proposed modules. Each module at different depths, ranging from shallow to deep, dynamically updates dictionar- ies and sparse representations through spatial and channel transformations, step-wise-ly reconstructing the final attention map.
3. We apply the proposed module various visual models since designed be plug- and-play, achieving better performance than various existing attention modules on image classification, semantic segmentation, and point cloud classification, with fewer parameters and faster computation.
Distionary-Based Attention Block(s)
Chousel
Dictionary
Normeaization
X
Decompasition
Reconstractonn
di
Spatial
Sparse Representation
XT, ggne
Tix
Attencion Output
Dietionary-Base Based Attention Module
Figure 1: The pipeline of our proposed Dic-Attn block. First, input features X into the Dic-Atth block. Within the Dic-Art block, the inner Dic-Atm module decomposes the normalized input X into a dictionary and corresponding sparse representations, which will be transformed in the spatial and channel domains. The Dic-Arth module can dynamically adjust the weights to promote the visual focus on influential elements. This process is both data and task-driven, thereby enhancing the effectiveness of the attention mechanism. Finally, the output of the Dic-Alin block is obtained by summing the input with the Attention Output. The Dic-Attn block can be stacked layer by layer to form a deep attention model, enabling capturing and integrating visual attention hierarchically across multiple levels.

---

2 Related Work
Attention Mechanism in Computer Vision Tasks, CV Attention methods simulate the phe- nomenon of the human visual system, where selective attention helps neural system analyze and understand complex scenes more efficiently. Recurrent Model [3 is pioneering work that develops the attention mechanism in DNNs. Later, various attention methods are proposed. Some of them try to improve the pre-processing method of the input images [8] [4]. while others modify the way of
obtaining the attention matrix, resulting in a series of variants, e.g., spatial transformer network (STN) [16.. self-attention (SA) 146], soneeeee-andexciiaaiiii
network (SENet) [14), dual attention (DA)
module [5) double attention (A²) module 4,, hamburger (HAM) module [6. etc. |1 48 25 39
The SA module is proposed in [46] and first introduced to computer vision in [49] It achieved great success in various vision tasks and made the Vision Transformer (ViT) successful in academia. [6] constructs the long-term global context learning
low-rank recovery problem and proposes the matrix decomposition (MD) module, focusing only on the low-rank property prior assumption of feature representations. It may have limitations in fully capturing the complexity and richness of global structure information in the data. Most recent related work (39 was inspired by the two promi- nent features of the human visual attention mechanism, i.e., recurrency and sparsity. The so-called recurrency attribute in 39 naturally comes from the ordinary differential equation description of the sparse coding update process, which is actually inside the attention module and the sparse repre- sentations require a multi-step update. It also lacks the analysis of attention modules with different depths. Moreover, the residual connections between multi-layer attention modules still remain under explanation. 23 measures sparsity as the percentage of nonzero entries in the activation maps of each layer. It finds that sparsity emerges in the parameters of all layers of the Transformer. However, it does not explicitly state whether the output of the attention module exhibits sparsity or not. Some studies [37 51 17 [3] have tried to post-hoc-interper these developed attention methods, and utilize
white-box approaches to do the analysis 13 41 58
Dictionary Learning in Computer Vision Tasks. Sparsity is an important and noticeable prior in the field of computer vision, e.g., the statistically redundant property of natural images is served for dictionary learning. Hence, image sample is reconstructed with the dictionary D and its corresponding sparse code Φ) i.e., samples can be decomposed into a dictionary and corresponding sparse representations. The dictionary D captures representative and critical features from the inputs and reveals the interference of insignificant information. Dictionary learning has been extensively studied and verified to be an outstanding approach for exploiting the underlying sparse structures hidden in images [29] 52 50 55 60), efficiently transferring and expressing information from high- dimensional image data into sparse space. It plays an important role in various computer vision tasks, e.g., image reconstruction [I] [29 3 38 64], image classification (52 47 [5] 24 60), visual tracking [61] and others. [39]. Note that, compared with non-negative matrix factorization and vector quantization applied in MD module [6] mentioned above, the dictionary learning algorithm performs
better in the case of natural images 226 Technical details about how we learn the dictionary is shown
in Sec. 3.2
3 Method
In this section, we present the proposed dictionary learning-based visual attention (Dic-Attt) method, First, we give a brief review and summary of the CV attention form and dictionary learning. Then, we describe the main idea of the Dic-Art module to obtain visual attention. Finally, we analyze the core concept behind the Dic-Arth module. The overall structure of the proposed Dic-Arta method is illustrated in Fig. [
3.1 Attention Form
Generally speaking, we can summarize CV attention form ||
Input :X,
Attention Module (X))
Attention Block (X)), X).
3

---

The input features X are firstly pre- peeppreeessdd Subsequently. the attention module obtains an attention map of the imput. Finally, g(-) introduced further connect attention map and input features, e.g.. the normalization operation, drop operation, skip-connection, etc.. All these constitute an Attention Block. Among them, the function f(X)= X r> Attention Output in the attention module the most critical mapping. Attention blocks can be added to the middle of the network, or form a ford-forward network. For example, in the ViT encoder with depth N, there will be attention blocks. Take VIT as an example, suppose there is feature tensor Z with shape b x h w, where n denotes the number of channels. The feature tensor is vectorized and the last two dimensions are transposed. Hence, the attention block takes sequence of image features as input, i.e,, X E Rxxax, where s = h x w. The Gelf-Attention module is introduced and subsequently outputs spatial attention map, essentially operating in the spatial domain. The formulation of the standard Self-Attentio module in ViT [46] is as follows:
the spatial domain
Q K, Vl = W WK W.X),
A = softmar( W scale (QK"),
Attention Output AV,
where X E R is the input, Q R K Rxxxx and V E R denote the query matrix, the key matrix, and the value matrix, respectively. They are obtained by full-connected layers. Wq.Wk WK and Wy are all square matrices with the shape n x n. A represents the attention matrix. Finally, through the residual structure, i.e. skip-connection, attention blocks have the final output result, i.e., Attention Output X.
The implicit variables or the spatial attention map results of these attention modules can be visualized, e.g., traditional SA blocks in ViT [46]. and Axternal-Attention (EA) blocks [10]. These works indicate that the attention module focuses on the discriminative regions of each image. Recalling the CV attention form we summarized at the beginning of this section, most important function when obtaining visual attention in DNNs. In this paper, f(-) contains the process of dictionary learning, the transformation of both learned dictionary & corresponding sparse representations, as well as reconstruction, which is to be introduced in Sec.
find that f(-) is the
3.2 Dictionary Learning
Formally, given data X' := [mi, .... Fn] E IRnxs, the standard dictionary learning algorithm is to find a dictionary D and the corresponding sparse code $. The dictionary = [dl...., dR has shape of
k. the sparse code +- ( (.... ... Φx) E Rixe selects atoms from the dictionary reconstruct the input X. The objective function of dictionary learning can be formulated as follows:
argmin h(D) + D.
I - D&. 122 g(0)),
2
s.t. D S(n, k), E R
where h(D) = a d dy |/ denotes the regularization term of the dictionary. It leads to mutual incoherence between pair atoms in the dictionary. Besides, D E S(n, k) = (D E Bank diag(D" = J. Namely, the set of dictionaries is a product manifold of s times the (k (1-=1)-dmensional unit sphere, i.e., S(n, k) restrict all atoms d, E R2* to have unit norm g((b) usually utilizes norm constraint to control the sparsity of the sparse code &. The 4 norm regularization is popular way to replace 4n-norm by /i-orrm convex relaxation, known as the Lasso 44 regression. It has numerous variants, e.g., elastic net. Elastic net inherits the stability of Ridge and the sparsity brought by Lasso, for combining both regularization terms of I norm and la norm. Hence in this paper, we have g(Φ,) = 3 [| || ,3 and À are regularization weights.
Eq. can be regarded as iterative optimization problem, where dictionary D and sparse repre- sentation alternately updates. Each sparse code is an implicit function about the temporarily fixed dictionary D. With Eq. (2). the incoherent assumption of dictionary atoms and the sparse prior g(),) 50 the close solution of sparse representation can be expressed as follows:
कं DD AI) (D Bvx).
(3)

---

where A : fi fiE {1, sil.....sHee i + 0} denotes the set of indexes of the non-zero entries of the solution $ = (1 Dial DA is the subset of the dictionary in which the indexes of atoms fall into A. |A] denotes the sparsity of Φ VA E [=1]1A] carries the signs of Φ Technically, it is easy to realize the calculation of the above polynomial formula and obtain the sparse representations.
Constructing or learning a dictionary is also crucial for achieving the goal of highlighting significant objects while suppressing distracting noisy elements. In the early stage, the dictionary is constructed by Cosine functions, Fourier. Wavelets, Contourlets, Gabor, a set of complete bases, etc. These approaches were widely employed in signal processing due their mathematical simplicity. However, such a fixed dictionary is manually designed under some mathematical constraints and is not flexible enough to represent complex natural image structures. Recently, researchers have turned directly initializing and learning the dictionary from image data, which can also improve the sparsity of + and the performance of downstream tasks [52 60 64). The dictionary contains prototype underlying features of vision data, based on which each sample can find its unique optimal sparse solution.
3.3
The Proposed Dic-Atta for Visual Attention
3.3.1 Dic-Alfn Method
The Dic-Arth module first decomposes the input into a dictionary and sparse representation. Then its primary objective is to provide visual attention to the model by reconstructing the attention map with residuals. Its process is follows,
X DΦ,
DWD,
(4)
A softmaz( W acate W
Attention Output = VA,
where X = XT,X, XT ..... x E R Each atom d; E IK" in the dictionary D E Ra*k will be we-ghttt by WD E I2 The elements of WD R are trainable mask weights for dictionary atoms.
Wa + E Beak denotes the selection mask matrix for sparse representations R bxxxxx
+ = [ 2%
Wecale denotes the scaling matrix. Note that k = an, a is the
hyper-parameterr
Firstly, the dictionary is learned on the entire training-datases aiming to be adaptive to all inputs X. It contains diverse and important features (in atoms) of the input, it can solve the sparse coding problem and obtain the corresponding code d. We have:
D ddf
dp
dgl [d]
db
d.
therefore,
Φ m
[01
Φ2
Φ = [
22
4t)"
X11
Xin
df
di
dí
X21
X22
X2n
dh
d
di
Xm
X,1
X,2
X
d do d
(5)
Xij d;
(d
where X.m denotes the m-th batch of X, 2m I...... H ( 31 E Rkx* di and di denotes the t-th column and the i-th row of D. The optimization process of dictionary learning maximizes the descriptiveness of the atoms de while minimizing the redundancy between original inputs X and @ Then, transformations WD and W. for dictionaries and sparse representations are introduced, respectively, which facilitate the integration of spatial and channel attention into the model by assigning weights in both the spatial and channel domains.
The diagonal transformed matrix WD and transformed matrix W are specifically designed to re-weight the dictionary and transform its corresponding sparse representations, respectively. Through
5

---

Algorithm 1: Dictionary Searning-Based Attention Module
Input: X E IR x depth N,
Parameter: D E S(n, k), W E B WD E R* W scale
Latent Variables: E R "
Output: Attention Output
# Initialize D E S(n, by randomly sampling from NN(,, 1) and atom-wise normalization;
# Initialize Wa. WD. W scatee with reference to kaiming-ini init method [12]
forward
## Update & by Eq. (3)
## Obtain Attention Output by Eq. 44:
# backward
## Update D. Wa. WD. W scate by minimizing loss function of the network: ;
the transformation of the dictionary, spatial attention is introduced during the reconstruction process, effectively leveraging the discriminative power of atoms [18 64 (7). Different sizes of the diagonal elements in the WD impose weights on different channels d in the reconstruction process. In view of sparse representations/cooees, the Dic-Art module employs sparse prior and exploits the intrinsic underlying structure of features. The sparsity involved in the attention module arises spontaneously from the statistically redundant natural images. The sparse representation serves as the coordinates of the dictionary. These sparse representations indicate the importance or relevance of each atom in the dictionary for reconstructing the input features. Importantly, the attention similarity between different tokens grid features in X is inherently encoded within the dictionary and the sparse representation. W4. modifies individual elements and W scale applies unified weights within each column of &, respectively. These transformation parameters are driven by the task objective function and are updated through the backpropagation process.
Algorithm| 1 summarizes the algorithm flow of the Dic-Art module. Furthermore, it is worth noting that constructing a deep Dic-Artn vision model is feasible. All these transformation matrices are trained in mack-forward manner and serve crucial link in connecting the dictionary, sparse representations, and reconstructed attention maps. To enhance the stability and effectiveness of the Dic-Att module, it is recommended include normalization step before the attention module as [46] Additionally, a residual connection between the input X and the Attention Ouptut facilitates the flow of information. This results in the formation of the Dic-Attn block, illustrated in Fig. []
In summary, the Dic-Attn module utilizes a data-initialized and task-driven dictionary strategy. which captures global information and serves comprehensive resource containing essential data constituents. The process of reconstructing attention maps fully capitalizes on the dictionary learning sparse coding approach, the ability to extract both channel and spatial information from nonlinear features. The Dic-Artn module has a simpler structure with fewer parameters and elegant internal operations while achieving higher performance. Moreover, the Dic-Attn module is convenient plug-and-play attention module that can be easily integrated into various vision networks.
3.3.2 General Dic-Attn-Based Vision Transformer
Herein, we give guide on how to build a Dic-Attn-based Vision Transformer and analyze the role of the proposed Dic-Art method. The model architecture can be divided into three main parts: the preprocessing stage, the deep attention stage, and the post-processing stage for downstream tasks.
Firstly, multi-layer convolution and linear operations are employed to extract primary visual features from the natural images. Then, multiple concatenated Dic-Atin blocks form the deep attention stage, generating the final attention map step by step. To conveniently introduce this process, we further summarize the Dic-Amm form as follows
Reconstruction (D, ( (+))
(6)
where N-(-) denotes non-linear scaling function. D the learned dictionary with re-weighted atoms. (L can be treated response prediction model [56). We assume that the input of (1 1)-layer attention block is Attention Output, + X,, i.e., the output of the previous layer's 's attention block. Learning- based dictionary D of each Ait-Att module groups significant features of different objects

---

in the input, and the most salient objects are preserved while those irrelevant are suppressed by further selecting sparse representations. The output of each layer has a common intersection as the input X of the first layer, which is task-independent bias for dictionary learning in Dic-Attn modules of any depth. Moreover, the output can be further considered as consisting of two components, one of which is the 1-layer module's attention area, and the other component represents the ignored part. The first component will affect deep attention modules layer by layer, and adjust the task-driven selection matrix through backpropagation. Hence, our proposed Dic-Attn module of shallow depth can bring selective biases with a lask-independent impact on deeper attention modules.
The proposed Dic-Atth method, based on feature decomposition and sparse prior, offers a white-box novel strategy for obtaining visual attention, which may be able be applied to scenarios with more stringent security requirements. Generally, in the visual perceptual system, selective attention is primarily driven by strong stimulus signals and attention to specific regions. Selective attention can be understood from three main aspects: bottom-up voluntary attention, involuntary attention, and "selection history" attention. Involuntary attention, similar the task-driven model in computer vision, is obtained through top-down rewards. "selection history" refers to the influence of previously attended regions or objects on future attention allocation. Meanwhile, the selective lingering biases have a task-independent impact the subsequent attention priority. In biology, it means that the nerve has better plasticity. While in the deep neural network, the structure of the multi-layer attention module is not just capable to acquire features with higher semantics. Shallow-depth attention modules capture data prototype features and affect deeper attention modules. It will make models with greater capacity, better generalization, and robustness. The multi-layer attention module structure in the Dic-Attn module goes beyond acquiring features with higher semantics. Shallow-depth attention modules capture data prototype features and influence deeper attention modules, resulting in models with increased capacity, better generalization, and improved robustness.
In summary, we demonstrate the effectiveness of the Dic-Attn module in capturing visual attention, enhancing visual model performance. We also provide the interpretability for deeper insights into the attention mechanism.
4 Experiments
In this section, we evaluate the per- formance of proposed Dic-Attn on 6 benchmarks. Firstly, we conduct ablation experiments on CIFAR- 10 dataset, described in Sec. 1.
91.6
91.5
Accuragy 3
91.4
Acciracy
91.4
91.3
Subsequently, we compare the com-
91.2
91.2
putational costs. In Sec. 4.2 we evaluate the performance of the Dic- Att module across various visual tasks, such as point cloud classi- fication, image segmentation, and image classification. We compare the accuracy and robustness of the proposed method with several no- table attention modules. More ex- periments and the code are available the Supplementary Material.
91.1
91
91
Depth(
Dimension(;) )
Encoder depth. The deeper the better, but fewer layers can also obtain good accuracy.
(b) Dictionary dimension k. An over-completed dictionary can im- prove performance.
Figure 2: backhone) on CIFAR- 10 dataset.
The Dic-Attn module ablation experiments (with ViT
4.1 Ablation Study and Computational Costs
We explore the influence of the hyper- pppee-parameters. i.e., the second dimension & of the dictionary and the number of attention blocks (depth of encoder). Fig. iillustrattss the significance of selecting appropriate values for and k. A dictionary with higher dimension & creates a comprehensive feature space for images, which enhances the performance of the Vision Transformer. As the depth 1 increases, the accuracy initially improves but starts to decline after reaching seven in CIFAR-10. The phenomenon of rising first can be attributed to the fact that deeper networks have the capability to capture more intricate relationships between inputs and outputs. However, they are also prone to
7

---

overfitting, particularly when the training dataset is relatively small, such as in the case of training from scratch using between depth and complexity when designing neural network, rather than assuming that deeper is always better.
the CIFAR-IU 10 dataset. Therefore, is important carefully consider the trade-offs
We evaluate the performance of the Dic-Attn module, and compare it with the SA module and notable variants, as presented in Tab. [ The results show that the proposed Dic-Artn module is more lightweight and has a lower computational burden, The total number of parameters (Params.) of the SA module is related n, which equal to 3n 2 For our proposed Dic-Arm module, the Params. is counted as k(n + k + 1), where Params. of D is nk, Params. of W & is k2 and Params. of Wy is k. Since &o an, can be inferred that Inn + k + 1) always less than n² when (0, an) ) n (a2. +oc). 01 and a2 are solutions of the quadratic equation no? + (n + 1)a - n² with variable a. Note that, always has real number solutions.
Table 1: Comparison of the efficiency of various attention modules. Indicators: Number of parameters (Params.), compared to Self-Attention and its variants.
operations (MACs) and training/inference time cost
Medhods/Indicators
Params. (M)
MACs. (G)
GPU Train / Inference Time (ms)
SA
46
1.00
292.00
242.00 82.20
DA
5
4.82
79.50
72.60 / 64.40
A
4
1.01
25.70
22.90 8:00
59
0.75
79.5
71.00 / 22.60
HAM
0,50
17.60
15.60 7.70
Dic-Attn (Ours)
0.60
20.00
78.00 / 24.70
Input: tensor with shape 1 x 512 128 128.
4.2 Computer Vision Tasks
We evaluate the performances of our proposed Dic-Altn module on public image datasets and point cloud datasets, including CIFAR-10. CIFAR-100[20], ModelNer40 (53), ScanObjectNN [45). and ADE20K [63]. Note that, to mitigate the impact of various complex factors that may arise from pre-training and fine-tuning processes, all models in this paper were trained from scratch, without pre- praitinin and adopted the same data augmentation method. Additionally, unless explicitly stated, the hyperparameter settings remained consistent across all experiments. More implementation settings are described in detail the Supplementary Material.
4.2.1 Image Classification
CIFAR- 10 dataset composed of 10 categories 60000 color images with the size of x2 32. CIFAR- 100 dataset has 100 classes. Both of CIFAR-10 10 and CIFAR-100 contain 50000 training images and 10000 test images. The number of images in each category is equal, so 6000 images per class in CIFAR- 10 and 600 in CIFAR- 100, respectively.
The classification results on image datasets are shown in Tab. 2 The image classification accuracy of ViT can be improved from 0.64% to 2.12% by changing the SA module to Dic-Att module.
In Tab. 3. we report the model performance on adversarial robustness under the Fast Gradient Sign Method (FGSM) 7 and Projected Gradient Descent (PGD) (28). We set the FGSM attack with the perturbation magnitude
255 For PGD, we set 255 iteration numbers t 10, and step size 2 We evaluate the model performance by using Attack Accuracy, including the FGSM accuracy and PGD accuracy on classifying images corrupted by adversarial attack FGSM and PGD, respectively. Note that we do not apply adversarial training here, but only clean samples train RVT networks [00] In this case, the proposed Dic-Attn module is still competitive.
255
4.2.2 Point Cloud Classification
ModelNet40 and ScanObjectNN datasets are widely used benchmarks for point cloud analysis.
ModelNet40 dataset comprises 12311 CAD-generated meshes categorized into 40 classes. Among them, 9843 meshes are used for training, while the remaining 2468 meshes are reserved for testing. ScanObjectNN is real-world dataset, that contains 15000 objects, divided into 15 and 2902 unique object instances. For all experiments conducted, only the coordinates zyzz of objects are utilized. In the case of ModelNet40, all basclines tested with 1024 input points.
8

---

Table 2: Image Classification Topl-Accuracy
Table 3: Robustness Evaluation Results on CIFAR-
100 dataset. Comparison of our approach with backbones and other attention baseline methods, against various adversarial attacks.
(%). The overall classification accuracy over all classes is for evaluation metrics.
Model
Datasets
(From Scrach)
CIFAR-10
CIFAR-T00
#Prram(M)
Model
Attacks
#Param(M)
VII (SA)
90.91
66.67
22.74
(From Scrach)
FGSM
PGD
Clean
Swin-S (WSA)
89.74
55.82
22.25
RVT (SA)
0.20
0.10
64.80
8.29
(Dic-Art))
91.35
68.79
19.33
RVT (VARS-D)
10.58
3.80
62.20
7.68
RVT (Dic-Attn)
10.00
4.30
55.93
7.48
The pre-processing of cloud points is performed according is encoded as a tixed-shape feature tensor to input the attention module, which preserves and incorporates the original point clouds' position correlation, etc. The classification results on point cloud datasets are shown in Tab. 4 It is evident that our proposed Dic-Attn module significantly enhances the classification performance of the Point Cloud Transformer (PCT) model 9 The average point cloud classification results achieved with our proposed module are highly competitive and demonstrate notable improvement over existing approaches. Our proposed Dic-Artn module is able to compute an attention map for the point cloud Transformer. In addition, all these indicate that our proposed Dic-Attn module has the advantage of processing complicated data and has the potential to measure the topological attention relationships of the underlying geometric structure.
Note that the point cloud data
Table 4: Point Cloud Classification task Topl-
Accuracy (%). The number of points is set to 1024.
Model
Datasets
SA
(From Scrach)
ModelNer40
ScanObjectNN
PointNet++
36
91.90
84.30
Input
PoinCCNN
22
91.70
85,50
PCT
92.95
80.60
Point-BERt
93.19
88.10
Dic-Attn
Point-MAA
93.80
88.30
PCT (Dic-Attn)
94.60
88.96
Figure 3: The comparison of attention maps generated by SA module and Dic-Att mod- ule (with backbone Segmenter, experiment on ADE20K dataset).
Point-MAR (Dic-Attn)
94.46
89.41
4.2.3 Image Segmentation
ADE20K dataset (63] contains total of 150 semantic categories, and can be used for scene perception, parsing, segmentation, multi-object recognition, and semantic understanding, which is one of the most challenging semantic segmentation datasets. The training set contains 20210 images. There about 2000 and 3352 images the validation and test set, respectively.
The performance of the Dic-Atm module is verified in the semantic segmentation task with the backbone Segmenter [40, which is a competitive and representative transformer- semantic segmentation model. We train Segmenter (SA) and Segmenter (Dic-Attn) from scratch on ADE20K dataset. Note that, Segmenter (Dic-Attn) denotes the proposed Dic-Attn module replacing the original SA module in the Segmenter backbone. The mean Intersection over Union (mloU) and the mean Pixel Accuracy (mPA) of Segmenter (Dic-Attn) are 1.00% and 1.25% higher than that of Segmenter (SA), respectively.
Furthermore, the two image examples and their corresponding segmentation attention map results, as shown in Fig. intuitively indicate that our proposed Dic-Artn module highly promotes the ability of the segmentation model distinguish objects. The learned attention maps are also more similar to human visual attention, which focus on referred objects, e.g., building, sky, door, cabinet, etc. The learned dictionary the Dic-Attn module contains more sufficient features, including the underlying nonlinear structure. All results above indicate that by fully extracting attention information in both the channel domain and spatial domain of feature space, the attention maps for different labeled objects can be reconstructed more accurately.

---

5 Conclusions
In this work, a novel dictionary learning-boo attention module, namely Dic-Attn, was proposed. Dic-Art takes the advantage of dictionary learning mechanism, enabling effective exploration of the underlying nonlinear structure information. By dynamically selecting learned dictionary atoms and sparse representations through spatial and channel transformations, multi-layer Dic-Artn modules achieve the accurate reconstruction of attention maps and allow for the extraction of features at different semantic levels, enhancing the comprehensive representation of visual information. Experimental results in various visual tasks and vision models showed that the Dic-Artn module can seamlessly integrate into various vision networks, delivering competitive performance comparable to state-of-the-en methods. The Dic-Attn module is also a promising solution with competitive performance, compatibility, and computational efficiency. Further research will be devoted to the robustness of the module and its applications in more tasks, c.g., temporal prediction tasks in scientific and engineering fields. Conducting further investigation and validation of the diverse levels of semantics exhibited by dictionaries learned at different depths in natural language processing scenarios would also be valuable.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No.4213014 12, No.41901335), KartoBit Research Network (No.KRN2201CA) and the Shanghai Municipality's "S&T Innovation Action Plan" for the year 2023, Shanghai Natural Science Fund, General Pro- gram (No. Noo.232R14193000 Shanghai Trusted Industry Internet Software Collaborative Innovation Center, "Digital Silk Road" Shanghai International Joint Lab of Trustworthy Intelligent Software (N0.22510750100)
References
[1] Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions signal processing, 54(11):4311 43222, 2006.
[2] Behnam Babagholami- Mohaaadabaddd,, Amin Jourabloo, Mohammadreza Zolfaghari, and Mohammad T Manzuri Shalmani. Bayesian supervised dictionary learning. In UAI Application Workshops, pages 1-19. Citeseer, 2013.
[3] Ori Bryt and Michael Elad. Compression of facial images using the k-svd algorithm. Journal of Visual Communication and Image Representation, 19(4):270-222, 2008
[4] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A" 2-nets: Double attention networks. Advances in Neural Information Processing Systems, 31, 2018.
[5] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3146-3154, 2019.
[6] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In International Conference on Learning Representations. 2021.
[7] lan J. Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[8] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471.
PMLR, 2015.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199 2021.
[10] Meng- Hao Guo, Zheng-Nin Ning Liu, Tai- Jiang Mu, and Shi-Min Hu. Beyond celf-attention: External attention using two linear layers for visual tasks. arXiv preprint arXiv:2105.022. 023558.88...... 2021.
10

---

[11] Meng-Haa Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Tao Jiang, Tai-Jiang Mu, Song- Zhang, Ralph R. Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. CoRR, abs/2111.07622 2021.
[12] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026-1034, 2015.
[13] Dichao Hu. An introductory survey on attention mechanisms in nlp problems. Proceedings of SAI Intelligent Systems Conference, pages 432 448. Springer, 2019.
[14] Hu, Li Shen, and Gang Sun. Squeeze-and-exceie -xxiittttiionni networks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 7132-7141, 2018.
[15] Lun Huang, Wenmin Wang. Jie Chen, and Xiaoyong Wei. Attention on attention for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4633 0442-464,220 2019.
[16] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. Advances in Neural Information Processing Systems, 28, 2015.
[17] Sarthak Jain and Byron C. Wallace. Attention is not explanation. arXiv preprint arXiv:
2019.
[18] Zhuolin Jiang, Zhe Lin, and Larry S Davis. Learning a discriminative dictionary for sparse coding via label consistent k-svd. 2011 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1697-1704. IEEE, 2011.
[19] Yan Karklin and Michael $ Lewicki. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, 457(7225):83-866 2009.
[20] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Tech Report, University of Toronto, 2009.
[21] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation-maxiiizzaae maximization attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9167-9176, 2019.
[22] Yangyan Li, Rui Bu, Mingehao Sun, and Baoquan Chen. Pointenn. CoRR, hhss/1801.00/ .07791, 2018.
[23] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Cher, Felix Yu. Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2023.
[24] Xiao-Chen Lian, Zhiwei Li, Bao-Liang Lu, and Lei Zhang. Max-margin dictionary learning for multiclass image categorization. In European Conference Computer Vision, pages 157-170. Springer, 2010.
[25] Zhaoyang Liu, Limin Wang. Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13708-13715 13718,,718, 2021.
[26] G. Madhuri and Atul Negi. Discriminative dictionary learning based statistical methods. CoRR, abs/2111.09022 2021.
[27] G Madhuri and Atul Negi. Discriminative dictionary learning based on statistical methods. arXiv preprint arXiv:2111.0903 2021.
[28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Repre- sentations. ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings OpenReview.net, 2018
[29] Julien Mairal, Michael Elad, and Guillermo Sapiro. Sparse representation for color image restoration. IEEE Transactions on Image Processing, 17(1):53 69, 2007.
[30] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li. Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In IEEE/CVF Conference on Computer Vision and Pottern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 12032-12041. IEEE, 2022.
[31] Clara Meister, Stefan Lazov, Isabelle Augenstein, and Ryan Cotterell. Is sparse attention more interpretable? arXiv preprint arXiv:2106.011 010077 2021.

---

[32] Volodymyr Mnih, Nicolas Heess. Alex Graves, et al. Recurrent models of visual attention. Advances in Neural Information Processing Systems, 27, 2014.
[33] Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M Khapra, Balaji Vasan Srini- vasan, and Balaraman Ravindran, Towards transparent and explainable attention models, arXiv preprint arXiv:2004. 14243, 2020.
[34] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision Research, (113)33311-3322) 1997.
[35] Yatian Pang. Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoen- coders for point cloud self supervised learning. CoRR, abs/2203 066044, 2022.
[36] Charles Ruizhongtai Qi, Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5099-5108, 2017.
[37] Sofia Serrano and Noah A Smith. Is attention interpretable? arViv preprint arXiv:1906.0373: 2019.
[38] Ling Shao, Ruomei Yan, Xuelong Li, and Yan Liu. From heuristic optimization to dictionary learning: review and comprehensive comparison of image denoising algorithms. IEEE Transactions on Cybernetics, 44(7):1001-10133 2013.
[39] Baifeng Shi, Yale Song, Neel Joshi, Trevor Darrell, and Xin Wang, Visual attention emerges from recurrent sparse reconstruction. arXiv preprint arXiv:2204. 20t.:22244.192,,, 2022.
[40] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the TEEE/CVF International Conference on Computer Vision, pages 7262-7272, 2021.
[41] Jiahao Sun, Jie Jiang, and Yang Liu. An introductory survey on attention mechanisms in computer vision problems. In 2020 6th International Conference on Big Data and Information Analytics, pages 295-300. IEEE, 2020.
[42] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: survey. ACM Computing Surveys, 55(6):1-28, 2022.
[43] Jan Theeuwes. Visual selective attention: A thcoretical analysis. Acta psychologica, 83(2)993-155 154, 1993.
[44] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58( 11:26))222888 288, 1996.
[45] Mikaela Angelina Uy, Quang-Hics Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1588-1597, 2019. 2019.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[47] Andrew Wagner, John Wright, Arvind Ganesh, Zihan Zhou, and Yi Ma. Towards practical face recognition system: Robust registration and illumination by sparse representation. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pages 597-604. IEEE, 2009.
[48] Qilong Wang. Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1531-11539, 2020.
[49] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794-7803, 2018.
[50] Xian Wei, Hao Shen, and Martin Kleinsteuber. Trace quotient with sparsity priors for learning low dimensional image representations. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019.
12

---

[51] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint 2019.
[52] John Wright, Allen Y Yang. Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE Transactions Pattern Analysis and Machine Intelligence, 31(2):210-22 227, 2008.
[53] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1912-199 1920, 2015.
[54] Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[55] Meng Yang and Lei Zhang. Gabor feature based sparse representation for face recognition with gabor occlusion dictionary. In European Conference Computer Vision, pages 448-461. Springer, 2010.
[56] Takashi Yoshida and Kenichi Ohki. Natural images are reliably represented by sparse and variable populations of neurons in visual cortex. Nature communications, 11(1):1-99, 2020.
[57] Xumin Yu, Lulu Tang. Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 19291-19300. IEEE, 2022.
[58] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition transformer factors. arXiv preprint arXiv:2 15949, 2021.
[59] Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xic. Co-occurrent features semantic segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 548-557, 2019.
[60] Qiang Zhang and Baoxin Li. Discriminative k-svd for dictionary learning in face recognition. 2010 IEEE computer society Conference on Computer Vision and Pattern Recognition, pages 2691-2698, IEEE, 2010.
[61] Xiang Zhang, Naiyang Guan, Dacheng Tao, Xiangang Qiu, and Zhigang Luo. Online multi-modal robust non-negative dictionary learning for visual tracking. PLoS One, 10(5)ce0124e) 24685, 2015.
[62] Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, and Xu Sun. Explicit sparse transformer: Concentrated attention through explicit selection. CoRR, abs/1912. 11637, 2019.
[63] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 633-641 641, 2017.
[64] Fengtao Zhou, Sheng Huang, and Yun Xing. Deep semantic dictionary learning for multi-labet image classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3572- 3580, 2021.
13


Reviews:
[{"review_id": "pyTaafvU5H", "reviewer": "Reviewer_2tL6", "summary": "The paper proposes a novel attention mechanism called Dic-Attn that enhances the performance of deep vision models on various computer vision tasks. Specifically, the Dic-Attn module allows for the disentanglement of underlying nonlinear structural information in visual data, providing an intuitive and elegant way to exploit discriminative information and provide visual attention. Moreover, the proposed method uses shallow-depth attention modules, which are computationally efficient and can be easily integrated into existing deep learning architectures. The paper presents experimental results on several computer vision tasks, demonstrating the effectiveness of the proposed method and its superiority over several state-of-the-art methods.", "questions": "For W_D and W_scale, why choose them as vectors instead of matrices?\nAre there any ablation study to verify the advantages of this choice?\n\nBesides, if they are vectors, they should be in bolded lower case.", "limitations": "No potential negative societal impact", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is intuitive and novel. It combines the advantages of dictionary learning with deep neural networks. Compared to previous related methods like EMANet and HamNet, it cleverly avoids the gradient problem in the iterative procedure. From the methodology perspective, this method is beyond the acceptance bar.\n\n- The proposed Dic-Attn module achieves promising performance on various computer vision tasks, including image classification and point cloud classification. Besides, the proposed method shows its efficiency over several attention methods on these tasks. For example, the paper compares the proposed method with several existing attention mechanisms, such as SE-Net, CBAM, and ECA-Net, and shows that the proposed method achieves better performance in terms of accuracy and efficiency.\n\n- The paper provides a detailed analysis of the proposed method, including ablation studies and visualization of attention maps, to better understand its behavior and performance. For example, the paper shows that the proposed method can capture both local and global features of the input data and that the learned attention maps are interpretable and meaningful.\n\n- The paper presents experimental results on several computer vision tasks, demonstrating the effectiveness of the proposed method and its potential for real-world applications. For example, the paper shows that the proposed method can be used for object recognition, scene understanding, and 3D point cloud classification, which are important tasks in computer vision.", "weaknesses": "# Inadequate referencing\nThe current manuscript overlooks referencing certain pertinent studies. The proposed method, while distinct in its own right, exhibits conceptual parallels with several prior works that should be acknowledged. Comparison to these similar works, which are neither referenced nor discussed, would greatly augment the overall discourse.\nReferences:\n1. Zhu Z, et al. Asymmetric non-local neural networks for semantic segmentation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 593-602.\n2. Li X, Zhong Z, Wu J, et al. Expectation-maximization attention networks for semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 9167-9176.\n\n# Limited results presentation\nI found the experimental results to be somewhat restrictive, with only one large-scale benchmark being demonstrated. For a more comprehensive understanding of the study's impact, I recommend showcasing the complete ADE2K results within the supplementary materials, instead of merely citing a single score within the body of the text.\n\n# Visualization and explanation disparity\nWhile Figures 3 in the main document and 1 in the supplementary material do an effective job of portraying the inner workings of the proposed method, the related explanation (lines 336-338) lacks clarity. Determining which attention map is closer to human visual attention or identifying the \"referred objects\" proves challenging due to the blandness of the given explanation. This area requires more in-depth exploration and improved elucidation to make the understanding of the model's performance more intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful review, positive feedback and constructive suggestions. The following responses might help address your questions about this paper:\n1. **Weakness 1.** \"**Inadequate referencing...**\n\n   **Answer.** Thank you for your feedback. We have indeed reviewed the two papers you mentioned. To address this concern, we revised the manuscript to include a discussion and made sure to properly cite these related papers in the paper, so that readers can easily access and review the relevant literature.\n\n\t[1] takes advantage of the non-local network is potent to capture the long-range dependencies that are crucial for semantic segmentation, and relieves the shortcoming of non-local block. [1] reduces the middle dimension of the key and value in the attention without affecting the final output dimension, $R^{N\\times N} \\times R^{N\\times C}\\rightarrow R^{N\\times S} \\times R^{S\\times C} \\rightarrow R^{N\\times C}$. In order to improve performance, [1] also adopts the Spatial Pyramid Pooling method to make multi-scale fusion with different $C$. It inspires us that multi-scale dictionaries may have potential for future research.\n\n\t[2] proposed a new attention mechanism for semantic segmentation, called Expectation Maximization Attention (EMA). The expectation (E) step works as estimating the expectation of the attention map and maximizing (M) step functions as updating the bases by maximizing the complete data likelihood. The output can be computed as the weighted sum of bases, where the weights are the normalized final attention maps.\n\n\tOur method is applying the dictionary learning method to obtain a set of basis vectors that can be used to represent data in a more efficient and compact way. The training of the task-driven dictionary atom transformed/weighting matrix $W_D$ can also be considered as a process of expectation maximization.\n    We appreciate your concern about the lack of references to certain pertinent studies in the current manuscript and your suggestion to include a comparison with these similar works. \n\n     &emsp;\n   \n    [1] Zhu Z, et al. Asymmetric non-local neural networks for semantic segmentation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 593-602.\n\n      [2] Li X, Zhong Z, Wu J, et al. Expectation-maximization attention networks for semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 9167-9176.\"\n\n     &emsp;\n   \n2. **Weakness 2.** \"**Limited results presentation...**\n\n      We now place more experimental results in the rebuttal box at the top. More experiments are still on evaluated, and the results will be published as soon as possible and added in **the revised version**.\n\n     &emsp;\n   \n3. **Weakness 3.** \"**Visualization and explanation disparity...**\"\n\n    **Answer.** Thank you for this valuable suggestion. We agree that visual interpretations can help readers better understand the experiments' results. In the **updated version**, we do the following two revisions.\n    \n    (1) We further clarify that the output Attention Maps are obtained through the reconstruction of the re-weighted dictionary and transformed sparse representations. \n\n    The diagonal transformed matrix $\\mathbf{W_D}$ and transformed matrix $\\mathbf{W_{\\phi}}$  are specifically designed to re-weight the dictionary and transform its corresponding sparse representations, respectively. $\\mathbf{W_D}$ operates in a vector-wise manner, where each element in the diagonal serves as a weight for the dictionary columns, i.e., atoms. On the other hand, $\\mathbf{W_{\\phi}}$ implements the sparse encoding of individual elements. These two parameters are updated via the backpropagation strategy and are thus driven by the task's final objective function. The dictionary $\\mathbf{D}$ belongs to $\\mathcal{S}(n,k): =$ { $\\mathbf{D}\\in \\mathbb{R}_{*}^{n\\times k}:diag(\\mathbf{D}^T\\mathbf{D})=\\mathbf{I}_k$ } and the corresponding sparse representations is subject to the Elastic Net constraint, they are also data-driven.\n    \n    (2) The output Attention Maps demonstrate that our proposed attention module exhibits sharper and more precise object boundaries, enhancing the segmentation accuracy of the model. Based on these observations, we infer that the advantage of our proposed attention module stems from the pipeline our proposed attention module introduces. It is suggested that the potential non-linear structural information disentanglement ability of dictionary learning can improve performance during segmentation.\n\n     &emsp;\n   \n4. **Question.** \"For W_D and W_scale, why choose...\"\n\n    **Answer.** We sincerely appreciate your careful review and constructive suggestions sincerely this paper. The typo has been corrected in the **revised version**.  \n    The diagonal transformed matrix $\\mathbf{W_D}$ and matrix $\\mathbf{W_{\\phi}}$  are specifically designed to re-weight the dictionary and transform its corresponding sparse representations, respectively. \n\n    (1) $\\mathbf{W_D}$ operates in a vector-wise manner, where each element in the diagonal serves as a weight for the dictionary columns, i.e., atoms. \n\n    (2) On the other hand, $\\mathbf{W_{\\phi}}$ implements the sparse encoding of individual elements. \n    These transformation parameters are driven by the task objective function and are updated through the backpropagation process."}, {"review_id": "L1ubVeFSks", "reviewer": "Reviewer_nmhX", "summary": "This paper introduces a new attention mechanism, dictionary learning-based attention (Dic-Attn), to replace existing attention modules (e.g., self-attention) in deep networks (e.g. Vision Transformer, ViT). The proposed Dic-Attn comes from the combination of dictionary learning and sparse coding, and sparse visual attention. The Dic-Attn module is used to replace self-attention in transformer models for image classification, point cloud classification, and image segmentation. Results show the Dic-Attn performs better than self-attention on these tasks.", "questions": "Please address the concerns above.", "limitations": "The authors did not discuss the limitations of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a very interesting idea to combine dictionary learning and sparse coding with attention mechanisms. Attention is proposed with the hypothesis that not all features are equal and we should only pay attention to important ones. The spirit is similar to sparse representation learning. The proposed Dic-Attn combines them in an elegant way.\n2. The proposed Dic-Attn plugged into the existing models is shown to outperform self-attention for various computer vision tasks: image classification, point cloud classification, and semantic segmentation, with noticeable improvements.\n3. The influence of hyper-parameters, dimension k and attention block number l, is studied and the results provide some insight of how they may affect the model performance. Essentially, larger k and larger l leads to better performance.\n", "weaknesses": "1. How does the backward process work in Algorithm 1? The sparse representation \\phi in Eq. 3 is not differentiable.\n2. What are the mIoU values of the proposed method and the counterpart using self-attention on the ADE20K dataset for semantic segmentation?\n3. Since the paper proposes to learn sparse representation using dictionary learning-based attention, some example sparse representations should be shown in order to verify the effectiveness of the proposed method. While Fig. 3 visualized some attention maps for certain categories, the sparsity of the representation is still unknown.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions. The following responses might help address your concerns and questions about this paper:\n1. **Weakness 1.** How does the backward process work in Algorithm 1? The sparse representation $\\phi$ in Eq. 3 is not differentiable.\n\n     **Answer.** We sincerely thank you for perusing this work. We now further clarify how the backward process works in Algorithm 1 in the **revised version** and express how to address the concern about the differentiability of the sparse representation $\\phi$. \n     \n    In Algorithm 1, the backward process refers to the optimization of the dictionary and the transformed vector/matrix by minimizing the total final loss. This process also involves updating the sparse representation $\\phi$ while keeping the dictionary $D$ and the input visual features fixed at the value obtained in the previous step.\n     Note that, we employ Elastic Net regularization to enforce sparsity in the dictionary learning/sparse coding problem $||x - D\\phi|| + \\lambda ||\\phi||_0$.\n     &emsp;\n\n     Elastic Net regularization transforms the problem from NP-hard to a convex optimization problem with a closed-form solution for $\\phi$.\n     The key objective function for dictionary learning, which decomposes data into a dictionary and its corresponding sparse representation, can be expressed as follows:\n\n     $\\min_{D, \\phi}J_{Dictionary\\ Learning},$ and $J_{Dictionary\\ Learning} = ||x - D\\phi||_2^2 + \\left( \\beta ||\\phi||_1 + \\frac{\\lambda}{2} ||\\phi||_2^2 \\right),$\n\n     where $x\\in R^{n}$ represents the input visual feature, $D\\in R^{n\\times k}$ represents the dictionary, $\\phi \\in R^{k}$ represents the sparse code, and $\\lambda_1$ and $\\lambda_2$ are hyper-parameters used to balance $l$-1 and $l$-2 regularization. \n       \n     Hence, given an initial unreliable dictionary \n     $\\mathbf{D} \\in \\mathcal{S}(n,k): =$ { $\\mathbf{D}\\in \\mathbb{R}_{*}^{n\\times k}:diag(\\mathbf{D}^T\\mathbf{D})=\\mathbf{I}_k$ }. \n\n     Then, by fixing dictionary D, the objective for obtaining $\\phi$ becomes a regularized least square problem. Specifically, let the derivative of the dictionary learning objective function with respect to $\\phi$ is zero, i.e.,  $\\frac{\\partial J}{\\partial \\phi} = 0$. We can derive an analytical expression for the sparse coding vector $\\phi$, as shown in equation 3 of Algorithm 1 in the main document:\n     $\\phi^*_i = \\left(\\mathbf{D}^{T} \\mathbf{D} + \\lambda \\mathbf{I} \\right)^{-1} \\left(\\mathbf{D}^{T} x_i + \\beta \\mathbf{v} \\right)$.\n     \n     In summary, the sparse code $\\phi$ is fixed to the value obtained in the previous step. The backward propagation updating strategy then optimizes the dictionary, the diagonal transformed matrix $\\mathbf{W_D}$, and transformed matrix $W_{\\Phi}$ by minimizing the total final loss.  Subsequently, the dictionary $D$ and the input visual features are updated, and the corresponding sparse representations will be recomputed again.\n\n&emsp;\n\n2. **Weakness 2.** \"What are the mIoU values of the proposed method and the counterpart using self-attention on the ADE20K dataset for semantic segmentation?\"\n\n     **Answer.**       We now place more experimental results in the rebuttal box at the top. More experiments are still on evaluated, and the results will be published as soon as possible.\n\n     &emsp;\n   \n3. **Weakness 3.** Since the paper proposes to learn sparse representation using dictionary learning-based visual attention, some example sparse representations should be shown in order to verify the effectiveness of the proposed method. While Fig. 3 visualized some attention maps for certain categories, the sparsity of the representation is still unknown.\n\n    **Answer.** We appreciate your careful comments. While Fig. 3 visualizes attention maps for certain categories, we acknowledge that the sparsity of the sparse representation $\\Phi$ is not explicitly demonstrated in that particular figure. \n\n    In the **revised version**, we further include the results of hyperparameters $\\lambda_2$ on the impact of the sparsity and accuracy of sparse representation. They show that the sparsity increases as $\\lambda_2$ increases. According to [1, 2, 3] and existing results, it can be shown that the sparsity of the sparse representation enhances the discriminative performance of the dictionary as a feature base, while the task-driven transformation matrix selects well-disentangled features. Finally, this enables the model to capture more accurate visual attention and achieve better performance.\n\n    It is important to note that the primary focus of our paper is on introducing the concept of the dictionary learning-based visual attention module and showcasing its potential for capturing visual attention. Fig. 3 illustrates the ability of the proposed Dic-Attn module to capture relevant features for specific categories. \n\n    &emsp;\n\n\t[1] Fuchs, J-J. \"On sparse representations in arbitrary redundant bases.\" _IEEE transactions on Information theory_ 50.6 (2004): 1341-1344.\n\n\t[2] Elad, Michael, and Alfred M. Bruckstein. \"On sparse signal representations.\" _Proceedings 2001 international conference on image processing (Cat. No. 01CH37205)_. Vol. 1. IEEE, 2001.\n\n\t[3] Wei, Xian, Hao Shen, and Martin Kleinsteuber. \"Trace quotient with sparsity priors for learning low dimensional image representations.\" _IEEE transactions on pattern analysis and machine intelligence_ 42.12 (2019): 3119-3135."}, {"review_id": "Z0qNKiHHZY", "reviewer": "Reviewer_Vec1", "summary": "This paper proposed a novel dictionary learning-based attention (Dic-Attn) module, The proposed Dic-Attn module can be plug-and-play and stacked layer by layer to form a deep attention encoder. Extensive experimental results on computer vision tasks, e.g., image classification and point cloud classification, demonstrate the performance of the proposed method.", "questions": "please refer to weakness.", "limitations": "please refer to weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper proposed a novel dictionary learning-based attention (Dic-Attn) module, The proposed Dic-Attn module can be plug-and-play and stacked layer by layer to form a deep attention encoder. This is an interesting idea, might be worth to keep on digging in.", "weaknesses": "1. Writing: This is not a well-organized paper. It is more like an application paper that pursues SOTA results rather than introducing an interpretable layer.  If I understand it correctly, this paper does not focus on any analysis of the sparse dictionary learning but applies it to the transformer. If so, I would expect some insights, e.g the self-attention module could be regarded as sparse dictionary learning under some conditions and offer the proof.  Then, the paper should squeeze the length of sections 3.1 and 3.2, and leave more space to discuss or explain the insights of the method. \n\n2. Experiments: As mentioned above, this paper is an application of sparse dictionary learning in the transformer, so readers would expect some explainable experiment results or better performance than the baseline. Just experimenting on small datasets like CIFAR is not convincing when we know that the transformer was commonly used in large datasets like ImageNet.  We know that sparse dictionary learning algorithms usually be applied in robustness or denoising tasks and could achieve good results. And the author should conduct some experiments that a transformer weapon with the proposed module should be significantly better than the baseline on robustness or denoising tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful comments and constructive suggestions. The followings are detailed responses to your questions/concerns:\n1. **Weakness 1.** Writing: This is not a well-organized paper. It is more like an application paper that pursues SOTA results rather than introducing an interpretable layer. If I understand it correctly, this paper does not focus on any analysis of sparse dictionary learning but applies it to the transformer. If so, I would expect some insights, e.g the self-attention module could be regarded as sparse dictionary learning under some conditions and offer the proof. Then, the paper should squeeze the length of sections 3.1 and 3.2, and leave more space to discuss or explain the insights of the method.\n    \n    **Answer.** We appreciate your thoughtful comments on the organization and focus of our paper. In the **revised version**, we strike a better balance between presenting SOTA results and providing a clear explanation of the method. \n\n    (1) Section 3.1 discusses the summarized CV attention form, which is also the foundation of our proposed Dic-Attn module. Section 3.2 introduces the dictionary learning problem and derives the close solution of the sparse representation. With these two crucial parts at hand, we propose the concept of dictionary learning-based attention within the context of transformer models. We analyzed the process of data processing in the model and discussed the way the proposed module extracts attention. At the same time, due to the correlation between dictionary learning and biological visual perception, we reviewed some biological-related viewpoints as evidence for the discussion.\n    \n    (2) While it is true that our paper primarily focuses on achieving state-of-the-art results, it is not solely an application paper. We also analysis the influencee of the encoder depth (model perspective), dictionary dimension, and the sparsity regularization coefficient (Dic-Attn module perspective). Further, we discuss the number of parameters and computational burden. Experimental results in image classification (CIFAR-10/100, ImageNet-1K), robust evaluation (CIFAR-10), real image denoise (SIDD), image segmentation (ADE20K) and Point Cloud Classification (ModelNet40, ScanObjectNN) showcasing the potential of dictionary learning-based visual attention module for capturing visual attention. The potential non-linear structural information disentanglement ability of dictionary learning can improve performance during segmentation, which has been validated in the comparison of attention maps within the ADE20K experiments.\n\n    (3) Regarding your suggestion to explore the self-attention module as sparse dictionary learning under certain conditions, in section 3.1 of our current paper, we provide a general attention form that also includes the self-attention mechanism. Although both have operations such as inner product, the operations inside the module are completely different, as can be seen by comparing Equation 1 and Equation 4. The most important step in obtaining attention maps for Dic-Attn is nonlinear decomposition and reconstruction, while the process of SA obtaining query, key, and value is a linear transformation. However, we agree that such analysis can provide valuable insights, and we will consider incorporating this aspect into future research or expanding our work to explore the connection between self-attention and sparse dictionary learning.\n\n\tThank you once again for your valuable feedback and constructive suggestions!\n\n     &emsp;\n   \n2. **Weakness 2**. Experiments: \"As mentioned above, this paper is an application of ...\"\n   \n    **Answer.** We now place more experimental results in the rebuttal box at the top.  We appreciate your valuable feedback, as it helps us improve the comprehensiveness and applicability of our experimental evaluation. More experiments are still on evaluated, and the results will be published as soon as possible and added to the **revised Manuscript and updated Supplementary Material**."}, {"review_id": "d9N0oxmsoR", "reviewer": "Reviewer_1dqN", "summary": "This paper is about a new attention module called Dic-Attn, which is based on dictionary learning and sparse coding in the human visual perception system. The module can extract nonlinear structural information in visual data and reconstruct attention maps. The paper emphasizes the potential of leveraging sparsity in attention algorithms and addresses the efficiency and effectiveness of current attention methods. Authors also conduct experiments on image classification, point cloud classification. \n", "questions": "Seeing weaknesses", "limitations": "Seeing weaknesses", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea is novel.This paper proposes a novel attention module, Dic-Attn, which combines dictionary learning and attention mechanism to effectively explore the underlying nonlinear structure information and enhance the comprehensive representation of visual information. \n\n2. The performance on point cloud classification is significant, which demonstrates the effectiveness of proposed method.\n\n3. The writing is esay to follow and visualization is reasonable. \n\n", "weaknesses": "1. Experiments on CIFAR10 for image classification are not convincing. Please conduct experiments on ImageNet dataset to compare swin transformer.  Besides, showing the result on ADE20K dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful review and feedback on this paper. \nThe following responses might help address your questions about this paper:\n\n**Question & Weakness .** \"Experiments on CIFAR10 for image classification are not convincing. Please conduct experiments on ImageNet dataset to compare Swin transformer. Besides, showing the result on ADE20K dataset.\"\n\n**Answer.** We highly appreciate your valuable advice. In the revision, we have illustrated the additional experimental results.\n\n   (1) We use \"Swin-T (WSA)\" and \"Swin-T (Dic-Attn)\" to denote the Swin Transformer-Tiny model and Swin Transformer-Tiny model using the windows Self-Attention (WSA) module and the proposed dictionary learning-based visual attention (Dic-Attn) module, respectively.\n    \n   (2) Specifically, the added Accuracy (Acc), Mean Intersection over Union (mIOU) / Mean Accuracy (mAcc) results of **the classification task on ImageNet-1K** and **the part segmentation task on ADE20K** are shown as follows:\n    \n  **Table 1.**  Additional classification results on ImageNet-1K. \n\t    \n| Methods          |   Dataset   |   Resolution    | Acc (50 Epochs) |  Acc  | \n| ---------------- |:-----------:|:---------------:|:---------------:|:-----:| \n| Swin-T (WSA)      | ImageNet-1K | $224\\times 224$ |      70.29      | 81.23 | \n| Swin-T (Dic-Attn) | ImageNet-1K | $224\\times 224$ |      71.66      | 81.29 |   \n     \n   Swin-T (WSA) and Swin-T (Dic-Attn) are trained from scratch on the ImageNet-1K dataset on four GeForce RTX 4090 GPUs for 300 epochs. The input images are all resized to $224^2$, and the batch size is 256. The training process for each epoch is expected to take approximately one to two hours. Our proposed Dic-Attn module brings up 1.37% gains in Top-1 accuracy compared with Swin-T(WSA) at the beginning of training (50 epochs).\n   \n#### **Table 2.  Additional segmentation results on ADE20K.** Backbones are trained from scratch on ImageNet-22K. ViT-Adapter-L applies UperNet framework.\n\n| Methods                |Backbone| mIOU  | mAcc  |\n| ---------------------- |:-----:|:-----:|:-----:|\n| Segmenter-B          |ViT(SA)| 49.20 | 59.32|   \n| ViT-Adapter-L       |Beit(SA)| 56.80 | 69.56 |  \n| Segmenter-B    |ViT(Dic-Attn)| 50.20 | 60.57  |\n| ViT-Adapter-L |Beit(Dic-Attn)| 56.93  | 69.78 | \n\nMore experiments are still on evaluated, and the results will be published as soon as possible and added to the **revised Manuscript and updated Supplementary Material**."}, {"review_id": "97plhOqMsM", "reviewer": "Reviewer_q2Dg", "summary": "This paper presents a new architecture that can be applied for various tasks, including image classification, point cloud classification and image segmentation. The key idea is to leverage a learnable dictionary module to replace the attention model in the transformer architecture. The model is able to achieve compelling results on multiple tasks with the low computational cost, including the fewer parameters, less GPU training time. Experimental results are conducted on multiple vision tasks and achieve compelling results. ", "questions": "Apart from a few questions that need clarifications listed above, the only additional results I might suggest is to also perform inference time reported as in some related works for a slightly fairer comparison. ", "limitations": "There are no discussion or broader societal imparts discussed in the main paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "### improved results over baseline models\n- Based on the quantitative results shown in Tables 2, 3 and 4, the proposed model outperforms the existing models on all tasks.\n\n", "weaknesses": "### (Major) Confusion on method \n- It took me a long time to actually get the main message of this paper, which is hidden in a bunch of overwhelming technical details and verbose writing.\n- Most importantly, the dictionary learning directly learn the sparse representation for the attention $A$. However, this is just a sparse representation, and I cannot get the attention similarity between tokens of different grid features. \n- The equation $D=[d_1^c,d_2^c,\\cdots,d_k^c]=[d_1^r, d_2^r,\\cdots,d_s^r]^T\\in R^{k\\times s}$ and $\\Phi=[\\phi_1^c,\\phi_2^c,\\cdots,\\phi_k^c]=[\\psi_1^r, \\psi_2^r,\\cdots,\\psi_s^r]^T\\in R^{n\\times s}$ are confused, which is different to the dictionary $D\\in R^{n\\times s}$and sparse representation $\\Phi\\in R^{k\\times s}$ defined before. \n- The equation is also confused. $x_{1*}$ denotes one grid feature with different dimension value, but every channel is represented by different code $\\phi$, which is hard to understand for the embedding.\n- The key optimisation function is also unclearly. The dimension of $\\mathbf{D}$ and $\\mathbf{\\Phi}$\nis different. How could they calculate the reconstruction in equation (6)?\n\n### (Major) More results should have been expected\n- I expected to see more segmentation results, but only the compared attention map is provided in Figure 3, and no results in supplemental material either.\n- A quantitative comparison is expected for the image segmentation task.\n- It would be helpful to provide some visual examples of the segmentation map in the main paper.\n\n### (Minor) Presentation\n- For figure 2, the paper claimed \"the deeper the better\", but it is not true from the figure visual results. It would be stronger to provide the accuracy analysis for the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank for your valuable suggestions on this paper. The following responses might help address your questions and concerns about this paper:\n\n#### 1. **Weakness 1. (Major) Confusion on method...** \n\n**Q(1)** \"...The key motivation seem...\"\n\n   **Answer to Q(1):** Thank you for your careful review and feedback. In the **revised version**, we further improve the clarity and organization of the paper to ensure that the main message is more easily accessible to readers. \n\n   The primary focus of our paper is on introducing the concept of dictionary learning-based visual attention module and showcasing its potential for capturing visual attention. The attention mechanism is one of the key structures of the very popular high-performance transformer network. Recalling that in human visual perception, a large number of studies have shown (some references are cited in the  Introduction and Section 3.3.2) the primary visual perception cortical signal encoding method and Dictionary learning is similar, which is also closely related to human visual attention. Inspired by this, we attempt to propose a dictionary learning-based visual attention (Dic-Attn) module in machine vision, exploiting the potential nonlinear structural information disentanglement ability from dictionary learning, and exploring its intuitive way to exploit discriminative information and visual attention.\n   \n   **Q(2)** \"...I cannot get attention similarity between tokens of different grid features...\"\n  \n   **Answer to Q(2):** \n   The sparse representation serves as the coordinates of the dictionary. These sparse representations indicate the importance or relevance of each atom in the dictionary for reconstructing the input features. Importantly, the attention similarity between different tokens or grid features is inherently encoded within the dictionary and the sparse representation. \n   The diagonal transformed matrix $\\mathbf{W_D}$ and transformed matrix $\\mathbf{W_{\\phi}}$  are specifically designed to re-weight the dictionary and transform its corresponding sparse representations, respectively. Moreover, $\\mathbf{W_D}$ operates in a vector-wise manner, where each element in the diagonal serves as a weight for the dictionary columns, i.e., atoms. On the other hand, $\\mathbf{W_{\\phi}}$ implements the sparse encoding of individual elements. \n   These transformation parameters are driven by the task objective function and are updated through the backpropagation process.\n\n**Q(3)** \"The equations are confused...\"\n   \n   **Answer to Q(3):** We apologize for the confusion and appreciate your correction. In the **revised version**, we have corrected these typos:\n\n   $D = [d^c_1  d^c_2  \\cdots  d^c_k]= [d^r_1 d^r_2 \\cdots d^r_n ]^T\\in R^{n\\times k},$\n\n   $\\Phi = [\\Phi_1  \\Phi_2  \\cdots  \\Phi_b]\\in R^{b\\times k\\times s},$\n\n   $\\Phi_i = [\\phi_1  \\phi_2  \\cdots  \\phi_s]= [\\psi_1 \\psi_2 \\cdots \\psi_k ]^T\\in R^{k \\times s}.$\n   \nIn our proposed method, we aim to capture the critical part of visual data by utilizing dictionary learning. According to experience, we build the dictionary as $n*k$, instead of building a dictionary at the fine-grained level of batch-level. The results also show that dictionary as a feature base is complete enough and discriminative. Hence, given the input $X\\in R^{b\\times n \\times s}$, sparse representations $\\Phi \\in R^{b\\times k\\times s}$ will be seen as its sparse embedding. \n\n**Q(4)** \"The key optimization function... calculate the reconstruction in equation (6)\"\n\n**Answer to Q(4):**  We appreciate for pointing out the unclear optimization function and the refactoring of the computation in Equation 6. \n\nFirstly, we clarify how the optimization process works in Algorithm 1 and the key optimization function for dictionary learning. Due to space limitations, please refer to our response to Weakness 1 from Reviewer nmhX. Additionally, in the revised version, we provide a more detailed explanation of the key optimization function for dictionary learning.\n\nAfter correcting Typo, we illustrate how could $D$ and $\\Phi$ calculate the reconstruction in equation 6.\nTherefore, let $X_i=[x_1, \\cdots, x_s]\\in R^{n\\times s}$ denotes the input, $D\\in R^{n\\times k}$ represents the dictionary, $\\phi$ represents the sparse coding vector, $\\Phi_i = [\\phi^c_1  \\phi^c_2  \\cdots  \\phi^c_k]\\in R^{k\\times s}$ . The non-linear scaling function NL(·) in equation 6 and equation 4 do not change the dimension of sparse representation $\\Phi$. Therefore, it is able to calculate the reconstruction and output the attention map, i.e., $R^{n\\times s} \\rightarrow R^{n\\times k} \\times R^{k\\times s} \\rightarrow R^{n\\times s}$.\n     \n#### 2. **Weakness 2 / Question.  \"(Major)More results should have been expected...; \"...perform inference time reported...\"**\n\n**Answer.** Specifically, we add more experiments to verify our proposed module explicitly. We now place added experimental results in the rebuttal box at the top. The added results of inference time are also shown in the rebuttal box at the top. We appreciate your suggestion and hope that the revised version will better address your concerns.\n\n#### 3. **Weakness 3. (Minor) Presentation...**\"\n\n**Answer.** Thank you for your careful review and feedback. Based on the results, it can be observed that as the depth increases, the accuracy initially improves, but after reaching a certain depth, the accuracy starts to decline. The phenomenon of rising first can be attributed to the fact that deeper networks have the capability to capture more intricate relationships between inputs and outputs. However, they are also prone to overfitting, particularly when the training dataset is relatively small, such as in the case of training from scratch using the CIFAR-10 dataset.\n\tTherefore, it is important to carefully consider the trade-offs between depth and complexity when designing a neural network, rather than assuming that deeper is always better. \n\n"}]

Author rebuttal:
# More Experimental Results

## **Answer** to Related Questions including: 

- Weakness 2 by Reviewer 2tL6
- Weakness 2 by Reviewer Vec1
- Weakness 2 by Reviewer nmhX
- Weakness 1 by Reviewer 1dqN
- Weakness 2 and Question 1 by Reviewer q2Dg

We appreciate your comments and your expectation for more explainable results or improved performance compared to the baseline. he following responses might help address your concerns about this paper: 
    
**(1) Image Classification:** We have conducted additional experiments on the ImageNet-1k dataset for image classification. We use "Swin-T(WSA)" and "Swin-T(Dic-Attn)" to denote the Swin Transformer-Tiny model and Swin Transformer-Tiny model using the Windows Self-Attention module the proposed dictionary learning-based visual attention(Dic-Attn) module, respectively. Specifically, the added results of the classification task on ImageNet-1K are shown as follows:
    
#### **Table 1.  Additional classification results on ImageNet-1K.** Models are trained from scratch on ImageNet-1K. The input images are all resized to $224^2$. Our proposed Dic-Attn module brings up 1.37% gains in Top-1 accuracy compared with backbone Swin-T(WSA) at the beginning of training (50 epochs).
     
|Methods|Dataset|Resolution|Acc (50 Epochs)|Acc|
|---|:-:|:-:|:-:|:-:|
|Swin-T(WSA)|ImageNet-1K|$224\times 224$|70.29|81.23|
| Swin-T(Dic-Attn) |ImageNet-1K |$224\times 224$|71.66|81.29|

**(2) Robustness:** Regarding your suggestion to explore the application of sparse dictionary learning algorithms in robustness or denoising tasks, we agree that such experiments could provide valuable insights into the capabilities of our proposed method. Actually, we have reported the model performance on adversarial robustness under the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) in Section 4.2.1 in the main document. We set the FGSM attack with the perturbation magnitude $\epsilon = \frac{4}{255}$ and set $\epsilon= \frac{8}{255}$ or PGD, with iteration numbers $t = 10$ and step size $\alpha = \frac{2}{255}$. We classify images corrupted by adversarial attack FGSM and PGD, respectively. Then we evaluate the model performance by their attack classification accuracy.

#### **Table 2. Robustness Evaluation Results on CIFAR-100 dataset.** Comparison of our approach with backbones and other attention baseline methods, against various adversarial attacks.
	
| Model          | Attacks |      |       | # Param(M) |
| -------------- |:-------:|:----:|:-----:|:----------:|
| (From Scrach)  |  FGSM   | PGD  | Clean |            |
| RVT (SA)       |  0.20   | 0.10 | 64.80 |    8.29    |
| RVT (VARS-D)   |  10.58  | 3.80 | 62.20 |    7.68    |
| RVT (Dic-Attn) |  10.00  | 4.30 | 55.93 |    7.48    |

**(3) Denoising:** In response to your feedback, we have conducted experiments on the SIDD dataset for image denoising. The results of these experiments show that the transformer augmented with our proposed module outperforms the baseline in denoising performance.

#### **Table 3. Additional classification results on SIDD.**

| Methods             | PSNR | SSIM |
| ------------------- |:----:|:----:|
| Restormer(SA)       | 39.53 | 0.960  |
| Restormer(Dic-Attn) |  39.54 |0.958|

**(4)Inference Time:** Specifically, we add the Inference Time test to compare the efficiency of various attention modules explicitly. The added results of inference time are shown as follows:
	
#### **Table 4. Comparison of the inference time cost (ms) of various attention modules.**

  | Methods/Indicators | GPU Inference Time (ms) |
  | ------------------ |:-----------------------:|
  | SA                 |          82.20          |
  | DA                 |          64.40          | 
  | $A^2$              |          8.00           |
  | HAM                |          7.70           |
  | ACF                |          22.60          |
  | Dic-Attn(Ours)     |          24.70          |
	
The added results report the inference time of our proposed module and the baseline models. We have added to Table 4 in the revised version, highlighting the advantages and disadvantages of our proposed model in terms of computational efficiency.
	
We appreciate valuable feedback, as it helps us improve the comprehensiveness and applicability of our experimental evaluation. More experiments are still on evaluated, and the results will be published as soon as possible and added to the **revised Manuscript and updated Supplementary Material**.

Meta-review: The paper describes a novel attention module based on dictionary learning, which can be employed for several architectures in different computer vision tasks. The methods is demonstrated on two applications, e.g. image classification and point cloud classification. The reviewers expressed concerns in terms of writing and requested more experiments on other benchmarks (ImageNet). There was a constructive discussion between authors and reviewers on missing elements in the paper. All the reviewers appreciated the rebuttal, increasing their score. At the end of the process, there was an agreement among reviewers that the paper can be accepted.
Decision: Accept (poster)