SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the transformation of the dictionary, spatial attention is introduced during the reconstruction process, effectively leveraging the discriminative power of atoms [18, 2, 64, 27]. Different sizes of the diagonal elements in the  $\mathbf{W}_{\mathrm{D}}$  impose weights on different channels  $\mathbf{d}_i\psi_i$  in the reconstruction process. In view of sparse representations/codes, the  $Dic$ -Attn module employs sparse prior and exploits the intrinsic underlying structure of features. The sparsity involved in the attention module arises spontaneously from the statistically redundant natural images. The sparse representation serves as the coordinates of the dictionary. These sparse representations indicate the importance or relevance of each atom in the dictionary for reconstructing the input features. Importantly, the attention similarity between different tokens or grid features in  $\mathbf{X}$  is inherently encoded within the dictionary and the sparse representation.  $\mathbf{W}_{\Phi}$  modifies individual elements and  $\mathbf{W}_{scale}$  applies unified weights within each column of  $\Phi$ , respectively. These transformation parameters are driven by the task objective function and are updated through the backpropagation process.
Algorithm 1 summarizes the algorithm flow of the  $Dic$ -Attn module. Furthermore, it is worth noting that constructing a deep  $Dic$ -Attn vision model is feasible. All these transformation matrices are trained in a back-forward manner and serve as a crucial link in connecting the dictionary, sparse representations, and reconstructed attention maps. To enhance the stability and effectiveness of the  $Dic$ -Attn module, it is recommended to include a normalization step before the attention module as [46]. Additionally, a residual connection between the input  $\mathbf{X}$  and the Attention Ouptut facilitates the flow of information. This results in the formation of the  $Dic$ -Attn block, as illustrated in Fig. 1.
In summary, the  $\text{Dic-Attn}$  module utilizes a data-initialized and task-driven dictionary strategy, which captures global information and serves as a comprehensive resource containing essential data constituents. The process of reconstructing attention maps fully capitalizes on the dictionary learning sparse coding approach, the ability to extract both channel and spatial information from nonlinear features. The  $\text{Dic-Attn}$  module has a simpler structure with fewer parameters and elegant internal operations while achieving higher performance. Moreover, the  $\text{Dic-Attn}$  module is a convenient plug-and-play attention module that can be easily integrated into various vision networks.
3.3.2 General  $\text{Dic-Attn}$ -Based Vision Transformer
Herein, we give a guide on how to build a  $\text{Dic-Attn}$ -based Vision Transformer and analyze the role of the proposed  $\text{Dic-Attn}$  method. The model architecture can be divided into three main parts: the preprocessing stage, the deep attention stage, and the post-processing stage for downstream tasks.
Firstly, multi-layer convolution and linear operations are employed to extract primary visual features from the natural images. Then, multiple concatenated  $\text{Dic-Attn}$  blocks form the deep attention stage, generating the final attention map step by step. To conveniently introduce this process, we further summarize the  $\text{Dic-Attn}$  form as follows
$$
\operatorname {R e c o n s t r u c t i o n} (\widetilde {\mathbf {D}}, \mathbf {N L} (\boldsymbol {\Phi})), \tag {6}
$$
where  $\mathbf{NL}(\cdot)$  denotes non-linear scaling function.  $\widetilde{\mathbf{D}}$  is the learned dictionary with re-weighted atoms.  $\mathbf{NL}(\Phi)$  can be treated as a response prediction model [56]. We assume that the input of  $(l + 1)$ -layer attention block is Attention Output $_l + \mathbf{X}_l$ , i.e., the output of the previous layer's attention block. Learning-based dictionary  $\mathbf{D}$  of each  $D_{ic}$ -Attn module groups significant features of different objects
in the input, and the most salient objects are preserved while those irrelevant are suppressed by further selecting sparse representations. The output of each layer has a common intersection as the input  $\mathbf{X}_1$  of the first layer, which is a task-independent bias for dictionary learning in  $Dic$ -Attn modules of any depth. Moreover, the output can be further considered as consisting of two components, one of which is the  $l$ -layer module's attention area, and the other component represents the ignored part. The first component will affect deep attention modules layer by layer, and adjust the task-driven selection matrix through backpropagation. Hence, our proposed  $Dic$ -Attn module of shallow depth can bring selective biases with a task-independent impact on deeper attention modules.
The proposed  $\text{Dic-Attn}$  method, based on feature decomposition and sparse prior, offers a white-box novel strategy for obtaining visual attention, which may be able to be applied to scenarios with more stringent security requirements. Generally, in the visual perceptual system, selective attention is primarily driven by strong stimulus signals and attention to specific regions. Selective attention can be understood from three main aspects: bottom-up voluntary attention, involuntary attention, and "selection history" attention. Involuntary attention, similar to the task-driven model in computer vision, is obtained through top-down rewards. "selection history" refers to the influence of previously attended regions or objects on future attention allocation. Meanwhile, the selective lingering biases have a task-independent impact on the subsequent attention priority. In biology, it means that the nerve has better plasticity. While in the deep neural network, the structure of the multi-layer attention module is not just capable to acquire features with higher semantics. Shallow-depth attention modules capture data prototype features and affect deeper attention modules. It will make models with greater capacity, better generalization, and robustness. The multi-layer attention module structure in the  $\text{Dic-Attn}$  module goes beyond acquiring features with higher semantics. Shallow-depth attention modules capture data prototype features and influence deeper attention modules, resulting in models with increased capacity, better generalization, and improved robustness.
In summary, we demonstrate the effectiveness of the  $D_{ic}$ -Attn module in capturing visual attention, enhancing visual model performance. We also provide the interpretability for deeper insights into the attention mechanism.
In this section, we evaluate the performance of proposed  $D_{ic}$  -Attn on 6 benchmarks. Firstly, we conduct ablation experiments on CIFAR-10 dataset, as described in Sec. 4.1. Subsequently, we compare the computational costs. In Sec. 4.2, we evaluate the performance of the  $D_{ic}$  Attn module across various visual tasks, such as point cloud classification, image segmentation, and image classification. We compare the accuracy and robustness of the proposed method with several notable attention modules. More experiments and the code are available in the Supplementary Material.
(a) Encoder depth. The deeper the better, but fewer layers can also obtain good accuracy.
Figure 2: The  $\text{Dic-Attn}$  module ablation experiments (with ViT backbone) on CIFAR-10 dataset.
(b) Dictionary dimension  $k$ . An over-completed dictionary can improve performance.
4.1 Ablation Study and Computational Costs
We explore the influence of the hyper-parameters, i.e., the second dimension  $k$  of the dictionary and the number  $l$  of attention blocks (depth of encoder). Fig. 2 illustrates the significance of selecting appropriate values for  $l$  and  $k$ . A dictionary with higher dimension  $k$  creates a comprehensive feature space for images, which enhances the performance of the Vision Transformer. As the depth  $l$  increases, the accuracy initially improves but starts to decline after reaching seven in CIFAR-10. The phenomenon of rising first can be attributed to the fact that deeper networks have the capability to capture more intricate relationships between inputs and outputs. However, they are also prone to
overfitting, particularly when the training dataset is relatively small, such as in the case of training from scratch using the CIFAR-10 dataset. Therefore, it is important to carefully consider the trade-offs between depth and complexity when designing a neural network, rather than assuming that deeper is always better.
We evaluate the performance of the  $Dic$ -Attn module, and compare it with the SA module and notable variants, as presented in Tab. 1. The results show that the proposed  $Dic$ -Attn module is more lightweight and has a lower computational burden. The total number of parameters (Params.) of the SA module is related to  $n$ , which is equal to  $3n^2$ . For our proposed  $Dic$ -Attn module, the Params. is counted as  $k(n + k + 1)$ , where Params. of  $\mathbf{D}$  is  $nk$ , Params. of  $\mathbf{W}_{\Phi}$  is  $k^2$  and Params. of  $W_{\mathbf{V}}$  is  $k$ . Since  $k = \alpha n$ , it can be inferred that  $k(n + k + 1)$  is always less than  $n^3$  when  $\alpha \in (0, \alpha_1) \cap (\alpha_2, +\infty)$ .  $\alpha_1$  and  $\alpha_2$  are solutions of the quadratic equation  $n\alpha^2 + (n + 1)\alpha - n^2$  with variable  $\alpha$ . Note that, it always has real number solutions.
Table 1: Comparison of the efficiency of various attention modules. Indicators: Number of parameters (Params.), multiply-accumulates operations (MACs) and training/inference time cost compared to Self-Attention and its variants.
Methods/Indicators	Params. (M)	MACs. (G)	GPU Train / Inference Time (ms)
SA [46]	1.00	292.00	242.00 / 82.20
DA [5]	4.82	79.50	72.60 / 64.40
A² [4]	1.01	25.70	22.90 / 8.00
ACF [59]	0.75	79.5	71.00 / 22.60
HAM [6]	0.50	17.60	15.60 / 7.70
Dic-Attn (Ours)	0.60	20.00	78.00 / 24.70
Input: tensor with shape  $1 \times 512 \times 128 \times 128$ .
4.2 Computer Vision Tasks
We evaluate the performances of our proposed  $\text{Dic-Attn}$  module on public image datasets and point cloud datasets, including CIFAR-10, CIFAR-100[20], ModelNet40 [53], ScanObjectNN [45], and ADE20K [63]. Note that, to mitigate the impact of various complex factors that may arise from pre-training and fine-tuning processes, all models in this paper were trained from scratch, without pre-training, and adopted the same data augmentation method. Additionally, unless explicitly stated, the hyperparameter settings remained consistent across all experiments. More implementation settings are described in detail in the Supplementary Material.
4.2.1 Image Classification
CIFAR-10 dataset is composed of 10 categories 60000 color images with the size of  $32 \times 32$ . CIFAR-100 dataset has 100 classes. Both of CIFAR-10 and CIFAR-100 contain 50000 training images and 10000 test images. The number of images in each category is equal, so 6000 images per class in CIFAR-10 and 600 in CIFAR-100, respectively.
The classification results on image datasets are shown in Tab. 2. The image classification accuracy of ViT can be improved from  $0.64\%$  to  $2.12\%$  by changing the SA module to  $D_{ic}$ -Attn module.
In Tab. 3, we report the model performance on adversarial robustness under the Fast Gradient Sign Method (FGSM) [7] and Projected Gradient Descent (PGD) [28]. We set the FGSM attack with the perturbation magnitude  $\epsilon = \frac{4}{255}$ . For PGD, we set  $\epsilon = \frac{8}{255}$ , iteration numbers  $t = 10$ , and step size  $\alpha = \frac{2}{255}$ . We evaluate the model performance by using Attack Accuracy, including the FGSM accuracy and PGD accuracy on classifying images corrupted by adversarial attack FGSM and PGD, respectively. Note that we do not apply adversarial training here, but only use clean samples to train RVT networks [30]. In this case, the proposed  $D_{ic}$ -Attn module is still competitive.
4.2.2 Point Cloud Classification
ModelNet40 and ScanObjectNN datasets are widely used benchmarks for point cloud analysis. ModelNet40 dataset comprises 12311 CAD-generated meshes categorized into 40 classes. Among them, 9843 meshes are used for training, while the remaining 2468 meshes are reserved for testing. ScanObjectNN is a real-world dataset, that contains 15000 objects, divided into 15 and 2902 unique object instances. For all experiments conducted, only the coordinates  $xyz$  of objects are utilized. In the case of ModelNet40, all baselines are tested with 1024 input points.
Table 2: Image Classification Top1-Accuracy  $(\%)$  . The overall classification accuracy over all classes is for evaluation metrics.
Model (From Scratch)	Datasets	#Param(M)
CIFAR-10	CIFAR-100
ViT (SA)	90.91	66.67	22.74
Swin-S (WSA)	89.74	55.82	22.25
ViT (Dic-Attn)	91.55	68.79	19.33
Table 3: Robustness Evaluation Results on CIFAR-100 dataset. Comparison of our approach with backbones and other attention baseline methods, against various adversarial attacks.
Model (From Scratch)	Attacks	#Param(M)
FGSM	PGD	Clean
RVT (SA)	0.20	0.10	64.80	8.29
RVT (VARS-D)	10.58	3.80	62.20	7.68
RVT (Dic-Attn)	10.00	4.30	55.93	7.48
The pre-processing of cloud points is performed according to [9]. Note that the point cloud data is encoded as a fixed-shape feature tensor to input the attention module, which preserves and incorporates the original point clouds' position correlation, etc. The classification results on point cloud datasets are shown in Tab. 4. It is evident that our proposed  $Dic\text{-}Attn$  module significantly enhances the classification performance of the Point Cloud Transformer (PCT) model [9]. The average point cloud classification results achieved with our proposed module are highly competitive and demonstrate notable improvement over existing approaches. Our proposed  $Dic\text{-}Attn$  module is able to compute an attention map for the point cloud Transformer. In addition, all these indicate that our proposed  $Dic\text{-}Attn$  module has the advantage of processing complicated data and has the potential to measure the topological attention relationships of the underlying geometric structure.
Table 4: Point Cloud Classification task Top1-Accuracy  $(\%)$ . The number of points is set to 1024.
Model
(From Scratch)	Datasets
ModelNet40	ScanObjectNN
PointNet++ [36]	91.90	84.30
PointCNN [22]	91.70	85.50
PCT	92.95	80.60
Point-BERT [57]	93.19	88.10
Point-MAE [35]	93.80	88.30
PCT (Dic-Attn)	94.60	88.96
Point-MAE (Dic-Attn)	94.46	89.41
Figure 3: The comparison of attention maps generated by SA module and  $D_{ic}$  Attn module (with backbone Segmenter, experiment on ADE20K dataset).
4.2.3 Image Segmentation
ADE20K dataset [63] contains a total of 150 semantic categories, and can be used for scene perception, parsing, segmentation, multi-object recognition, and semantic understanding, which is one of the most challenging semantic segmentation datasets. The training set contains 20210 images. There are about 2000 and 3352 images in the validation and test set, respectively.
The performance of the  $\text{Dic-Attn}$  module is verified in the semantic segmentation task with the backbone Segmenter [40], which is a competitive and representative transformer-based semantic segmentation model. We train Segmenter (SA) and Segmenter (Dic-Attn) from scratch on ADE20K dataset. Note that, Segmenter (Dic-Attn) denotes the proposed  $\text{Dic-Attn}$  module replacing the original SA module in the Segmenter backbone. The mean Intersection over Union (mIoU) and the mean Pixel Accuracy (mPA) of Segmenter (Dic-Attn) are 1.00% and 1.25% higher than that of Segmenter (SA), respectively.
Furthermore, the two image examples and their corresponding segmentation attention map results, as shown in Fig. 3, intuitively indicate that our proposed  $\text{Dic-Attn}$  module highly promotes the ability of the segmentation model to distinguish objects. The learned attention maps are also more similar to human visual attention, which focus on referred objects, e.g., building, sky, door, cabinet, etc. The learned dictionary in the  $\text{Dic-Attn}$  module contains more sufficient features, including the underlying nonlinear structure. All results above indicate that by fully extracting attention information in both the channel domain and spatial domain of feature space, the attention maps for different labeled objects can be reconstructed more accurately.
In this work, a novel dictionary learning-based attention module, namely  $\text{Dic-Attn}$ , was proposed.  $\text{Dic-Attn}$  takes the advantage of dictionary learning mechanism, enabling effective exploration of the underlying nonlinear structure information. By dynamically selecting learned dictionary atoms and sparse representations through spatial and channel transformations, multi-layer  $\text{Dic-Attn}$  modules achieve the accurate reconstruction of attention maps and allow for the extraction of features at different semantic levels, enhancing the comprehensive representation of visual information. Experimental results in various visual tasks and vision models showed that the  $\text{Dic-Attn}$  module can seamlessly integrate into various vision networks, delivering competitive performance comparable to state-of-the-art methods. The  $\text{Dic-Attn}$  module is also a promising solution with competitive performance, compatibility, and computational efficiency. Further research will be devoted to the robustness of the module and its applications in more tasks, e.g., temporal prediction tasks in scientific and engineering fields. Conducting further investigation and validation of the diverse levels of semantics exhibited by dictionaries learned at different depths in natural language processing scenarios would also be valuable.
This work was supported by the National Natural Science Foundation of China (No.42130112, No.41901335), KartoBit Research Network (No.KRN2201CA), and the Shanghai Municipality's "S&T Innovation Action Plan" for the year 2023, Shanghai Natural Science Fund, General Program (No.23ZR1419300), Shanghai Trusted Industry Internet Software Collaborative Innovation Center, "Digital Silk Road" Shanghai International Joint Lab of Trustworthy Intelligent Software (No.22510750100).
[1] Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11):4311-4322, 2006.
[2] Behnam Babagholami-Mohamadabadi, Amin Jourabloo, Mohammadreza Zolfaghari, and Mohammad T Manzuri Shalmani. Bayesian supervised dictionary learning. In UAI Application Workshops, pages 11-19. CiteSeer, 2013.
[3] Ori Bryt and Michael Elad. Compression of facial images using the k-svd algorithm. Journal of Visual Communication and Image Representation, 19(4):270-282, 2008.
[4] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A^2-nets: Double attention networks. Advances in Neural Information Processing Systems, 31, 2018.
[5] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3146-3154, 2019.
[6] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In International Conference on Learning Representations, 2021.
[7] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[8] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471. PMLR, 2015.
[9] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021.
[10] Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv preprint arXiv:2105.02358, 2021.
[51] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint arXiv:1908.04626, 2019.
[52] John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2):210-227, 2008.
[53] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1912-1920, 2015.
[54] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[55] Meng Yang and Lei Zhang. Gabor feature based sparse representation for face recognition with gabor occlusion dictionary. In European Conference on Computer Vision, pages 448-461. Springer, 2010.
[56] Takashi Yoshida and Kenichi Ohki. Natural images are reliably represented by sparse and variable populations of neurons in visual cortex. Nature communications, 11(1):1-19, 2020.
[57] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 19291-19300. IEEE, 2022.
[58] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint arXiv:2103.15949, 2021.
[59] Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xie. Co-occurrent features in semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 548–557, 2019.
[60] Qiang Zhang and Baoxin Li. Discriminative k-svd for dictionary learning in face recognition. In 2010 IEEE computer society Conference on Computer Vision and Pattern Recognition, pages 2691-2698. IEEE, 2010.
[61] Xiang Zhang, Naiyang Guan, Dacheng Tao, Xiaogang Qiu, and Zhigang Luo. Online multi-modal robust non-negative dictionary learning for visual tracking. PLoS One, 10(5):e0124685, 2015.
[62] Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, and Xu Sun. Explicit sparse transformer: Concentrated attention through explicit selection. CoRR, abs/1912.11637, 2019.
[63] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 633-641, 2017.
[64] Fengtao Zhou, Sheng Huang, and Yun Xing. Deep semantic dictionary learning for multi-label image classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3572-3580, 2021.