Question: Why is the proposed method coupled with CapsGCN if the RL module is to tune depth and width of convolutional layers?
Evidence: Image restoration algorithms aim to recover a high-quality image from a contaminated input image by solving an ill-posed image restoration problem. There are various image restoration tasks depending on the type of corruptions, such as image denoising, demosaicing, single image super-resolution, and image compression artifacts reduction. To restore corrupted information from the contaminated image, a variety of image priors were proposed.
Recently, image restoration methods based on deep neural networks have achieved great success. Inspired by the widely used non-local prior, most recent approaches based on neural networks adapt non-local attention into their neural network to enhance the representation learning, following the non-local neural networks. In a nonlocal block, a response is calculated as a weighted sum over all pixel-wise features on the feature map to account for long-range information. Such a module was initially designed for high-level recognition tasks such as image classification, and it has been proven to be beneficial for low-level vision tasks.
Though attention modules have been shown to be effective in boosting the performance. Most attention modules only explore the correlation among features at the same layer. Actually, features at different intermediate layers encode variant information at different scales, and might be helpful to augment the information used in recovering the high-quality image. Motivated by the potential benefit of exploring feature correlation across intermediate layers, Holistic Attention Network (HAN) is proposed to find interrelationship among features at hierarchical levels with a Layer Attention Module (LAM). However, LAM assigns a single importance weight to all features at the same layer and neglects the difference of spatial positions of these features. Recent research in omnidirectional representation suggests that exploring the relationship among features at different layers can benefit the representation learning of neural networks. Nevertheless, calculating correlation among features at hierarchical layers is computationally expensive due to the quadratic complexity of dot product attention. The complexity of such cross-layer attention design is increased from (HW ) 2 L to (HW L) 2 , where H, W are the height and width of the feature map and L is the number of layers.
1.1 CONTRIBUTIONS Our contributions are listed as follows.
First, in order to address the problem caused by only referring to keys within the same layer in most attention modules, we propose a novel attention module termed Cross-Layer Attention (CLA), which searches for keys across different layers for each query feature. With the help of the deformation mechanism, CLA only attends to a small set of keys at different layers for each query feature.
Second, we propose an improved CLA termed Adaptive Cross-Layer Attention (ACLA) which selects an adaptive number of keys at each layer for each query, and searches for the optimal insert positions of ACLA modules. We deploy ACLA modules on commonly used neural network models, e.g. EDSR, for image restoration. Extensive experiments on single image super-resolution, image denoising, image compression artifacts reduction, and image demosaicing demonstrate the effectiveness of our approach.
To evaluate the effects of the depth and width in CapsGNN, we change them to observe the performance of the model on two graph datasets, ENZYMES and IMDB-MULTI (Kersting et al., 2016), in bioinformatics and social fields, respectively. It is shown in Figure 1 that if the model has small D and W, it can not sufficiently extract the features of graphs. On the contrary, when the model structure is too complicated, overfitting occurs frequently. Thus, CapsGNN is sensitive to changes in depth and width just because of the vectorized graph representation in it. Since the impacts of D
and W are distinct on different datasets, it is necessary to adaptively adjust the model architecture according to the latent characteristics of particular workloads. Figure 1: Results of classification accuracy when increasing D and W. The third subfigure depicts the changes of both D and W, and the x-axis represents the increase of either of them (for simplicity, the growth scales of D and W are set the same here).
Answer: CapsGNN extracts multi-scale node features from different convolutional layers and represents the extracted features in the form of capsules. Compared to classic GCN models, CapsGNN has been proven to have better representation ability on graph data. Nevertheless, CapsGNN is sensitive to changes in depth and width just because of the vectorized graph representation in it, as shown in Figure 1. Therefore, based on CapsGNN, we propose an RL-based assistant module (AM) to find optimal depth-width settings on different datasets. Note that we also added an experiment on the effectiveness of AM on GIN, the model with the best classification performance in all the baseline methods, and recorded corresponding accuracy results in Table 3.