Title: Rubik's Cube: High-Order Channel Interactions with a Hierarchical Receptive Field
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image restoration

low-light image enhancement

image de-noising
Abstract: Image restoration techniques, spanning from the convolution to the transformer paradigm, have demonstrated robust spatial representation capabilities to deliver high-quality performance.Yet, many of these methods, such as convolution and the Feed Forward Network (FFN) structure of transformers, primarily leverage the basic first-order channel interactions and have not maximized the potential benefits of higher-order modeling. To address this limitation, our research dives into understanding relationships within the channel dimension and introduces a simple yet efficient, high-order channel-wise operator tailored for image restoration. Instead of merely mimicking high-order spatial interaction, our approach offers several added benefits: Efficiency: It adheres to the zero-FLOP and zero-parameter principle, using a spatial-shifting mechanism across channel-wise groups. Simplicity: It turns the favorable channel interaction and aggregation capabilities into element-wise multiplications and convolution units with $1 \times 1$ kernel. Our new formulation expands the first-order channel-wise interactions seen in previous works to arbitrary high orders, generating a hierarchical receptive field akin to a Rubik's cube through the combined action of shifting and interactions. Furthermore, our proposed Rubik's cube convolution is a flexible operator that can be incorporated into existing image restoration networks, serving as a drop-in replacement for the standard convolution unit with fewer parameters overhead. We conducted experiments across various low-level vision tasks, including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring. The results consistently demonstrate that our Rubik's cube operator enhances performance across all tasks. Code is publicly available at https://github.com/zheng980629/RubikCube.

Rubik's Cube: High-Order Channel Interactions with a Hierarchical Receptive Field

Abstract

Image restoration techniques, spanning from the convolution to the transformer paradigm, have demonstrated robust spatial representation capabilities to deliver high-quality performance. Yet, many of these methods, such as convolution and the Feed Forward Network (FFN) structure of transformers, primarily leverage the basic first-order channel interactions and have not maximized the potential benefits of higher-order modeling. To address this limitation, our research dives into understanding relationships within the channel dimension and introduces a simple yet efficient, high-order channel-wise operator tailored for image restoration. Instead of merely mimicking high-order spatial interaction, our approach offers several added benefits: Efficiency : It adheres to the zero-FLOP and zero-parameter principle, using a spatial-shifting mechanism across channel-wise groups. Simplicity : It turns the favorable channel interaction and aggregation capabilities into element-wise multiplications and convolution units with 1 × 1 kernel. Our new formulation expands the first-order channel-wise interactions seen in previous works to arbitrary high orders, generating a hierarchical receptive field akin to a Rubik's cube through the combined action of shifting and interactions. Furthermore, our proposed Rubik's cube convolution is a flexible operator that can be incorporated into existing image restoration networks, serving as a drop-in replacement for the standard convolution unit with fewer parameters overhead. We conducted experiments across various low-level vision tasks, including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring. The results consistently demonstrate that our Rubik's cube operator enhances performance across all tasks. Code is publicly available at https://github.com/zheng980629/RubikCube.

1 Introduction

Image restoration aims at restoring a high-quality image from its degraded counterpart. This task is inherently ill-posed, given that a multitude of valid solutions could potentially exist for a single degraded image. The convolutional neural network (CNN), with its remarkable learning capabilities, has been a driving force behind the surge of learning-driven methodologies for image restoration, including the residual block [1, 2], dense connection [3, 4], multi-stage strategy [5, 6], and dropout [7].

High-order spatial modeling. As shown by recent studies [8, 9, 10, 11, 12], vision transformers have overtaken convolutional neural networks as the leading method for image restoration. The key factor driving this shift is the powerful mechanism for modeling spatial interactions inherent in vision transformers. Unlike the local first-order spatial modeling found in the convolution family [13, 14, 15], vision transformers [16] employ a dot-product self-attention paradigm. This paradigm performs matrix multiplication among queries, keys, and values, thereby enhancing the model's capacity by

Conv

Conv

C

H

W

(a) Receptive Field of a Standard Convolution Layer

Rubik`s

Rubik`s

C

H

W

(b) Receptive Field of a Rubik`s Cube Convolution Layer

1

4

Interaction-Order

two successive global spatial interactions. Rao et al. [17] investigate the reasons behind the superior performance of vision transformers compared to convolutional networks. The study concludes that the credit predominantly goes to the high-order spatial interaction modeling of vision transformers.

Motivation. While significant strides have been made in exploring the spatial dimension, the interaction inherent in the alternative channel dimension remains largely untapped. Specifically, most existing image restoration methods, such as the convolution unit and the transformer's FFN layer (which can be considered as a 1 × 1 convolution), only benefit from the first-order channel interaction through linear channel-wise weighting, leaving ample opportunity for exploring high-order modeling. This observation motivates us to explore the modeling of channel-dimension relationships.

Solution. The primary goal of this paper is to examine the potential of channel interactions and propose an alternative operator for high-order channel-wise modeling in the context of image restoration. To achieve rich and meaningful channel interactions, we introduce a high-order channelwise operator that is simple, effective and efficient. Specifically, instead of merely imitating high-order spatial interaction, our operator is uniquely formulated on the principles of a zero-FLOP and zeroparameter spatial-shifting mechanism, which is applied over channel-wise groups. We achieve high-order channel interactions by performing element-wise multiplications among these groups. Our newly proposed operator extends beyond the first-order channel-wise interactions found in prior works, raising them to arbitrary orders without introducing significant computational overhead. Through the combination of shifting and channel interactions, our operator constructs a hierarchical, Rubik's cube-like receptive field (see Figure 1). As such, we have dubbed our operator the Rubik's cube convolution.

The key contribution of this paper is a novel high-order operator featuring a Rubik's cube-like hierarchical receptive field. The Rubik's cube convolution we propose can be seamlessly integrated into existing image restoration networks, serving as a drop-in replacement for the standard convolution unit, but with a reduced parameter count. Comprehensive experiments across various low-level tasks, including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring, demonstrate the effectiveness of our operator.

2 Related Work

Image Restoration. Image restoration is tasked with the recovery of a latent clean image from observations corrupted by degrading factors such as noise [18, 19, 20], blur [21, 22], or inadequate lighting conditions [23, 24, 25, 26, 27]. It is a fundamentally ill-posed problem, as an infinite number of feasible solutions can be conceived for a single degraded image. Traditional image restoration techniques address this issue by formulating the task as an optimization problem, assuming specific priors of the latent high-quality images to regularize the solution space. For instance, the low-rank prior [28] and total variation regularization [29] have been devised for image deblurring, while histogram distribution prior [30] and bright channel prior [31] have been developed for low-light image enhancement. Despite their effectiveness, these hand-crafted priors are challenging to design and often exhibit limited representational capabilities in complex scenarios, restricting their practical applicability. The advent of deep learning has sparked a surge in learning-based algorithms within the image restoration community. Early works in this domain, such as SRCNN [32] for super-resolution and DnCNN [33] for image denoising, leveraged stacked convolution layers to capture effective feature representations. Subsequent research introduced various powerful model designs, such as residual blocks [1, 2], dense connections [3], multi-stage strategies [5], and dropout [7], to image restoration tasks. Furthermore, researchers have begun to probe into spatial and channel interaction mechanisms [34, 35, 9], aiming to excavate more meaningful degradation and content cues within the feature space of image restoration networks.

Channel-Dimension Modeling. While the self-attention mechanism within the Transformer [16] has shown the importance of modeling spatial interaction for vision tasks, fewer studies have explored the channel relationship, especially in image restoration tasks. Zhang et al. [36] integrated channel-wise attention to enhance performance in super-resolution. Chen et al. [6] applied instance normalization to half of the intermediate features and kept the remaining content information untouched to improve model capacity. Li et al. [37] performed hybrid-distorted image restoration using a feature-level divide-and-conquer strategy, disentangling the feature representation of different distortions into distinct channels. Existing approaches typically benefit explicitly from first-order channel-wise interaction or channel disentanglement, neglecting its potential for high-order interaction.

3 Rubik's Cube

3.1 Revisit Channel Interaction

The global spatial interaction in the self-attention family [8, 9, 11, 12] has validated its effectiveness in the image restoration community. Hornet [17] further demonstrates that the key ingredient behind its success is the high-order spatial interaction modeling.

Deepening into the channel dimension, we first revisit channel interaction modeling within the CNN and transformer families. The standard convolution layer and the inherent FFN layer in transformers only apply linear channel-wise weighting to the input feature. The Squeeze-andExcitation (SE) block [38, 39, 40, 41] accomplishes first-order channel interaction by adaptively recalibrating channel-wise feature responses using the squeezed input feature. Restormer [10], with the self-attention mechanism, carries out two consecutive first-order interactions in the channel dimension by conducting matrix multiplication among queries, keys, and values. These operations only explore the first-/second-order channel interaction implicitly.

3.2 Rubik's Cube Convolution

In order to devise an operator that effectively and efficiently facilitates high-order interaction among channels, we turn to the principles of spatial shifting, a mechanism that operates with zero FLOPs and does not require any parameters. We apply this spatial shifting to each group with channel dimension, thereby enabling high-order channel interaction. The details of the shift operations, high-order channel interaction, and the associated receptive field are discussed in the following paragraphs.

Shift Operation. As shown in Figure 2, given an input feature map X ∈ R H × W × C , we evenly divide X into five parts by the channel dimension, where the first is kept unchanged and the remaining four ones are shifted in a distinct spatial direction: left, right, top, and down. Subsequent to the shifting operation, we discard out-of-focus pixels and any vacant pixels are filled with zeros.

Input Feature

Split

Identity

Conv

Conv

C

Conv

Conv

Conv

Shift

Shift

Shift

Shift

The shifted feature ˆ X can be written as:

where C id is the number of channels of the unchanged identity part, C g is the number of channels of a shifted group, and C id +4 ∗ C g = C . In this work, the default shift of pixel is set to 1, and its robustness is validated in Sec. 4.4. Next, the shifted feature ˆ X is split into ˆ X ori ∈ R H × W × C id and { ˆ X c1 , ˆ X c2 , ˆ X c3 , ˆ X c4 } ∈ R H × W × C g along the channel dimension. The zero-FLOP and zeroparameter shifting operation ensure the efficiency of the proposed Rubik's cube convolution.

High-order Channel Interaction. After the shifting and grouping operations, we proceed to model the channel-wise relationship as depicted in Figure 3. More precisely, we use a convolution layer with a 1 × 1 kernel to consolidate information within a shifted group. Subsequently, we facilitate a first-order channel-wise interaction by performing an element-wise multiplication with the succeeding shifted group. The first-order channel interaction can be written as follows:

where ⊙ indicates the element-wise multiplication. Then, the high-order channel interaction can be successively achieved as:

Finally, we aggregate the original unchanged feature and the advanced feature after the high-order channel interaction to obtain the result of the Rubik's cube convolution:

The details of the designed operator and its pseudo-code are summarized in Figure 2 The proposed Rubik's cube convolution achieves high-order channel interaction and information aggregation by element-wise multiplication and convolution layers with 1 × 1 kernel, which is simple to implement.

Rubik's Cube-like Hierarchical Receptive Field. Examining from the channel dimension, our proposed Rubik's cube convolution segregates the input feature into five non-overlapping groups, thereby establishing element-wise interaction and information aggregation among them. This process encourages the attainment of high-order channel-wise interactions in a simple and effective manner.

Shifting our focus to the spatial dimension, as depicted in Figure 3, we observe that the coupling of the shifting operation with channel interaction progressively expands the receptive field as the channelwise interaction order increases. This results in a stratified receptive field along the channel dimension, reminiscent of a Rubik's cube. As illustrated in Figure 4, the effective receptive field [42] of each group successively broadens post-shifting in each direction. Moreover, it is worth noting that the effective receptive field undergoes further expansion with an increase in the pixels shifted. This mechanism is further dissected in Sec. 4.3 and Figure 6.

3.3 Parameter Analysis

We conduct the complexity analysis between the standard convolution layer with a 3 × 3 kernel and our proposed operator. Assuming the number of input channels and output channels is identical, i.e., C → C and the ratio of the shifted channel is equal to 1 / 2 (default setting in all experiments and its robustness is verified in Sec. 4.4) as an example. The required number of trainable parameters is:

The proposed Rubik's cube convolution is implemented by convolution layers with 1 × 1 kernel over the shifted groups, where the trainable parameters are:

Shift & Group

Conv1x1

Mul &

Conv1x1

Mul &

Conv1x1

Mul &

Conv1x1

Conv

C

C

H

W

C

H

W

෡

෩

𝐗𝐜𝟏

෩

𝐗𝐜𝟐

෩

𝐗𝐜𝟑

෩

𝐗𝐜𝟒

Therefore, the parameter ratio is

4 Experiments

We conduct extensive experiments on multiple image restoration and enhancement tasks to demonstrate the effectiveness of our proposed Rubik's cube convolution. We provide more experimental results on the recognition tasks in the supplementary material.

4.1 Experimental Settings

Image Enhancement. We conducted experiments on two widely used low-light image enhancement datasets: LOL [43] and Huawei [44]. The LOL dataset consists of 500 paired low-/normal images,

(a) Baseline

(b)

෡

(c)

෩

𝐗𝐜𝟏

(d)

෩ 𝐗

𝐜𝟐

(e)

෩ 𝐗

𝐜𝟑

(f)

෩

𝐗𝐜𝟒

中

中

中

中

and we split 485 for training and 15 for testing as the official selection. For the Huawei dataset, we randomly selected 2,200 images for training and kept the remaining 280 images for testing purposes. To evaluate the effectiveness of our proposed RubikConv, we compared it against two classical algorithms, namely SID [48] and DRBN [49], which served as baselines in our evaluation.

Image De-blurring. We evaluate RubikConv using the methodology of MPRNet [5] and Restormer [10] on the GoPro dataset [45], which includes 2,103 training pairs and 1,111 testing pairs. Furthermore, we evaluate the generalization capability of RubikConv on diverse datasets: HIDE [46] (synthetic), RealBlur-R [47] (real-world), and RealBlur-J [47] (real-world). The generalization test is performed using the GoPro-trained model. For comparison, we adopt three baselines: DeepDeblur [45] (classical), MPRNet [5], and Restormer [10] (latest image de-blurring techniques).

Image Denoising. Following [50, 10], we choose the popular SIDD dataset [51] as the training benchmark and evaluate the effectiveness of the proposed RubikConv on the test set of the SIDD dataset [51] and the DND dataset [52]. We employ three representative image denoising algorithms, DnCNN [33], MPRNet [5], and Restormer [10], as the baselines.

Guided Image Super-resolution. Following [53, 54], we evaluate the proposed RubikConv on the pan-sharpening task, a representative guided image super-resolution task. For comparison, we select a classical method (PanNet [55]) and two state-of-the-art algorithms (MutNet [56] and INNformer [57]) as baselines and evaluate on the WorldView II and GaoFen2 datasets [57, 58].

We employ widely-used image quality assessment metrics, including peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), relative dimensionless global error in synthesis (ERGAS)[59], and spectral angle mapper (SAM)[60].

4.2 Implementation Details

Based on the above-mentioned competitive baselines, we create several variants of the baselines by replacing the standard convolution with the proposed Rubik's cube convolution:

Original : the baseline without any changes;

RubikConv : replacing the standard convolution in the original model with our designed Rubik's cube convolution;

Conv1x1 : a baseline that replaces the RubikConv in the setting of 2) with four convolution layers with 1 × 1 kernel for a fair comparison with approximately the same number of trainable parameters as 2) .

For a fair comparison, we ensure each competitive baseline and its variants are subjected to the same training configuration and optimization strategy.

4.3 Comparison and Analysis

Quantitative Comparison. We conduct performance comparisons using the configurations defined above. The quantitative results for low-light image enhancement, image de-blurring, image denoising, and guided image super-resolution are presented in Tables 1 through 4, with the best results highlighted in bold. We observe a performance boost across all datasets and tasks when replacing the standard convolution layer with our proposed Rubik's cube convolution layer. In the case of the denoising and guided super-resolution tasks, our operator not only improves the performance of traditional methods like DnCNN and PanNet, but also helps push the performance of state-of-the-art methods like Restormer and INNFormer. Since Restormer has achieved the upper bound of the performance with a large model capacity, there are no significant performance gains for Restormer. Examining the results for SID/DRBN in Table 1, our Rubik's cube convolution achieves gains of 0.39dB/0.58dB over the original baseline on the LOL [43] dataset, while also improving generalizability on the Huawei [44] dataset with gains of 0.2dB/0.2dB. It also performs better than the 'Conv1x1' variant, with gains of 0.52dB/0.51dB on the LOL dataset. These experiments demonstrate the potential of our Rubik's cube convolution in improving the performance of baseline models with minimal parameter overhead.

Qualitative Comparison. Due to the limited space, we only present the visual results of the low-light image enhancement task in Figure 5. It can clearly demonstrate that incorporating the RubikConv with the original baseline achieves more visually pleasing results. Specifically, in Figure 5 the model with the RubikConv achieves better lightness enhancement and color consistency with the corresponding normal-light images. More results can be found in the supplementary material.

(a) Input

(c) Conv1x1

(d) RubikConv

(b) Original

(a) Baseline

(b)

𝐩 = 𝟏

(c)

𝐩 = 𝟐

(d)

𝐩 = 𝟑

(e)

𝐩 = 𝟒

(f)

𝐩 = 𝟓

Expanding Receptive Field. The vanilla convolution layer enables first-order channel interaction through linear channel-wise weighting, resulting in a consistent receptive field across the channel dimension. In contrast, our RubikConv generates a hierarchical receptive field through shifting and high-order channel-wise interactions. An interesting characteristic of RubikConv is that its receptive field expands as the number of shifted pixels increases. Figure 6 presents the effective receptive field [42] of the DnCNN [33] baseline, in comparison with our RubikConv across various pixel shift values. As can be observed, the baseline's effective receptive field approximates a square, while the RubikConv expands it in the left, right, up, and down directions. Moreover, as the number of shifted pixels increases, the overall size of the effective receptive field grows correspondingly.

4.4 Ablation Studies

To verify the robustness of shifting displacement and the ratio of identity, we perform the comparison over the following configurations:

RubikConv-p : replacing the standard convolution layer with the designed RubikConv where the number of the shifted pixel is ' p ';

RubikConv-r : substituting the standard convolution layer with our designed RubikConv, wherein the proportion of the unchanged identity component is denoted by ' r '.

We conduct ablation studies on the low-light image enhancement task with the SID [48] network and image denoising task with the DnCNN [33] network, and all the experiments with different configurations follow the same optimization strategy.

Number of the Shifted Pixel. To ascertain the influence of the number of shifted pixels in our proposed RubikConv, we conduct a series of experiments using the configuration denoted as 'RubikConvp'. As observed from Table 5 and Figure 7, the performance remains consistent as the number of shifted pixels increases, up to a value of 3. Beyond this point, any further increase in p leads to a decline in performance, potentially due to the increased complexity in reconstructing the structure. Therefore, in all the experiments presented in this paper, we set p to 1 as the default value for the number of shifted pixels.

Ratio of the Shifted Channel. We investigate the impact of the ratio of the shifted channel following the definition in 'RubikConv-r'. Experimental results from both the SID dataset (as shown in Table 5) and the DnCNN network (as depicted in Figure 7) suggest that optimal performance is reached when

37.1992

37.3990

37.4016

37.4123

37.4186

37.4105

0.8954

0.8980

0.8982

0.8986

0.8986

0.8981

0.890

0.892

0.894

0.896

0.898

0.900

37.15

37.20

37.25

37.30

37.35

37.40

37.45

37.50

37.55

Original

1/4

1/3

1/2

2/3

3/4

Ablation Studies of RubikConv-r

PSNR

SSIM

37.1992

37.4123

37.4138

37.4029

37.3919

37.3874

0.8954

0.8986

0.8988

0.8984

0.8980

0.8978

0.890

0.892

0.894

0.896

0.898

0.900

37.15

37.20

37.25

37.30

37.35

37.40

37.45

37.50

37.55

Original

1

2

3

4

5

Ablation Studies of RubikConv-p

PSNR

SSIM

the ratio is set to either 1/2 or 2/3. Further increases in the ratio r subsequently result in a decline in performance. We have set the default ratio r to 1/2 for all experiments conducted in this study.

5 Limitations

Wewill validate the effectiveness of the proposed Rubik's cube convolution on broader low-level tasks, such as image de-raining and de-hazing. As a generic operator, diverse comprehensive architectures integrated with our proposed operator should be explored. The primary aim of our research is beyond designing a universal operator for improving the performance of existing networks. We hope to offer an alternative tool for high-order channel-wise modeling. Consequently, our ongoing efforts will focus on showcasing its effectiveness across broader computer vision tasks.

6 Conclusion

In this paper, we study the modeling of relationships in the channel dimension of image restoration tasks, and propose a Rubik's cube convolution operator. It constructs the high-order channel-wise interactions and creates a Rubik's cube-like hierarchical receptive field in an efficient and effective manner. Note that it can be seamlessly incorporated into existing image restoration networks as a drop-in replacement for the standard convolution unit with fewer parameters. Extensive experiments across four distinct image restoration and enhancement tasks demonstrate the performance gains achieved by replacing our designed operator with the standard convolution uint.

Broader Impact

Image restoration technology holds significant value across diverse fields, including remote sensing, medicine, astronomy, and consumer imaging equipment. Our proposed Rubik's cube convolution has the potential to enhance restoration algorithms by constructing high-order channel-wise relationships and producing a hierarchical receptive field along the channel dimension. This could lead to clearer, more detailed images that improve decision-making capabilities in these fields. While we anticipate no negative societal consequences from our work, future research and applications may reveal unforeseen impacts. Thus, the responsible use and ongoing evaluation of such technology is essential.

References

Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3147-3155, 2017.

Max Ehrlich and Larry S Davis. Deep residual learning in the JPEG transform domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3484-3493, 2019.

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(7):2480-2495, 2020.

Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. MemNet: A persistent memory network for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4539-4547, 2017.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, MingHsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14821-14831, 2021.

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. HINet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 182-192, 2021.

Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6002-6012, 2022.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1833-1844, 2021.

Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17683-17693, 2022.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5728-5739, 2022.

Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xin Yuan, et al. Cross aggregation transformer for image restoration. Advances in Neural Information Processing Systems , 35:2547825490, 2022.

Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. arXiv preprint arXiv:2210.01427 , 2022.

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11030-11039, 2020.

Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. Advances in Neural Information Processing Systems , 29, 2016.

Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Convbert: Improving bert with span-based dynamic convolution. Advances in Neural Information Processing Systems , 33:12837-12848, 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems , 35:10353-10366, 2022.

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189 , 2018.

Zichun Wang, Ying Fu, Ji Liu, and Yulun Zhang. Lg-bpn: Local and global blind-patch network for self-supervised real-world denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18156-18165, June 2023.

Junyi Li, Zhilu Zhang, Xiaoyu Liu, Chaoyu Feng, Xiaotao Wang, Lei Lei, and Wangmeng Zuo. Spatially adaptive self-supervised learning for real-world image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9914-9924, June 2023.

Qi Zhu, Man Zhou, Naishan Zheng, Chongyi Li, Jie Huang, and Feng Zhao. Exploring temporal frequency spectrum in deep video deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 12428-12437, October 2023.

Lingshun Kong, Jiangxin Dong, Jianjun Ge, Mingqiang Li, and Jinshan Pan. Efficient frequency domain-based transformers for high-quality image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5886-5895, June 2023.

Naishan Zheng, Jie Huang, Feng Zhao, Xueyang Fu, and Feng Wu. Unsupervised underexposed image enhancement via self-illuminated and perceptual guidance. IEEE Transactions on Multimedia , 2022.

Naishan Zheng, Jie Huang, Qi Zhu, Man Zhou, Feng Zhao, and Zheng-Jun Zha. Enhancement by your aesthetic: An intelligible unsupervised personalized enhancer for low-light images. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6521-6529, 2022.

Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, and Feng Zhao. Empowering low-light image enhancer through customized learnable priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1255912569, October 2023.

Naishan Zheng, Jie Huang, Man Zhou, Zizheng Yang, Qi Zhu, and Feng Zhao. Learning semantic degradation-aware guidance for recognition-driven unsupervised low-light image enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 3678-3686, 2023.

Jie Huang, Feng Zhao, Man Zhou, Jie Xiao, Naishan Zheng, Kaiwen Zheng, and Zhiwei Xiong. Learning sample relationship for exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9904-9913, June 2023.

Wenqi Ren, Xiaochun Cao, Jinshan Pan, Xiaojie Guo, Wangmeng Zuo, and Ming-Hsuan Yang. Image deblurring via enhanced low-rank prior. IEEE Transactions on Image Processing , 25(7):3426-3437, 2016.

S Derin Babacan, Rafael Molina, and Aggelos K Katsaggelos. Variational bayesian blind deconvolution using a total variation prior. IEEE Transactions on Image Processing , 18(1):1226, 2008.

Chong-Yi Li, Ji-Chang Guo, Run-Min Cong, Yan-Wei Pang, and Bo Wang. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Transactions on Image Processing , 25(12):5664-5677, 2016.

Xueyang Fu, Delu Zeng, Yue Huang, Xinghao Ding, and Xiao-Ping Zhang. A variational framework for single low light image enhancement using bright channel prior. In IEEE Global Conference on Signal and Information Processing , pages 1085-1088. IEEE, 2013.

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In Proceedings of the European Conference on Computer Vision , pages 184-199. Springer, 2014.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing , 26(7):3142-3155, 2017.

Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In International Conference on Learning Representations , 2019.

Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via a holistic attention network. In Proceedings of the European Conference on Computer Vision , pages 191-207. Springer, 2020.

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image superresolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision , pages 286-301, 2018.

Xin Li, Xin Jin, Jianxin Lin, Sen Liu, Yaojun Wu, Tao Yu, Wei Zhou, and Zhibo Chen. Learning disentangled feature representation for hybrid-distorted image restoration. In Proceedings of the European Conference on Computer Vision , pages 313-329. Springer, 2020.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7132-7141, 2018.

Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 510-519, 2019.

Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. ECANet: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11534-11542, 2020.

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision , pages 3-19, 2018.

Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in Neural Information Processing Systems , 29, 2016.

Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep Retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 , 2018.

Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2rnet: Low-light image enhancement via real-low to real-normal network. arXiv preprint arXiv:2106.14501 , 2021.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3883-3891, 2017.

Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5572-5581, 2019.

Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In Proceedings of the European conference on computer vision , pages 184-201. Springer, 2020.

Lijun Zhang, Xiao Liu, Erik Learned-Miller, and Hui Guan. Sid-nism: A self-supervised low-light image enhancement framework. arXiv preprint arXiv:2012.08707 , 2020.

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3063-3072, 2020.

Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17399-17410, 2022.

Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1692-1700, 2018.

Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1586-1595, 2017.

man zhou, Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng, and Chongyi Li. Deep Fourier up-sampling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qi Xie, and Xiangyong Cao. Memoryaugmented deep unfolding network for guided image super-resolution. International Journal of Computer Vision , 131(1):215-242, 2023.

Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. PanNet: A deep network architecture for pan-sharpening. In Proceedings of the IEEE International Conference on Computer Vision , pages 5449-5457, 2017.

Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual informationdriven pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1798-1808, 2022.

Man Zhou, Jie Huang, Yanchi Fang, Xueyang Fu, and Aiping Liu. Pan-sharpening with customized transformer and invertible neural network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3553-3561, 2022.

Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu, Xian Wei, and Feng Zhao. Spatial-frequency domain information integration for pan-sharpening. In Proceedings of the European conference on computer vision , pages 274-291. Springer, 2022.

Luciano Alparone, Lucien Wald, Jocelyn Chanussot, Claire Thomas, Paolo Gamba, and Lori Mann Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing , 45(10):3012-3021, 2007.

Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop , 1992.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Rubik’s cube convolution operator to model the high-order channel-wise interactions. The Rubik’s cube convolution applies the spatial-shifting mechanism across channel-wise groups, which is the zero-FLOP and zero-parameter. Moreover, only the point-wise convolution and dot-product are applied in Rubik’s cube convolution. The results on image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring show the effectiveness of the proposed Rubik’s cube convolution.\n\nSTRENGTHS:\n1. The idea of Rubik’s cube convolution is reasonable. This is a promising step to complement research in modeling high-order channel dimension information.\n2. The design of Rubik’s cube convolution is simple yet efficient and easy to follow since it only includes shift, dot-product and point-wise convolution.\n3. The paper is well-writing. The authors provide extensive analysis of the proposed methods. For example, the illustration of a high-order hierarchically receptive field in Figures 1, 3, and 4, and the corresponding analysis are convincing.\n4. The main results and ablation study are extensive and demonstrate the superiority of the proposed method.\n5. In supplementary material, the authors provide more experiments on high-level and low-level tasks, further showing the effectiveness of the Rubik’s cube convolution.\n\nWEAKNESSES:\n1. The FLOPs of the Rubik’s cube convolution are not provided. Although the parameter of the Rubik’s cube convolution is small (Sec. 3.3), the FLOPs may be large since there are many dot-product in it.\n2. Although the spatial-shifting mechanism is zero-FLOP and zero-parameter, it is a high-latency operation. Therefore, the actual latency of the Rubik’s cube convolution may be high, hindering its application.\n\nQUESTIONS:\n1. The proposed Rubik’s cube convolution structure is similar to the g^nConv proposed in Hornet [17]. Clarify the difference between the Rubik’s cube convolution and g^nConv.\n2. In Figure 6 and Line 230, the receptive field of the network progressively expands as the number of shifted pixels increases. However, the performance of RubikConv-p is best when the p=1. Please give some explanation.\n3. The authors only provide the visual comparison of the low-light image enhancement task and image denoising task. More visual results on guided image super-resolution and image de-blurring should be provided.\n\nLIMITATIONS:\nThe authors have discussed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Rubik’s cube convolution operator, which is very simple and efficient, especially requiring zero FLOPS / parameters. This novel component improves the several low-level vision networks, enabling the high-order channel interaction and enlarging receptive field of the standard convolutions. Simply dividing feature maps into some groups and performing shifting operations makes the existing convolutional layers capture high-order channel interaction beyond first-order interaction. The experimental results show that multiple low-level vision tasks are improved by the proposed Rubik’s cube. However, the limitation could be handled by the authors that the performance gains and the reduced number of parameters are marginal.\n\nSTRENGTHS:\nRubik’s cube convolution does not require the number of operations and parameters. It can also replace any standard CNN architectures. The effective receptive field of the proposed component is successfully enlarged when compared to vanilla CNN, as shown in Fig. 3. The main experimental results, such as Tabs. 1,2,3,4, can show superiority of the proposed component over trivial 1x1 convolution replacement. As a result, the performances of various image restoration networks can be enhanced (but the marginal gains will be mentioned in “weakness”). Moreover, the ablation studies (Tab. 5) demonstrates that the hyper-parameters for Rubik’s cube convolution have been carefully decided. This paper is well-presented and easy to follow.\n\nWEAKNESSES:\n1. The performance gains of Rubik’s cube convolution are somewhat marginal. Especially, the low-light enhancement and deblurring could be further improved. The authors are suggested to consider more robust components to different tasks. Or if the improvement differences from task-by-task are explained and the insight for better variants of Rubik’s cube convolution resulting from these explanations can be drawn, the reviewer thinks this paper can be compelling.\n2. Furthermore, the number of reduced parameters led by the novel architecture is also marginal. So, (if possible) the reviewer requires an additional experiment that highly (e.g., less than 60% of “Original”) reduces the number of parameters of existing networks by changing channels or depths and replaces existing CNN or FFN with Rubik’s cube convolution. If reduced “Original” models with Rubik’s cube convolution are still better than “Original” models while requiring a small number of parameters, the robustness of this work can be reinforced.\n\nQUESTIONS:\nHow about applying Rubik’s cube convolution to QKV projection before self-attention of Transformers? As the authors know, almost all image restoration fields are dominated by Transformer-based methods. And self-attention is the most core component of Transformers. However, this work seems to apply their novel component to only FFN of Transformers. So, the reviewer guesses how much difference could be made by replacement to Rubik’s cube convolution for elements that directly influence self-attention operation (e.g., QKV projection).\n\nLIMITATIONS:\nSee weakness and question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes the Rubik’s Cube, which can replace the standard convolution layer in the traditional paradigm. With shift operation and high-order channel Interaction, the Rubik’s Cube can generate a hierarchical receptive field and activate the potential of channel interactions. Experiment results show that the Rubik’s Cube enhances performance across a variety of low-level vision tasks.

STRENGTHS:
1. This Rubik’s Cube proposed in this paper can be widely applied in the framework where the convolution layers are used. If this operation can prove its effectiveness in multiple tasks, it could make a lot of sense.
2. The Rubik's Cube is easy to implement and requires fewer parameters than the original convolutional layer. It's simple yet effective and efficient.

WEAKNESSES:
1. The experiment results on GaoFen2 make me confused since in terms of ERGAS, RubikConv has made a very big improvement (from 0.9576 to 0.5486). This increase does not fit well with other experimental results.
2. The experiments of many tasks in this paper do not adopt the best methods at present. Many of the experiments were done using methods that were used two or three years ago. In order to make the experimental results more reliable, the paper should provide a comparison with the best method as much as possible.
3. Since This Rubik's Cube splits the original convolutional layer operation into multiple convolutional layer operations, in addition to the number of parameters, it is better to provide inference time to better prove the efficiency improvement of This Rubik's Cube.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
The authors should validate the effectiveness of the proposed Rubik’s cube convolution on broader low-level tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05