Rubik's Cube: High-Order Channel Interactions with a Hierarchical Receptive Field
Authors: Naishan Zheng
Man Zhou
Chong Zhou
Chen Change Loy
Venue: NeurIPS 2023 poster (2023)
Keywords: Image restoration
low-light image enhancement
image de-noising

Abstract: Image restoration techniques, spanning from the convolution to the transformer paradigm, have demonstrated robust spatial representation capabilities to deliver high-quality performance.Yet, many of these methods, such as convolution and the Feed Forward Network (FFN) structure of transformers, primarily leverage the basic first-order channel interactions and have not maximized the potential benefits of higher-order modeling. To address this limitation, our research dives into understanding relationships within the channel dimension and introduces a simple yet efficient, high-order channel-wise operator tailored for image restoration. Instead of merely mimicking high-order spatial interaction, our approach offers several added benefits: Efficiency: It adheres to the zero-FLOP and zero-parameter principle, using a spatial-shifting mechanism across channel-wise groups. Simplicity: It turns the favorable channel interaction and aggregation capabilities into element-wise multiplications and convolution units with $1 \times 1$ kernel. Our new formulation expands the first-order channel-wise interactions seen in previous works to arbitrary high orders, generating a hierarchical receptive field akin to a Rubik's cube through the combined action of shifting and interactions. Furthermore, our proposed Rubik's cube convolution is a flexible operator that can be incorporated into existing image restoration networks, serving as a drop-in replacement for the standard convolution unit with fewer parameters overhead. We conducted experiments across various low-level vision tasks, including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring. The results consistently demonstrate that our Rubik's cube operator enhances performance across all tasks. Code is publicly available at https://github.com/zheng980629/RubikCube.

Rubik's Cube: High-Order Channel Interactions with a Hierarchical Receptive Field
Naishan Zheng" Man Zhou Chong Zhou Chen Change Loy
S-Lab, Nanyang Technological University
naishan zheng, man zhou, chong033, ccl ccloyyntu.edu edu.sg
Abstract
Image restoration techniques, spanning from the convolution the transformer paradigm, have demonstrated robust spatial representation capabilities to deliver high-quality performance. Yet, many of these methods, such convolution and the Feed Forward Network (FFN) structure of transformers, primarily leverage the basic first-order channel interactions and have not maximized the potential benefits of higher-order modeling. To address this limitation, our research dives into understanding relationships within the channel dimension and introduces a simple yet efficient, high-order channel-wise operator tailored for image restoration. Instead of merely mimicking high-order spatial interaction, our approach offers several added benefits: Efficiency: It adheres to the zero-FLOP and paraameeet principle, using spatial-shifting mechanism across channel-wis groups. Sim- plicity: It turns the favorable channel interaction and aggregation capabilities into element-wist multiplications and convolution units with formulation expands the first-order channel-wise interactions seen in previous works to arbitrary high orders, generating hierarchical receptive field akin a Rubik's cube through the combined action of shifting and interactions. Fur- thermore, our proposed Rubik's cube convolution flexible operator that can be incorporated into existing image restoration networks, serving as drop-in replacement for the standard convolution unit with fewer parameters overhead. We conducted experiments across various low-level vision tasks, including im- age denoising, low-light image enhancement, guided image super-resolution, and image de-blurring The results consistently demonstrate that our Rubik's 's cube operator enhances performance across all tasks. Code publicly available at https///gghlbb..ooomm/o/o/gos/ct
kernel. Our new
Introduction
Image restoration aims at restoring a high-quality image from its degraded counterpart. This task inherently ill-posed, given that multitude of valid solutions could potentially exist for a single degraded image. The convolutional neural network (CNN), with its remarkable learning capabilities, has been driving force behind the surge of learning-drie -drvvn methodologies for image restoration, including the residual block 2 dense connection
multi-stage strategy 5 6
and dropout 7
vision transformers have overtaken convolutional neural networks as the leading method for image restoration. The key factor driving this shift is the powerful mechanism for modeling spatial interactions inherent vision transformers. Unlike the local first-order spatial modeling found in the convolution family [13 [4 15). vision transformers [16] employ a dot-product self-attention paradigm. This paradigm performs matrix multiplication among queries, keys, and values, thereby enhancing the model's 's capacity by
High-order spatial modeling. As shown by recent studies 8
10
'These authors contributed equally to the work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Car
Can
C
W
H
(a) Receptive Field of Standard Convolution
OnteradiionOdder
Rubik's
CubeCar
Rubik's Cubr:
C
4
W
(b) Receptive Field of a Rubik's Cube Landutonnt
Figure 1: Comparison between the vanilla convolution and the proposed Rubik's cube convo- lution operator. Different from the previous first-order linear weighting in the channel dimension, the proposed operator formulates high-order channel interactions and produces a Rubik's ss cube-like hicrarchical receptive field (the receptive field expands as the order of channel interaction increases).
two successive global spatial interactions. Rao et al. [17] investigate the reasons behind the superior performance of vision transformers compared to convolutional networks. The study concludes that the credit predominantly goes to the high-order spatial interaction modeling of vision transformers.
Motivation. While significant strides have been made in exploring the spatial dimension, the interaction inherent in the alternative channel dimension remains largely untapped. Specifically, most existing image restoration methods, such as the convolution unit and the transformer's FFN layer (which can be considered through linear channel-wise weighting, leaving ample opportunity for exploring high-order modeling. This observation motivates to explore the modeling of channel-dimension relationships.
x convolution), only benefit from the first-order channel interaction
Solution. The primary goal of this paper is examine the potential of channel interactions and propose an alternative operator for high-order channel-wise modeling in the context of image restoration. achieve rich and meaningful channel interactions, we introduce a high-order channel- wise operator that is simple, effective and efficient. Specifically, instead of merely imitating high-order spatial interaction, our operator uniquely formulated on the principles of zero-FLÖP and zero- parameter spatial-shifting mechanism, which is applied over channel-wise groups. We achieve high-order channel interactions by performing element-win multiplications among these groups. Our newly proposed operator extends beyond the first-order channel-wise interactions found in prior works, raising them to arbitrary orders without introducing significant computational overhead. Through the combination of shifting and channel interactions, operator constructs hierarchical, Rubik's ss cube-like receptive field (see Figure T. As such, we have dubbed our operator the Rubik's cube convolution.
The key contribution of this paper is a novel high-order operator featuring a Rubik's cube-like hierarchical receptive field. The Rubik's cube convolution we propose can be seamlessly integrated into existing image restoration networks, serving as a drop-in replacement for the standard convolution unit, but with a reduced parameter count. Comprehensive experiments across various low-level tasks, including image denoising, low-light image enhancement, guided image super-resolution, and image de- de-blurinng, demonstrate the effectiveness of our operator.

---

2 Related Work
Image Restoration, Image restoration tasked with the recovery of a latent clean image from observations corrupted by degrading factors such noise [18] 19 20) blur [21 22], or inadequate lighting conditions [23 24 25 26 27]. It is fundamentally ill pol-posed problem, as an infinite number of feasible solutions can be conceived for a single degraded image. Traditional image restoration techniques address this issue by formulating the task as an optimization problem, assuming specific priors of the latent high-quality images regularize the solution space. For instance, the low-rank prior (28) and total variation regularization (29] have been devised for image deblurring, while histogram distribution prior [30] and bright channel prior [31 have been developed for low-light image enhancement. Despite their effectiveness, these hand-crafted priors are challenging to design and often exhibit limited representational capabilities in complex scenarios, restricting their practical applicability. The advent of deep learning has sparked a surge in learning- algorithms within the image restoration community. Early works in this domain, such SRCNN ?2 for super-resolution and DrCNN [33] for image denoising, leveraged stacked convolution layers to capture effective feature representations. Subsequent research introduced various powerful model designs, such as residual blocks 21. dense connections [3, multi-stage strategies [3], and dropout [7]. to image restoration tasks. Furthermore, researchers have begun to probe into spatial and channel interaction mechanisms (34 (5] 9) aiming to excavate more meaningful degradation and content cues within the feature space of image restoration networks.
Channel-Dimennion Modeling, While the self-attention mechanism within the Transformer [6 has shown the importance of modeling spatial interaction for vision tasks, fewer studies have explored the channel relationship, especially in image restoration tasks. Zhang et al. (36) integrated channel-wise attention to enhance performance in super-resolution. Chen et al. (6.(6) applied instance normalization to half of the intermediate features and kept the remaining content information untouched to improve model capacity. Li et al. [37] performed hybrid- image restoration using a feature-level divide-and -nd--uquere strategy. disentangling the feature representation of different distortions into distinet channels. Existing approaches typically benefit explicitly from first-order channel-wise interaction or channel disentanglement, neglecting its potential for high-order interaction.
3 Rubik's Cube
3.1 Revisit Channel Interaction
The global spatial interaction in the self-attention family || 11
has validated its effectiveness in the image restoration community, Hornet (17) further demonstrates that the key ingredient behind its success is the high-order spatial interaction modeling.
12
Deepening into the channel dimension, we first revisit channel interaction modeling within the CNN and transformer families. The standard convolution layer and the inherent FFN layer in transformers only apply linear channel-wise weighting to the input feature. The Squeeze-and- accomplishes first-order channel interaction by adaptively recalibrating channel-win -wiee feature responses using the squeezed input feature. Restormer [10), with the self-attention mechanism, carries out two consecutive first-order interactions in the channel dimension by conducting matrix multiplication among queries, keys, and values. These operations only explore the first-/second-orde channel interaction implicitly.
Excitation (SE) block [38 39 40
3.2 Rubik's Cube Convolution
In order to devise an operator that effectively and efficiently facilitates high-order interaction among channels, we turn to the principles of spatial shifting, a mechanism that operates with zero FLOPs and does not require any parameters. We apply this spatial shifting to each group with channel dimension, thereby enabling high-order channel interaction. The details of the shift operations, high-order channel interaction, and the associated receptive field are discussed in the following paragraphs.
Shift Operation. As shown in Figure given an input feature map E HxXWxC we evenly divide X into five parts by the channel dimension, where the first is kept unchanged and the remaining four ones are shifted in a distinet spatial direction: left, right, top, and down. Subsequent to the shifting operation, we discard out-of-focus pixels and any vacant pixels are filled with zeros.
3

---

Input Feature
..InEs. (ouif, tr,chernala, espar (..ti__..))
Split
well tn,channais greup coss_aiiis Im salf
welf g group
renge(03)
Iduenity
Shift
Shift
Shift
Shift
Caavialin chazosle tn,chamnnla,
ded sbittinglawi,, s, seal: aats torch aeros_1stat))
saster, ress, orbiongrees chittl,, orieing oriving. 1:3
=Ei.rors, to sientity
stc, orsiiggeesiinng 5,
ori-ing eriting, 1-1]
Corw
sastel, oni-ing briggeninnng 35, 3 sts, ortring cesting. 15l, 3] doon
sazrtf, azte ceaning. 1-1, xft, orar2ng craving. 1]
return shift
Coro
ruthifted self chifting(s, ozs, sashitted terch uptis sbiffed, (oulf craeld .). dim-1)
self
shtre_ sarct cheskts shifteed phrtte 4, dieri)
s.leds selt cas aas. (oaifr x.right neld atsr tett atsrs_tan(()) seap col.-co (ourigh abirt_]lie(())) torch (ast, s_left, retern self. colf.coor.oooceee)
CoLv
s_domn), diavi)
Figure 2: Overview of the Rubik's cube convolution layer. stands on the principle of zero-FL.OP and zero- prooparmeeter apatial-shifting mechanism over channel-wise groups. It achieves high-orde channel interactions with clement-wise multiplications and convolution layers with x 1 kernel, rendering it simple and efficient. We also provide the corresponding PyTorch-style code the right.
The shifted feature X can be written as:
X[0: H, : Ca.] < x W,O: Cia],
{ H, : W. Cia Cai + C₂] < X(0: H, W - 1, C.i Cia + Cy)s
X : H, W 1--,,, Ca + Cg Ci + 2C,] < X(0: 1: W, Cia + Cg Cia + 2C,],
(I)
{0: H - 1,0: W, Cia + 20g- Ci + 3C,] < X|1 H,0:: W, Cia + 2Cg Cia + 3C,l
X[1 H. W, Cia + 3Cg- Ci + 4C,] X H - 1,0 W, Cia + 3C, Cia + 4C,).
where Cja is the number of channels of the unchanged identity part, Cg is the number of channels of a shifted group. and Cia + 4* C, = C. In this work, the default shift of pixel is set to 1, and its robustness is validated in Sec. 4.4 Next, the shifted feature X is split into X HXXCa and (Xer X Xes, Xet} IR H along the channel dimension. The zero-FLOP and zero- parameter shifting operation ensure the efficiency of the proposed Rubik's 's cube convolution.
High-order Channel Interaction. After the shifting and grouping operations, proceed model the whis relationship as depicted in Figure 3 More precisely, we use a convolution layer with 1 kernel to consolidate information within shifted group. Subsequently, facilitate first-order channel-wise interaction by performing an element-wiss multiplication with the succeeding shifted group. The first-order channel interaction be written follows:
X Convix1 (Xxr),
(2)
2 = Convixi (X.1 @ X
where 3 indicates the element-wise multiplication. Then, the high-order channel interaction can be successively achieved as:
X = (oovIxx(Xx22 @ Xea),
(3)
X.1 = Convix1 (Xes @ Xea).
Finally, we aggregate the original unchanged feature and the advanced feature after the high-order channel interaction to obtain the result of the Rubik's eube convolution:
Xout Convix 1 (Concat [Rari, Xx1, X.2, X.s, X
(4)
The details of the designed operator and its pseudo-code are summarized in Figure 2 The proposed Rubik's 's cube convolution achieves high-order channel interaction and information aggregation by element-wise multiplication and convolution layers with
kernel, which is simple implement.
Rubik's Cube-like Hierarchical Receptive Field. Examining from the channel dimension, our proposed Rubik's 's cube convolution segregates the input feature into five non-overlapping groups, thereby establishing element-wise interaction and information aggregation among them. This process encourages the attainment of high-order channel-win interactions simple and effective manner.

---

Shifting our focus to the spatial dimension, as de- picted Figure we observe that the coupling of the shifting operation with channel interaction pro- gressively expands the receptive field as the channel- wise interaction order increases. This results in stratified receptive field along the channel dimen- sion, reminiscent of Rubik's cube. As illustrated in Figure the effective receptive field 42 each group successively broadens post-shifting in each di- rection. Moreover, is worth noting that the effective receptive field undergoes further expansion with an increase in the pixels shifted. This mechanism further dissected in Sec. 4.3 and Figure 6
H
W
Mui Cuny
X
Nx1
Mid & Cony/cl
Pari
3.3 Parameter Analysis
Shifft & Group
We conduct the complexity analysis between the stan- dard convolution layer with 3 x 3 kernel and our proposed operator. Assuming the number of imput channels and output channels is identical, i.e., C - C 1/2
W
and the ratio of the shifted channel is equal (default setting in all experiments and its robustness verified Sec. 4.4) as an example. The required number of trainable parameters is:
Figure Illustration of Rubik's cube- like hierarchically receptive field. Through the combined action of shifting and interac- tions, the effective receptive field [42] of each group gradually expands with the order of channel-wise interaction increases. The red- shaded region indicates the receptive field of the corresponding group
(5) The proposed Rubik's cube convolution is imple- mented by convolution layers with 1 x kernel over the shifted groups, where the trainable parameters are:
ConVpara x x C x C= 9C
RubikConvpara Therefore, the parameter ratio is
(1x1x + C2
(6)
RubikConv para
17
ConVpars
144
Experiments
We conduct extensive experiments multiple image restoration and enhancement tasks to demon- strate the effectiveness of our proposed Rubik's 's cube convolution. We provide more experimental results on the recognition tasks in the supplementary material.
4.1 Experimental Settings
Image Enhancement. We conducted experiments two widely used low-light image enhancement datasets: LOL 43 and Huawei [44]. The LOL dataset consists of 500 paired low-/normal images,
(a) Baseline (b) 8
(c) X
(d)
(e) Xxx
Xx
Figure 4: Visualization of the effective receptive field 42 (a) The baseline indicates the effective receptive field the default architecture, and the last five describe the effective receptive field (b) identity group, X (c) left-shifting group, X.1. (d) right-shifting group. X (c) up-shifting group, X.s, (f) and down-shifting group, X.4 after interactions in the Rubik's 's cube convolution.
5

---

Table I: Quantitative comparison of low-light image enhancement the LOL
43 and Huawei 44
datasets. + B means generalization whose source dataset is A and the target dataset is B.
LOL.
Huawei
Huawei +
++ LOL
Model
Config
#Params
PSNR
SSIM
PSNR
SSTM
PSNR
SSIM
PSNR
SSTM
Original
19.7931
0,8361
17.7929
0.6247
20. 20.1549
0.6851
18.0856
0.7543
0.55M
DRBN
Convix1
19.8648
0.8340
16.2748
18
20.0189
0.6854
17.9643
0.7518
0.55M
RubikConv
20.3769
0.8400
17.9811
0.6337
20.2363
0.6876
18.2066
0.7543
0.55M
Original 20.1062
0.7895
16.5874
0.5925
20.1742
0.6659
18.5468
0.7441
7.76M
SID
Convlx1 19.9709
0.7753
16.3357
0.5780
20.1742
0.6659
18.5468
0.7441
7.7IM
RubikConv
20.4972
0.7979
16.7867
0.5941
20.2044
0.6665
18.6600
0.7476
7.69M
Table 2: Quantitative comparison of image de-bluring. The model is only trained on the GoPro 45 training set and directly tested the GoPro 45 testing set, HIDE [46]. and RealBlur 47 datasets.
DeepDeblur
MPRNet
Restormer
Model
Metric
Original Conv 1xl RubikConv Original Conv Ixl RubikConv Original Conv 1xl RubikConv
PSNR
28.9423 28.8247
29.1919
32.6546
32,5638
32.7197
32.9117
32.7920
32.9305
GoPro
SSIM
0,8716
0.8708
0.8761
0,9576
0.9566
0,9579
0.9603
0.9590
0.9608
PSNR
26,9770
26.9094
27.2508
30.9181
30.8306
30,9243
30.1568
30.0827
30.1977
HIDE
SSIM
0.8468
0.8460
0.8525
0.9377
0.9358
0.9377
0.9405
0.9397
0.9411
PSNR
1580
26.0855
26.3216
28.6370
28.5714
28.7067
28.9636
28.9273
28.9698
RealBlur-J
SSIM
0.8094
0.8088
0.8127
0.8706
0.8649
0.8710
0.8792
0.8785
0.8796
PSNR
33,5516
33.4871
33.6879
35,9682
35,8307
36.0277
36.2017
3501
36.2136
RealBlur-R
SSIM
0.9359
0,9349
0.9377
0,9472
0.9325
0.9476
0.9572
0.9563
0.9575
#Params
11.72M 11.46M
10.51M
15.74M
13.89M
13.33M
25,31M
25.06M
24.92M
and we split 485 for training and 15 for testing as the official selection. For the Huawei dataset, we randomly selected 2,200 images for training and kept the remaining 280 images for testing purposes. To evaluate the effectiveness of our proposed RubikConv, we compared it against two classical algorithms, namely SID (48) and DRBN
49 which served as baselines in our evaluation.
Image De-blurring. We evaluate RubikConv using the methodology of MPRNet 5 and Restormer [10] on the GoPro dataset [45), which includes 2,103 training pairs and 1,111 test- ing pairs. Furthermore, we evaluate the generalization capability of RubikConv on diverse datasets: HIDE 446] (synthetic), RealBlur-R [47 (real-world), and RealBlur-J
47 (real-world). The general- ization test is performed using the GoPro-trained model. For comparison, we adopt three baselines:
DeepDeblur 45 (classical), MPRNet [5) and Restormer [10 (latest image de-blurrin techniques).
Image Denoising. Following 50 10 we choose the popular SIDD dataset [5I] as the training benchmark and evaluate the effectiveness of the proposed RubikConv the test of the SIDI) dataset [51] and the DND dataset
[52]. We employ three representative image denoising algorithms,
DnCNN 33 MPRNet [ and Restormer [10], the baselines.
Guided Image Super-resolution. Following 53 54], we evaluate the proposed RubikConv on the
pan-sharpening task, a representative guided image super-resolution task. For comparison, select classical method (PanNet (55)) and two state-of-the- -rrt--rrttt--tttee algorithms (MuiNet (56) and INNformer (57)
baselines and evaluate on the WirldView II and GaoFen2 datasets [7 58
We employ widely-used image quality assessment metrics, including peak signal-to- ra-
tio (PSNR). structural similarity index (SSIM), relative dimensionless global error in synthesis (ERGAS) 59) and spectral angle mapper (SAM)
4.2 Implementation Details
Based on the alove-mentioned competitive baselines, we create several variants of the baselines by replacing the standard convolution with the proposed Rubik's 's cube convolution:
1) Original: the baseline without any changes;

---

Table 3: Quantitative comparison of image denoising. 'hee model is trained only the SIDD [51 training set and directly tested on the SIDD [1]] testing set and DND [52] dataset.
DrCNN
MPRNet
Restormer
Model
Metric
Original Conv 1xl RubikConv Original Convlx
RubikConv
Original Conv CxlvIxl RubikConv
PSNR
37.1992
37.2247
37.4123
39.7129
39.6220
39.7741
40.0124
39.9429
40.0853
SIDD
SSIM
0.8954
0.8955
0.8986
0,9524 0,9502
0.9536
0.9587
0,9568
0.9593
PSNR
38,3305
38,3946
38.5759
39.6276
39.5834
39.9043
40.0182
39,9885
40.0994
DND
SSIM
0,9307
0,9296
0.9328
0,9531
0,9524
0.9558
0,9560
0,9537
0,9564
#Params
1.51M
1.36M
1.28M
15.74M
13.89M
13.33M
25.31M 25.2
25.21M
Table 4: Quantitative comparisons of guided image super-resolution.
WorldView- III
GaoFen2
Model
Config
#Params
PSNR SSIM SAMI
ERGAS
PSNRT SSIM SAM, EGAS!
Original
40.8212
0.9630 0.0257
1.0561
42.1699 0.9569 0.0192 0.9565
0.068M
PanNet
Convix1
40.8015
0.9624 0.0261
1.0576
42.1150 0.9527 0.0197 0.9576
0.061M
RubikConv
41.2692
0.9646 0.0248
1.0237
43.1764 0.9693 0.0176 0.8516
0.047M
Original
41.0692 0.9646 0.0248
1.0237
47.1699
0.9569 0.0192 0.9565
0.116M
MutNet
Conv1x1
40.9372 0.9624 0.02749
1.0072
47.1668
0.9563 0.0185 0.9576
0.115M
RubikConv
41.5258
0.9692 0.0230
0.9719
47.3274
0.9885 0.0103 0.5486
0.115M
Original
41 41,5363 0.9696 0.0229
0.9649
47.1108
0.9882 0.0105 0.5610
0.061M
INNFormer
Cxl
41.5276 0.9696 0.0226
0.9451
47.1295
0.9882 0.0104 0.5611
0.060M
RubikConv
41.5720 0.9699 0.0228
0.9649
47.2458 0.9886 0.0104 0.5609
0.060M
2) RubikConv: replacing the standard convolution in the original model with designed
Rubik's ' cube convolution;
3) Convix1: baseline that replaces the RubikConv in the setting of 2) with four convolution layers with x kernel for a fair comparison with approximately the same number of trainable parameters 2).
For a fair comparison, we ensure each competitive baseline and its variants are subjected to the same training configuration and optimization strategy.
4.3 Comparison and Analysis
Quantitative Comparison. We conduct performance comparisons using the configurations defined above. The quantitative results for low-light image enhancement, image de-blurring, image denois- ing, and guided image super-resolution are presented in Tables through [4] with the best results highlighted in bold. We observe a performance boost across all datasets and tasks when replacing the standard convolution layer with our proposed Rubik's cube convolution layer. the case of the denoising and guided super-resolution tasks, our operator not only improves the performance of traditional methods like DrCNN and PanNet, but also helps push the performance of state-of-the-ar methods like Restormer and INNFormer. Since Restormer has achieved the upper bound of the performance with a large model capacity, there are no significant performance gains for Restormer. Examining the results for SID/DRBN in Table [ our Rubik's cube convolution achieves gains of 399dB/0.58dB over the original bascline on the LOL 43 dataset, while also improving general- izability on the Huawei [44] dataset with gains of 0.2dB/O.2dB. It also performs better than the "Conv Ix1" variant, with gains 0.52dB/0.5 on the LOL dataset. These experiments demonstrate the potential of our Rubik's 'ss cube convolution in improving the performance of baseline models with minimal parameter overhead.
Due to the limited space, we only present the visual results of the low-light image enhancement task in 3 can clearly demonstrate that incorporating the RubikConv with the original baseline achieves more visually pleasing results. Specifically, in Figure 5 the model with the RubikConv achieves better lightness enhancement and color consistency with the corresponding normal-light images. More results can be found in the supplementary material.
Qualitative Comparison.
7

---

Imput
(b) Original
(c) Conv1x1
(c) RubikCorn
Figure Visual comparison of DRBN
on the LOI. dataset.
(a) Baseline
pb)p=
p 2
(d) =
@pp=
(p 5
Figure 6: The effective receptive field 42 of the DrCNN convolution and p indicates the shifted pixels in the RubikConv. The results show that the effective receptive field of the network progressively expands the number of shifted pixels increases.
33
with the proposed Rubik's cube
Expanding Receptive Field. The vanilla convolution layer enables first-order channel interaction through linear channel-wise weighting, resulting in consistent receptive field across the channel dimension. In contrast, our RubikConv generates hierarchical receptive field through shifting and high-order channel-wise interactions. An interesting characteristic of RubikConv that receptive field expands as the number of shifted pixels increases. Figure [6] presents the effective receptive field [22 of the DnCNN [3) baseline, in comparison with our RubikConv across various pixel shift values. As can be observed, the baseline's effective receptive field approximates a square, while the RubikConv expands it in the left, right, up, and down directions. Moreover, as the number of shifted pixels increases, the overall size of the effective receptive field grows correspondingly.
4.4 Ablation Studies
To verify the robustness of shifting displacement and the ratio of identity, perform the comparison over the following configurations:
1) RubikConv-p: replacing the standard convolution layer with the designed RubikConv where the number of the shifted pixel is "p":
2) RubikConv-r: substituting the standard convolution layer with our designed RubikConv, wherein the proportion of the unchanged identity component is denoted by "r".
We conduct ablation studies on the low-light image enhancement task with the SID 48 network and image denoising task with the DrCNN 33 network, and all the experiments with different configurations follow the same optimization strategy.
Number of the Shifted Pixel. To ascertain the influence of the number of shifted pixels in our pro- posed RubikConv, conduct series of experiments using the configuration denoted 'RubikConv- p'. As observed from Table and Figure 7 the performance remains consistent as the number of shifted pixels increases, up to a value of 3. Beyond this point, any further increase in p leads to decline in performance, potentially due to the increased complexity in reconstructing the structure. Therefore, in all the experiments presented in this paper, set p to as the default value for the number shifted pixels.
Ratio of the Shifted Channel. We investigate the impact of the ratio of the shifted channel following the definition in "RubikConv-r". Experimental results from both the SID dataset (as shown in Table and the DnCNN network (as depicted in Figure 7 suggest that optimal performance is reached when

---

Table 5: Ablation studies of the low-light image enhancement network SID 48 on the LOL training set and tested on the LOL 43
The model is trained
testing set and Huawei
RubikConv-p
Dataset
Metric
Original
(default)
2
3
5
PSNR
20.1
20.4972
20.5016
20.4858
20.4762
20.4506
LOL
SSIM
0.7895
0.7979
0.7981
0.7976
0.7962
0.7954
PSNR
16.5874
16.7867
16.7829
16.7913
16.7309
16.7288
Huawei
SSIM
0.5925
0.5941
0.5940
0.5940
0.5935
0.5934
RubikConv-r
Dataset
Metric
Original
1/4
1/3
1/2 (detault)
2/3
3/4
PSNR
20.1062
20.4892
20.4860
20.4972
20.4992
20.4795
LOL
SSIM
0.7895
0.7971
0.7969
0.7979
0.7961
0.0.7964
PSNR
16.5874
16.7550
16.7548
16.7867
16.7782
16.7824
Huawei
SSIM
0.5925
0.5940
0.5937
0.5941
0.5937
0.5939
Ablation Studies of Rubil-CoNN-p
Ablation Stulles of Rubil&CorN-r
37.30
37.95
018886
(12ER0)
0.8811
0,560
0,8778
(1900
37.53
37.30
0,8862
0,8886
02886
05900
37.35
37.40
0.8054
37.4123
37741138
37/4009
37,3919
37,357
0.896
37.40
37:35
37.390
37/0016
37,4123
37,4106
37/106
0896
37.30
01811
37.30
0/91
37.25
0,822
37,20
37.1902
37.20
37.15
(1150
37.15
(
Original
3
Original
-
PSNR
SSIM
-
PSPPS
SSIM
Figure Ablation studies of the image denoising network DrCNN on the SIDD dataset.
the ratio is set either 1/2 or 2/3. Further increases in the ratio subsequently result in decline in performance. We have set the default ratio to 1/2 for all experiments conducted in this study.
Limitations
We will validate the effectiveness of the proposed Rubik's cube convolution on broader low-level tasks, such image de-raining and de-hazing. As a generic operator, diverse comprehensive architectures integrated with our proposed operator should explored. The primary aim of our research beyond designing universal operator for improving the performance of existing networks. We hope offer an alternative tool for high-order channel- modeling. Consequently, our ongoing efforts will focus showcasing its effectiveness across broader computer vision tasks.
6 Conclusion
In this paper, we study the modeling of relationships in the channel dimension of image restoration tasks, and propose a Rubik's s cube convolution operator. constructs the high-order channel-win interactions and creates Rubik's cube-like hierarchical receptive field in an efficient and effective manner. Note that it can be seamlessly incorporated into existing image restoration networks as drop-in replacement for the standard convolution unit with fewer parameters. Extensive experiments across four distinct image restoration and enhancement tasks demonstrate the performance gains achieved by replacing designed operator with the standard convolution uint.
Broader Impact
Image restoration technology holds significant value across diverse fields, including remote sensing, medicine, astronomy, and consumer imaging equipment. Our proposed Rubik's 's cube convolution has the potential to enhance restoration algorithms by constructing high-order channel -wine relationships and producing hierarchical receptive field along the channel dimension. This could lead to clearer, more detailed images that improve decision decision-making capabilities in these fields. While we anticipate no negative societal consequences from our work, future research and applications may reveal unforeseen impacts. Thus, the responsible and ongoing evaluation of such technology is essential.

---

Acknowledgements
This study is supported under the RIE2020 Industry Alignment Fund- - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
10

---

References
[1] Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3147-3155, 2017.
[2] Max Ehrlich and Larry S Davis. Deep residual learning in the JPEG transform domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3484-3493, 2019.
[3] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence. 43(7):2440-244 2020.
[4] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. MemNet: persistent memory network for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4539-4547, 2017.
[5] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the HEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14821-14831, 2021.
[6] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. HINet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 182-192, 2021.
[7] Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6002-6012, 2022.
[8] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang. Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1833-1844, 2021.
[9] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang
Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17683-17693, 2022.
[10] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728-5739, 2022.
[11] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xin Yuan, et al. Cross aggregation trans- former for image restoration. Advances in Neural Information Processing Systems, 35:25478- 25490. 2022.
[12] Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. arXiv preprint 110042221.001427, 2022.
[13] Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 11030-1 1039, 2020.
[14] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. Advances Neural Information Processing Systems, 29, 2016.
[15] Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Con- vbert: Improving bert with basn- dynamic convolution. Advances Neural Information Processing Systems, 33: 33:12837-12848 2020.

---

[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[17] Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Horeet: Efficient high-order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 35: 10353-100666, 2022.
[18] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189 2018.
[19] Zichun Wang, Ying Fu, Ji Liu, and Yulun Zhang. Lg-bpn: Local and global blind-patch network for self-supervised real-world denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18156-18165, June 2023.
[20] Junyi Li, Zhilu Zhang, Xiaoyu Liu, Chaoyu Feng, Xiaotao Wang, Lei Lei, and Wangmeng Zuo. Spatially adaptive self-supervised learning for real-world image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9914-9924, June 2023.
[21] Qi Zhu, Man Zhou, Naishan Zheng, Chongyi Li, Jie Huang, and Feng Zhao. Exploring temporal frequency spectrum in deep video deblurring. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 12428-12437 October 2023.
[22] Lingshun Kong, Jiangxin Dong, Jianjun Ge, Mingqiang Li, and Jinshan Pan. Efficient frequency domain-based transformers for high-quality image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5886-5895, June 2023.
[23] Naishan Zheng, Jie Huang, Feng Zhao, Xueyang Fu, and Feng Wu. Unsupervised underexposed image enhancement via self-illuminated and perceptual guidance. IEEE Transactions on Multimedia, 2022.
[24] Naishan Zheng, Jie Huang, Qi Zhu, Man Zhou, Feng Zhao, and Zheng-Jun Zha. Enhancement by your aesthetic: An intelligible unsupervised personalized enhancer for low-light images. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6521-6529 2022.
[25] Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, and Feng Zhao. Empowering low-light image enhancer through customized learnable priors. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12559- 12569, October 2023.
[26] Naishan Zheng, Jie Huang, Man Zhou, Zizheng Yang, Qi Zhu, and Feng Zhao. Learning semantic degradation-awan guidance for pecognition-driver unsupervised low-light image enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3678 36866, 2023.
[27] Jie Huang, Feng Zhao, Man Zhou, Jie Xiao, Naishan Zheng, Kaiwen Zheng, and Zhiwei Xiong. Learning sample relationship for exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9904-990999911 9913, June 2023.
[28] Wenqi Ren, Xiaochun Cao, Jinshan Pan, Xiaojie Guo, Wangmeng Zuo, and Ming- Hsuan Yang. Image deblurring via enhanced low-rank prior. IEEE Transactions on Image Processing, 25(7):3426-347,, 2016.
[29] S Derin Babacan, Rafael Molina, and Aggelos K Katsaggelos. Variational bayesian blind deconvolution using a total variation prior. IEEE Transactions on Image Processing, 18(1):12- 26, 2008
[30] Chong-Yi Li, Ji-Chang Guo, Run-Min Cong, Yen-Wei Pang, and Bo Wang. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Transactions on Image Processing, 25(12):5664-5677, 2016.
12

---

[31] Xueyang Fu, Delu Zeng, Yue Huang, Xinghao Ding, and Xiao-Ping Zhang. A variational framework for single low light image enhancement using bright channel prior. In IEEE Global Conference on Signal and Information Processing, pages 1085-1088. IEEE, 2013.
[32] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In Proceedings of the European Conference on Computer Vision, pages 184-199. Springer, 2014.
[33] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trunsactions on Image Processing, 26(7):3142-3155 2017.
[34] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In International Conference on Learning Representations, 2019.
[35] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via a holistic attention network. In Proceedings of the European Conference Computer Vision, pages 191-207. Springer, 2020.
[36] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision, pages 286-301, 2018.
[37] Xin Li, Xin Jin, Jianxin Lin, Sen Liu, Yaojun Wu, Tao Yu, Wei Zhou, and Zhibo Chen. Learning disentangled feature representation for hybrid-distorted image restoration. In Proceedings of the European Conference Computer Vision, pages 313-329 Springer, 2020.
[38] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-eecitation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7132-7141, 2018.
[39] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 510-519, 2019.
[40] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. ECA- Net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11544-11542, 2020.
[41] Sanghyun Woo, Jongchan Park, Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision, pages 3- 2018
[42] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in Neural Information Processing Systems, 29, 2016.
[43] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep Retinex decomposition for low-light enhancement. arXiv preprint arXiv: /00.:/888..00 005.04560, 2018
[44] Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2rnet: Low-light image enhancement via real-low to real-normal network. arXiv preprint arXiv:2/06. 14501, 2021.
[45] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 3883-3891, 2017.
[46] Ziyi Shen, Wenguan Wang. Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 5572-5581, 2019.
13

---

[47] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In Proceedings of the European conference on computer vision, pages 184-201. Springer, 2020.
[48] Lijun Zhang, Xiao Liu, Erik Learned-Miller, and Hui Guan. Sid-nism: A self-supervised low-light image enhancement framework. arXiv preprint arXiv:2012.08777, 2020
[49] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to percep- tual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3063-3072, 2020.
[50] Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17399-17410, 17410, 2022.
[51] Abdelrahman Abdelhamed, Stephen Lin, and Michael Brown. high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1692-1700, 2018.
[52] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1586-1599 1595, 2017.
[53] man zhou, Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng. and Chongyi Li. Deep Fourier up-sampling, In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[54] Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qi Xie, and Xiangyong Cao. Memory- augmented deep unfolding network for guided image super-resolution. International Journal of Computer Vision, 131(1):215-442, 2023.
[55] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. PanNet: deep network architecture for saa-shapenning. In Proceedings of the IEEE International Conference Computer Vision, pages 5449-5457. 2017.
[56] Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual information- driven pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1798-1808 1808, 2022.
[57] Man Zhou, Jie Huang, Yanchi Fang. Xueyang Fu, and Aiping Liu. Pan-sharpening with customized transformer and invertible neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3553-3561, 2022.
[58] Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu, Xian Wei, and Feng Zhao Spatial-frequency domain information integration for pan-sharpening. In Proceedings of the European conference on computer vision, pages 274-291. Springer, 2022.
[59] Luciano Alparone, Lucien Wald, Jocelyn Chanussot, Claire Thomas, Paolo Gamba, and Lori Mann Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10):3012330 2007.
[60] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In JPL. Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume I: AVIRIS Workshop, 1992.
14


Reviews:
[{"review_id": "pVpSRPZ8DZ", "reviewer": "Reviewer_CEjA", "summary": "This paper proposes the Rubik’s Cube, which can replace the standard convolution layer in the traditional paradigm. With shift operation and high-order channel Interaction, the Rubik’s Cube can generate a hierarchical receptive field and activate the potential of channel interactions. Experiment results show that the Rubik’s Cube enhances performance across a variety of low-level vision tasks.", "questions": "Please see the weaknesses.", "limitations": "The authors should validate the effectiveness of the proposed Rubik’s cube convolution on broader low-level tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This Rubik’s Cube proposed in this paper can be widely applied in the framework where the convolution layers are used. If this operation can prove its effectiveness in multiple tasks, it could make a lot of sense.\n2. The Rubik's Cube is easy to implement and requires fewer parameters than the original convolutional layer. It's simple yet effective and efficient.", "weaknesses": "1. The experiment results on GaoFen2 make me confused since in terms of ERGAS, RubikConv has made a very big improvement (from 0.9576 to 0.5486). This increase does not fit well with other experimental results.\n2. The experiments of many tasks in this paper do not adopt the best methods at present. Many of the experiments were done using methods that were used two or three years ago. In order to make the experimental results more reliable, the paper should provide a comparison with the best method as much as possible.\n3. Since This Rubik's Cube splits the original convolutional layer operation into multiple convolutional layer operations, in addition to the number of parameters, it is better to provide inference time to better prove the efficiency improvement of This Rubik's Cube.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Results on Gaofen2.**\n\nThanks for your reminder. It is a typo and we will revise it. The performance of MutNet on GaoFen2 is shown in Table 1.\n\nTable 1: Quantitative comparisons of guided image super-resolution\n| Model  | Config    | PSNR    | SSIM   | SAM    | EGAS    |   |   |   |   |\n|--------|-----------|---------|--------|--------|---------|---|---|---|---|\n|        | Original  | 47.1699 | 0.9569 | 0.0192 | 0.5626  |   |   |   |   |\n| MutNet | Conv1x1   | 47.1668 | 0.9563 | 0.0185 | 0.5584  |   |   |   |   |\n|        | RubikConv | 47.3274 | 0.9885 | 0.0103 | 0.5486  |   |   |   |   |\n\n**2. Comparison with the best algorithms.**\n\nIn the manuscript, we conducted experiments on image denoising and de-blurring tasks using Restormer, which achieved performance that closely approximated the best results on both restoration tasks. In addition, we made an effort to integrate the proposed RubikConv into leading algorithms, such as SNR [1] for low-light image enhancement, FFTformer [2] for image de-blurring, and UAPN [3] for guided image resolution. The quantitative results presented below further demonstrate the effectiveness of our method.\n\n| Model | Config    | LOL     |        | Huawei  |         |   |   |   |   |\n|-------|-----------|---------|--------|---------|---------|---|---|---|---|\n|       |           | PSNR    | SSIM   | PSNR    | SSIM    |   |   |   |   |\n|       | Original  | 24.5276 | 0.8407 | 21.4308 | 0.7065  |   |   |   |   |\n| SNR   | Conv1x1   | 24.4360 | 0.8273 | 21.2811 | 0.6906  |   |   |   |   |\n|       | RubikConv | 24.6507 | 0.8426 | 21.5094 | 0.7113  |   |   |   |   |\n\nTable 3: Quantitative comparison of image de-blurring. The model is only trained on the GoPro training set and directly tested on the GoPro testing set, HIDE, and RealBlur datasets.\n| Model     | Config    | GoPro   |        | HIDE    |        | RealBlur-J |         | RealBlur-R  |         |\n|-----------|-----------|---------|--------|---------|--------|------------|---------|-------------|---------|\n|           |           | PSNR    | SSIM   | PSNR    | SSIM   | PSNR       | SSIM    | PSNR        | SSIM    |\n|           | Original  | 34.0694 | 0.9527 | 31.2796 | 0.9476 | 29.5407    | 0.8860  | 36.8165     | 0.9607  |\n| FFTformer | Conv1x1   | 33.6308 | 0.9455 | 30.8762 | 0.9403 | 28.8752    | 0.8789  | 36.1029     | 0.9352  |\n|           | RubikConv | 34.0867 | 0.9533 | 31.2969 | 0.9480 | 29.5549    | .0.8863 | 36.8353     | 0.9612  \n\nTable 4: Quantitative comparison of guided image super-resolution.\n| Model | Config    | WroldView-II |        |        |        | GaoFen2  |        |        |         |\n|-------|-----------|--------------|--------|--------|--------|----------|--------|--------|---------|\n|       |           | PSNR         | SSIM   | SAM↓   | ERGAS↓ | PSNR     | SSIM   | SAM↓   | ERGAS↓  |\n|       | Original  | 41.7156      | 0.9657 | 0.0227 | 0.9506 | 47.4635  | 0.9895 | 0.0100 | 0.5382  |\n| UAPN  | Conv1x1   | 41.6826      | 0.9615 | 0.0231 | 0.9557 | 47.4091  | 0.9890 | 0.0104 | 0.5392  |\n|       | RubikConv | 41.7675      | 0.9660 | 0.0220 | 0.9454 | 47.5260  | 0.9898 | 0.0097 | 0.5380  |\n\n[1] Xu X, Wang R, Fu C W, et al. SNR-aware Low-Light Image Enhancement, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.\n\n[2] Kong L, Dong J, Ge J, et al. Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.\n\n[3] Zheng K, Huang J, Zhou M, et al. Deep Adaptive Pansharpening via Uncertainty-aware Image Fusion, IEEE Transactions on Geoscience and Remote Sensing, 2023.\n\n**3. Efficiency of the proposed RubikConv.**\n\nWe report the model size, FLOPs (an image with 400\\*600\\*3 pixels), and average running time on the LOL test set (including 15 400\\*600\\*3 images) in Table 5. The running time is measured on a workstation with an NVIDIA GTX 3090 GPU. We only replace two standard convolution layers in the DRBN baseline with the proposed RubikConv, thus, the extra running time introduced by the RubikConv is negligible. Since the RubikConv only requires convolution with a 1x1 kernel, the FLOPs and parameters are fewer than the baseline while DRBN -RubikConv achieves a 0.58db performance improvement. \n\nTable 5: The quantitative results, FLOPs, and average running time of the DRBN baseline on the LOL test set.\n| Model | Config    | PSNR    | SSIM   | Flops (G) | Running time (s) |   |   |   |   |\n|-------|-----------|---------|--------|-----------|------------------|---|---|---|---|\n|       | Original  | 19.7931 | 0.8361 | 39.037    | 0.256            |   |   |   |   |\n| DRBN  | Conv1x1   | 19.8648 | 0.8340 | 38.445    | 0.255            |   |   |   |   |\n|       | RubikConv | 20.3769 | 0.8400 | 38.563    | 0.263            |   |   |   |   |"}, {"review_id": "tMluf0YgyL", "reviewer": "Reviewer_xjVk", "summary": "This paper proposes Rubik’s cube convolution operator, which is very simple and efficient, especially requiring zero FLOPS / parameters. This novel component improves the several low-level vision networks, enabling the high-order channel interaction and enlarging receptive field of the standard convolutions. Simply dividing feature maps into some groups and performing shifting operations makes the existing convolutional layers capture high-order channel interaction beyond first-order interaction. The experimental results show that multiple low-level vision tasks are improved by the proposed Rubik’s cube. However, the limitation could be handled by the authors that the performance gains and the reduced number of parameters are marginal.", "questions": "How about applying Rubik’s cube convolution to QKV projection before self-attention of Transformers? As the authors know, almost all image restoration fields are dominated by Transformer-based methods. And self-attention is the most core component of Transformers. However, this work seems to apply their novel component to only FFN of Transformers. So, the reviewer guesses how much difference could be made by replacement to Rubik’s cube convolution for elements that directly influence self-attention operation (e.g., QKV projection).", "limitations": "See weakness and question.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Rubik’s cube convolution does not require the number of operations and parameters. It can also replace any standard CNN architectures. The effective receptive field of the proposed component is successfully enlarged when compared to vanilla CNN, as shown in Fig. 3. The main experimental results, such as Tabs. 1,2,3,4, can show superiority of the proposed component over trivial 1x1 convolution replacement. As a result, the performances of various image restoration networks can be enhanced (but the marginal gains will be mentioned in “weakness”). Moreover, the ablation studies (Tab. 5) demonstrates that the hyper-parameters for Rubik’s cube convolution have been carefully decided. This paper is well-presented and easy to follow.", "weaknesses": "1. The performance gains of Rubik’s cube convolution are somewhat marginal. Especially, the low-light enhancement and deblurring could be further improved. The authors are suggested to consider more robust components to different tasks. Or if the improvement differences from task-by-task are explained and the insight for better variants of Rubik’s cube convolution resulting from these explanations can be drawn, the reviewer thinks this paper can be compelling.\n2. Furthermore, the number of reduced parameters led by the novel architecture is also marginal. So, (if possible) the reviewer requires an additional experiment that highly (e.g., less than 60% of “Original”) reduces the number of parameters of existing networks by changing channels or depths and replaces existing CNN or FFN with Rubik’s cube convolution. If reduced “Original” models with Rubik’s cube convolution are still better than “Original” models while requiring a small number of parameters, the robustness of this work can be reinforced.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. About the improvement differences and the variants of RubikConv.**\n\nThanks for the suggestion. Relationship modeling in the channel dimension has been proven effective in the literature for tasks like low-light image enhancement and image de-blurring. Previous works, such as the bright channel prior [1, 2, 3] for low-light image enhancement and the dark channel prior [4] and extreme channel [5] prior for image de-blurring, have explored the channel dimension and made significant progress. In this paper, our intention is to demonstrate the simplicity of RubikConv in achieving high-order channel interaction compared to these designs and its applicability as a generic formulation across various tasks. The main goal is not to achieve the best results within each task with dedicated components.\n\nNevertheless, to demonstrate that RubikConv can work well with other components, we show an experiment on low-light image enhancement. In particular, we customized the exposure-invariant feature extraction of SID by adding an instance normalization operation in the first branch, while keeping the remaining operations unchanged. The instance normalization maps different exposure features to the exposure-invariant feature space without introducing extra computational costs. This variant is named RubikConv-IN. The results in Table 1 show that it can further improve the PSNR/SSIM from 20.4972/0.7979 to 20.6708/0.8079 on the LOL dataset.\n\nTable 1: Quantitative comparisons of variant RubikConv on low-light image enhancement.\n| Model | Config       | LOL     |        | Huawei  |         |   |   |   |   |\n|-------|--------------|---------|--------|---------|---------|---|---|---|---|\n|       |              | PSNR    | SSIM   | PSNR    | SSIM    |   |   |   |   |\n|       | Original     | 20.1062 | 0.7895 | 20.1742 | 0.6659  |   |   |   |   |\n| SID   | RubikConv    | 20.4972 | 0.7979 | 20.2044 | 0.6655  |   |   |   |   |\n|       | RubikConv-IN | 20.6708 | 0.8079 | 20.3251 | 0.6702  |   |   |   |   |\n\n[1] Tao L, Zhu C, Song J, et al. Low-light image enhancement using CNN and bright channel prior, 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017.\n\n[2] Lee H, Sohn K, Min D. Unsupervised low-light image enhancement using bright channel prior, IEEE Signal Processing Letters, 2020, 27: 251-255.\n\n[3] Zhao Z, Xiong B, Wang L, et al. RetinexDIP: A unified deep framework for low-light image enhancement, IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(3): 1076-1088.\n\n[4] Pan J, Sun D, Pfister H, et al. Blind image deblurring using dark channel prior, Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1628-1636.\n\n[5] Yan Y, Ren W, Guo Y, et al. Image deblurring via extreme channels prior, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 4003-4011.\n\n**2. About the robustness of RubikConv in networks with fewer parameters.**\n\nThanks for your suggestion. We conduct the experiments to further validate the robustness of the proposed RubikConv on the low-light image enhancement task. Firstly, we implement the baseline with fewer parameters (about 50% of baseline) by reducing channels and depths, named “Original-S”. Then, we replace the standard convolution layer with the proposed RubikConv, named “S-RubikConv”. The quantitative performance on LOL [33] and Huawei is shown in Table 2. Although the PSNR/SSIM of “Original-S” dropped from 19.7931/0. 8361 to 19.2439/0. 8175, the performance of “S-RubikConv” surpassed the “Original” and achieved 19.8146/0.8365 on the LOL dataset.\n\nTable 2: Quantitative performance of DRBN with fewer parameters on low-light image \n| Model | Config      | LOL     |        | Huawei  |        |          |   |   |   |\n|-------|-------------|---------|--------|---------|--------|----------|---|---|---|\n|       |             | PSNR    | SSIM   | PSNR    | SSIM   | #Params  |   |   |   |\n|       | Original    | 19.7931 | 0.8361 | 20.1549 | 0.6851 | 0.55M    |   |   |   |\n| DRBN  | Original-S  | 19.2439 | 0.8175 | 19.9054 | 0.6758 | 0.21M    |   |   |   |\n|       | S-RubikConv | 19.8146 | 0.8365 | 20.1643 | 0.6857 | 0.19M    |   |   |   |\n\n**3. Replacing RubikConv with the QKV-projection of ViT.**\n\nWe attempted to replace the convolution layer in the QKV projection with the proposed RubikConv (named RubikConv-QKV), however, the performance improvement is marginal. We conjecture that the self-attention operation itself realizes the global receptive field, thus the proposed RubikConv with a Hierarchical Receptive Field will not further improve the performance. Thus, we conducted an experiment by replacing the FFN of Restormer with RubikConv. Due to the limited time, it is only performed on the image de-blurring task.\n\nTable 1: Quantitative comparison of replacing the QKV projection in Restormer with the RubikConv on image de-blurring. The model is only trained on the GoPro training set and directly tested on the GoPro testing set, HIDE, and RealBlur datasets.\n| Model     | Config        | GoPro   |        | HIDE    |        | RealBlur-J |        | RealBlur-R  |         |\n|-----------|---------------|---------|--------|---------|--------|------------|--------|-------------|---------|\n|           |               | PSNR    | SSIM   | PSNR    | SSIM   | PSNR       | SSIM   | PSNR        | SSIM    |\n|           | Original      | 32.9117 | 0.9603 | 30.1568 | 0.9405 | 28.9636    | 0.8792 | 36.2017     | 0.9572  |\n| Restormer | RubikConv-QKV | 32.9134 | 0.9603 | 30.1706 | 0.9406 | 28.9641    | 0.8790 | 36.2029     | 0.9572  |\n|           | RubikConv     | 32.9305 | 0.9608 | 30.1977 | 0.9411 | 28.9698    | 0.8796 | 36.2136     | 0.9575  |\n"}, {"review_id": "QHkcpxGZqU", "reviewer": "Reviewer_2pSr", "summary": "This paper proposes the Rubik’s cube convolution operator to model the high-order channel-wise interactions. The Rubik’s cube convolution applies the spatial-shifting mechanism across channel-wise groups, which is the zero-FLOP and zero-parameter. Moreover, only the point-wise convolution and dot-product are applied in Rubik’s cube convolution. The results on image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring show the effectiveness of the proposed Rubik’s cube convolution.", "questions": "1. The proposed Rubik’s cube convolution structure is similar to the g^nConv proposed in Hornet [17]. Clarify the difference between the Rubik’s cube convolution and g^nConv.\n2. In Figure 6 and Line 230, the receptive field of the network progressively expands as the number of shifted pixels increases. However, the performance of RubikConv-p is best when the p=1. Please give some explanation.\n3. The authors only provide the visual comparison of the low-light image enhancement task and image denoising task. More visual results on guided image super-resolution and image de-blurring should be provided. ", "limitations": "The authors have discussed the limitations and potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea of Rubik’s cube convolution is reasonable. This is a promising step to complement research in modeling high-order channel dimension information.\n2. The design of Rubik’s cube convolution is simple yet efficient and easy to follow since it only includes shift, dot-product and point-wise convolution.\n3. The paper is well-writing. The authors provide extensive analysis of the proposed methods. For example, the illustration of a high-order hierarchically receptive field in Figures 1, 3, and 4, and the corresponding analysis are convincing.\n4. The main results and ablation study are extensive and demonstrate the superiority of the proposed method.\n5. In supplementary material, the authors provide more experiments on high-level and low-level tasks, further showing the effectiveness of the Rubik’s cube convolution.", "weaknesses": "1. The FLOPs of the Rubik’s cube convolution are not provided. Although the parameter of the Rubik’s cube convolution is small (Sec. 3.3), the FLOPs may be large since there are many dot-product in it.\n2. Although the spatial-shifting mechanism is zero-FLOP and zero-parameter, it is a high-latency operation. Therefore, the actual latency of the Rubik’s cube convolution may be high, hindering its application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Efficiency of the proposed RubikConv.**\n\nWe report the model size, FLOPs (an image with 400\\*600\\*3 pixels), and average running time on the LOL test set (including 15 400\\*600\\*3 images) in Table 5. The running time is measured on a workstation with an NVIDIA GTX 3090 GPU. We only replace two standard convolution layers in the DRBN baseline with the proposed RubikConv, thus, the extra running time introduced by the RubikConv is negligible. Since the RubikConv only requires convolution with a 1x1 kernel, the FLOPs and parameters are fewer than the baseline while DRBN -RubikConv achieves a 0.58db performance improvement.\n\nTable 5: The quantitative results, FLOPs, and average running time of the DRBN baseline on the LOL test set.\n| Model | Config    | PSNR    | SSIM   | Flops (G) | Running time (s) |   |   |   |   |\n|-------|-----------|---------|--------|-----------|------------------|---|---|---|---|\n|       | Original  | 19.7931 | 0.8361 | 39.037    | 0.256            |   |   |   |   |\n| DRBN  | Conv1x1   | 19.8648 | 0.8340 | 38.445    | 0.255            |   |   |   |   |\n|       | RubikConv | 20.3769 | 0.8400 | 38.563    | 0.263            |   |   |   |   |\n\n\n**2. Discussion with the g^nConv**\n\nFirst, the g^nConv only formulates the high-order spatial interaction at the same position, i.e., two-order interaction is built by the dot product between (i, j) in feature x1 and (i, j) in feature x2. In contrast, the proposed RubikConv conducts high-order interaction between the center point and the surrounding points in the channel interaction, i.e., high-order interaction is formulated by the dot product among (i, j) in feature x_c1 of x, (i-1, j) in feature x_c2 of x, (i+1, j) in feature x_c3 of x, (i, j-1) in feature x_c4 of x, (i, j+1) in feature x2 of x.\nSecond, the receptive field in g^nConv is constant in the channel dimension, while the proposed RubikConv embraces a Rubik’s cube-like hierarchical receptive field benefitting from the shifting operation.\n\n**3. Clarification on the performance of RubikConv-p.**\n\nWe clarify that p=2 is the best number of the shifted pixel in ablation studies (see Figure 7 in the manuscript). Then, we will give the explanation from two perspectives.\n\n(1) As illustrated in Figure 7 in the manuscript, with the larger shifted pixel (p > 2), the effective receptive field will focus more on the horizontal and vertical direction while less on the four corners (upper left, upper right, bottom left, and bottom right), which will lead to insufficient feature extraction for the corner pixels.\n\n(2) There is a trade-off between the insufficient feature extraction caused by the large shifted pixels and ensuring the completeness of the information.\nAlthough the first branch in the RubikConv is unchanged to preserve original information, it cannot compensate for the insufficient feature representation when the shift pixel is large.\nTherefore, the performance will drop when the shifted pixel is larger than 2.\nWe will present the explanation in the revised version.\n\n**4. Visualization.**\n\nThanks for your suggestion. We will provide more visual results in the revised version.\n"}]

Author rebuttal:

Meta-review: This paper is highly evaluated by all reviewers for its good novelty, and sufficient experiments. All review comments have been well addressed in the rebuttal. I thus suggest an acceptance.
Decision: Accept (poster)