SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1) Original: the baseline without any changes;
Table 3: Quantitative comparison of image denoising. The model is trained only on the SIDD [51] training set and directly tested on the SIDD [51] testing set and DND [52] dataset.
Model	Metric	DnCNN	MPRNet	Restormer
OriginalConv1x1	RubikConv	OriginalConv1x1	RubikConv	OriginalConv1x1	RubikConv
SIDD	PSNR	37.1992	37.2247	37.4123	39.7129	39.6220	39.7741	40.0124	39.9429	40.0853
SSIM	0.8954	0.8955	0.8986	0.9524	0.9502	0.9536	0.9587	0.9568	0.9593
DND	PSNR	38.3305	38.3946	38.5759	39.6276	39.5834	39.9043	40.0182	39.9885	40.0994
SSIM	0.9307	0.9296	0.9328	0.9531	0.9524	0.9558	0.9560	0.9537	0.9564
#Params	1.51M	1.36M	1.28M	15.74M	13.89M	13.33M	25.31M	25.21M	25.21M
Table 4: Quantitative comparisons of guided image super-resolution.
Model	Config	WorldView-II	GaoFen2	#Params
PSNR↑	SSIM↑	SAM↓	ERGAS↓	PSNR↑	SSIM↑	SAM↓	EGAS↓
PanNet	Original	40.8212	0.9630	0.0257	1.0561	42.1699	0.9569	0.0192	0.9565
Conv1x1	40.8015	0.9624	0.0261	1.0576	42.1150	0.9527	0.0197	0.9576
RubikConv	41.2692	0.9646	0.0248	1.0237	43.1764	0.9693	0.0176	0.8516
MutNet	Original	41.0692	0.9646	0.0248	1.0237	47.1699	0.9569	0.0192	0.9565
Conv1x1	40.9372	0.9624	0.02749	1.0072	47.1668	0.9563	0.0185	0.9576
RubikConv	41.5258	0.9692	0.0230	0.9719	47.3274	0.9885	0.0103	0.5486
INNFormer	Original	41.5363	0.9696	0.0229	0.9649	47.1108	0.9882	0.0105	0.5610
Conv1x1	41.5276	0.9696	0.0226	0.9451	47.1295	0.9882	0.0104	0.5611
RubikConv	41.5720	0.9699	0.0228	0.9649	47.2458	0.9886	0.0104	0.5609
2) RubikConv: replacing the standard convolution in the original model with our designed Rubik's cube convolution:
3) Conv1x1: a baseline that replaces the RubikConv in the setting of 2) with four convolution layers with  $1 \times 1$  kernel for a fair comparison with approximately the same number of trainable parameters as 2).
For a fair comparison, we ensure each competitive baseline and its variants are subjected to the same training configuration and optimization strategy.
Quantitative Comparison. We conduct performance comparisons using the configurations defined above. The quantitative results for low-light image enhancement, image de-blurring, image denoising, and guided image super-resolution are presented in Tables 1 through 4, with the best results highlighted in bold. We observe a performance boost across all datasets and tasks when replacing the standard convolution layer with our proposed Rubik's cube convolution layer. In the case of the denoising and guided super-resolution tasks, our operator not only improves the performance of traditional methods like DnCNN and PanNet, but also helps push the performance of state-of-the-art methods like Restormer and INNFormer. Since Restormer has achieved the upper bound of the performance with a large model capacity, there are no significant performance gains for Restormer. Examining the results for SID/DRBN in Table 1, our Rubik's cube convolution achieves gains of  $0.39\mathrm{dB} / 0.58\mathrm{dB}$  over the original baseline on the LOL [43] dataset, while also improving generalizability on the Huawei [44] dataset with gains of  $0.2\mathrm{dB} / 0.2\mathrm{dB}$ . It also performs better than the "Conv1x1" variant, with gains of  $0.52\mathrm{dB} / 0.51\mathrm{dB}$  on the LOL dataset. These experiments demonstrate the potential of our Rubik's cube convolution in improving the performance of baseline models with minimal parameter overhead.
Qualitative Comparison. Due to the limited space, we only present the visual results of the low-light image enhancement task in Figure 5. It can clearly demonstrate that incorporating the RubikConv with the original baseline achieves more visually pleasing results. Specifically, in Figure 5 the model with the RubikConv achieves better lightness enhancement and color consistency with the corresponding normal-light images. More results can be found in the supplementary material.
(c) Conv1x1
(d) RubikConv
Figure 5: Visual comparison of DRBN [49] on the LOL [43] dataset.
(a) Baseline
Figure 6: The effective receptive field [42] of the DnCNN [33] with the proposed Rubik's cube convolution and  $\mathfrak{p}$  indicates the shifted pixels in the RubikConv. The results show that the effective receptive field of the network progressively expands as the number of shifted pixels increases.
(b)  $\mathbf{p} = \mathbf{1}$
(c)  $\mathbf{p} = 2$
(d)  $\mathbf{p} = 3$
(e)  $\mathbf{p} = 4$
(f)  $\mathbf{p} = 5$
Expanding Receptive Field. The vanilla convolution layer enables first-order channel interaction through linear channel-wise weighting, resulting in a consistent receptive field across the channel dimension. In contrast, our RubikConv generates a hierarchical receptive field through shifting and high-order channel-wise interactions. An interesting characteristic of RubikConv is that its receptive field expands as the number of shifted pixels increases. Figure 6 presents the effective receptive field [42] of the DnCNN [33] baseline, in comparison with our RubikConv across various pixel shift values. As can be observed, the baseline's effective receptive field approximates a square, while the RubikConv expands it in the left, right, up, and down directions. Moreover, as the number of shifted pixels increases, the overall size of the effective receptive field grows correspondingly.
To verify the robustness of shifting displacement and the ratio of identity, we perform the comparison over the following configurations:
1) RubikConv-p: replacing the standard convolution layer with the designed RubikConv where the number of the shifted pixel is "p";
2) RubikConv-r: substituting the standard convolution layer with our designed RubikConv, wherein the proportion of the unchanged identity component is denoted by "r".
We conduct ablation studies on the low-light image enhancement task with the SID [48] network and image denoising task with the DnCNN [33] network, and all the experiments with different configurations follow the same optimization strategy.
Number of the Shifted Pixel. To ascertain the influence of the number of shifted pixels in our proposed RubikConv, we conduct a series of experiments using the configuration denoted as 'RubikConv'. As observed from Table 5 and Figure 7, the performance remains consistent as the number of shifted pixels increases, up to a value of 3. Beyond this point, any further increase in  $p$  leads to a decline in performance, potentially due to the increased complexity in reconstructing the structure. Therefore, in all the experiments presented in this paper, we set  $p$  to 1 as the default value for the number of shifted pixels.
Ratio of the Shifted Channel. We investigate the impact of the ratio of the shifted channel following the definition in "RubikConv-r". Experimental results from both the SID dataset (as shown in Table 5) and the DnCNN network (as depicted in Figure 7) suggest that optimal performance is reached when
Table 5: Ablation studies of the low-light image enhancement network SID [48]. The model is trained on the LOL [43] training set and tested on the LOL [43] testing set and Huawei [44].
Dataset	Metric	RubikConv-p
Original	1 (default)	2	3	4	5
LOL	PSNR	20.1062	20.4972	20.5016	20.4858	20.4762	20.4506
SSIM	0.7895	0.7979	0.7981	0.7976	0.7962	0.7954
Huawei	PSNR	16.5874	16.7867	16.7829	16.7913	16.7309	16.7288
SSIM	0.5925	0.5941	0.5940	0.5940	0.5935	0.5934
Dataset	Metric	RubikConv-r
Original	1/4	1/3	1/2 (default)	2/3	3/4
LOL	PSNR	20.1062	20.4892	20.4860	20.4972	20.4992	20.4795
SSIM	0.7895	0.7971	0.7969	0.7979	0.7961	0.0.7964
Huawei	PSNR	16.5874	16.7550	16.7548	16.7867	16.7782	16.7824
SSIM	0.5925	0.5940	0.5937	0.5941	0.5937	0.5939
Figure 7: Ablation studies of the image denoising network DnCNN [33] on the SIDD [51] dataset.
the ratio is set to either  $1/2$  or  $2/3$ . Further increases in the ratio r subsequently result in a decline in performance. We have set the default ratio r to  $1/2$  for all experiments conducted in this study.
We will validate the effectiveness of the proposed Rubik's cube convolution on broader low-level tasks, such as image de-raining and de-hazing. As a generic operator, diverse comprehensive architectures integrated with our proposed operator should be explored. The primary aim of our research is beyond designing a universal operator for improving the performance of existing networks. We hope to offer an alternative tool for high-order channel-wise modeling. Consequently, our ongoing efforts will focus on showcasing its effectiveness across broader computer vision tasks.
In this paper, we study the modeling of relationships in the channel dimension of image restoration tasks, and propose a Rubik's cube convolution operator. It constructs the high-order channel-wise interactions and creates a Rubik's cube-like hierarchical receptive field in an efficient and effective manner. Note that it can be seamlessly incorporated into existing image restoration networks as a drop-in replacement for the standard convolution unit with fewer parameters. Extensive experiments across four distinct image restoration and enhancement tasks demonstrate the performance gains achieved by replacing our designed operator with the standard convolution uint.
Image restoration technology holds significant value across diverse fields, including remote sensing, medicine, astronomy, and consumer imaging equipment. Our proposed Rubik's cube convolution has the potential to enhance restoration algorithms by constructing high-order channel-wise relationships and producing a hierarchical receptive field along the channel dimension. This could lead to clearer, more detailed images that improve decision-making capabilities in these fields. While we anticipate no negative societal consequences from our work, future research and applications may reveal unforeseen impacts. Thus, the responsible use and ongoing evaluation of such technology is essential.
This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).