Real-Time Image Demoir$\acute{e}$ing on Mobile Devices
Authors: Yuxin Zhang
Mingbao Lin
Xunchao Li
Han Liu
Guozhi Wang
Fei Chao
Ren Shuai
Yafei Wen
Xiaoxin Chen
Rongrong Ji
Venue: ICLR 2023 poster (2023)
Keywords: Image Demoireing
Network Acceleration

Abstract: Moir$\acute{e}$ patterns appear frequently when taking photos of digital screens, drastically degrading the image quality. Despite the advance of CNNs in image demoir$\acute{e}$ing, existing networks are with heavy design, causing massive computation burden for mobile devices. In this paper, we launch the first study on accelerating demoir$\acute{e}$ing networks and propose a dynamic demoir$\acute{e}$ing acceleration method (DDA) towards a real-time deployment on mobile devices. Our stimulus stems from a simple-yet-universal fact that moir${\'e}$ patterns often unbalancedly distribute across an image. Consequently, excessive computation is wasted upon non-moir$\acute{e}$ areas. Therefore, we reallocate computation costs in proportion to the complexity of image patches. In order to achieve this aim, we measure the complexity of an image patch by a novel moir$\acute{e}$ prior that considers both colorfulness and frequency information of moir$\acute{e}$ patterns. Then, we restore higher-complex image patches using larger networks and the lower-complex ones are assigned with smaller networks to relieve the computation burden. At last, we train all networks in a parameter-shared supernet paradigm to avoid additional parameter burden. Extensive experiments on several benchmarks demonstrate the efficacy of our DDA. In addition, the acceleration evaluated on the VIVO X80 Pro smartphone equipped with the chip of Snapdragon 8 Gen 1 also shows that our method can drastically reduce the inference time, leading to a real-time image demoir$\acute{e}$ing on mobile devices. Source codes and models are released at https://github.com/zyxxmu/DDA.


Published as a conference paper at ICLR 2023
REAL-TIME IMAGE DEMOIRÉING ON MOBILE DE- VICES
Yuxin Zhang', Mingbao Lin', Xunchao Li', Han Liu?, Guozhi Wang', Fei Chao',
Shuai Ren", Yafei Wen", Xiaoxin Chen', Rongrong Ji Ji4.34
MACC Lab, School of Informatics, Xiamen University. 2VIVO Al Lab, Instiitte of Artificial Intelligence, Xiamen University, "Pengcheng Lab
ABSTRACT
Moiré patterns appear frequently when taking photos of digital screens, drastically degrading the image quality. Despite the advance of CNNs in image demoiréing, existing networks are with heavy design, causing redundant computation bur- den for mobile devices. In this paper, we launch the first study on accelerat- ing demoiréing networks and propose a dynamic demoiréing acceleration method (DDA) towards a real-time deployment on mobile devices, Our stimulus stems from a unnive-at-univeerall fact that moiré patterns often unbalancedly distribute across an image. Consequently, excessive computation is wasted upon non-moiré areas. Therefore, we reallocate computation costs in proportion of image patches. In order to achieve this aim, we measure the complexity of an image patch by designing a novel moiré prior that considers both colorfulness and frequency information of moiré patterns. Then, we restore image patches with higher-complexity using larger networks and the ones with lower-complexity are assigned with smaller networks to relieve the computation burden. At last, we train all networks in a parameter-shared supernet paradigm to avoid additional parameter burden. Extensive experiments on several benchmarks demonstrate the efficacy of our proposed DDA. In addition, the acceleration evaluated on the VIVO X80 Pro smartphone equipped with a chip of Snapdragon 8 Gen 1 shows that our method can drastically reduce the inference time, leading to real-time image demoiréing on mobile devices. Source codes and models are released https: /ttt:///gtthb com/ Syxxmu/DDA.
the complexity
INTRODUCTION
Moiré patterns (Sun et al. 2018 Yang et al. 2017b) describe an artifact of images that in particular appears in television and digital photography. In contemporary society, using mobile phones to take screen pictures has become one of the most productive ways to record information quickly. Nevertheless, moiré patterns occur frequently from the interference between the color filter array (CFA) of camera and high-frequency repetitive signal. The resulting stripes of different colors and frequencies on the captured photos drastically degrade the visual quality. Therefore, developing demoiréing algorithms has received long-time attention in the research community, and yet remains unsolved in particular when running algorithms on mobile devices.
Primitive studies on image demoiréing resort to traditional machine learning techniques such as low-rank and sparse matrix decomposition (Liu et al. 2015) and bandpass filters (Yang et al. 2017a). The rising of convolutional neural networks (CNNs) has vastly boosted the efficacy of image demoiréing (He et al.] 2019; Zhheng et al.] 2020). However, the improved quantitative per- formance, such PSNR (Peak Signal-to-Noise Ratio), comes the increasing costs of energy and computation. For example, MBCNN (Zheng et al. (020) eats up 4.22T floating-poim operations (FLOPs) in order to restore a 1920x 1080 smartphone-taken moiré image. Given that the moiré patterns mostly emerge in mobile photography, such massive computations carry considerable infer- ence latency, preventing real-time demoiréing experience from the users. Such a handicap could
Work done during Yuxin Zhang's 's internship at VIVO Al Lab 1 *Corrsponning author: rrji @ xmm.edu.cn

---

Published as a conference paper at ICLR 2023
SAUURE OSC FOLLOW US DON'T MISS
Figure I: Images with moiré patterns. The blue boxes show a blue zoomed in area. Two phenomena be observed: (1) The moiré complexity varies significantly across different areas of image. (2) Moiré patterns are mainly characterized by frequency and colorfulness.
be more serious when it comes to video demoiréing. Therefore, it is of great need to bridge the technology gap between academia and industry.
To tackle the aforementioned issue, we initiate the first study on accelerating demoiréing networks towards a real-time deployment on mobile devices. Our motivation arises from empirical actuality in moiré images. As shown in Fig. Some areas are filled with intensive moiré stripes, some are much relieved while some are kept away from moiré pollution. It is natural to hand out computation these moiré centralized areas but less these di diluted areas. In the extreme case, it needless to cleanse uninfluenced areas. Unfortunately. existing methods (Sun al. 2018) Zheng al., (020)) have not distinguished the treatment the different areas in an image. They not only waste excessive computation on non-moiré areas but also bring about side effects, such as over- weiteneee image contents. Therefore, reallocating computation costs in compliance with the complexity of moiré area can be a potential solution to accomplish real-time image demoiréing on mobile devices.
image is often partially contaminated by the moiré patter..
Stimulated by the above analysis, opt split whole image into several sub-image patches. To measure the patch moiré complexity, we introduce a novel moiré prior. can be referred in Fig. moiré patterns are featured with either high frequency or rich color information. Thus, we define the moiré prior as the product of frequency and color information in a patch. In detail, we model the frequency information by Gaussian filter and the colorfulness metric is linear combination of the mean and standard deviation the pixel cloud in the RGB colour space (Hasler & Suesstrunk 2003]. Using this prior to measure the moiré complexity, each image patch is then processed by unique network with its computation costs in proportion to the moiré complexity. In this fashion, larger networks utilized restore moiré centralized areas ensure the recovery quality while smaller networks are leveraged to restore moiré diluted areas to relieve computation burden. Thus, we have better tradeoff between the image quality and resource requirements on mobile devices.
Nevertheless, multiple networks lead to more parameter burden, which also causes deployment pres- sure due to the short-supply memory on mobile devices. To mitigate this issue, leverage the supernet paradigm (Yang et al. 202T) to jointly train all networks in a parameter-shared manner. Concretely, we regard the vanilla demoiréing network as a supernet, and weight-shared subnets of different sizes are directly extracted from this supernet to process image patches of different demoiré complexity. During the training phase, each subnet dynamically trained using corresponding im- age patches with similar moiré complexity. Consequently, the overall running overhead can be effectively reduced without introduction of any additional parameters.
We have conducted extensive experiments for accelerating existing demoiréing networks the LCDMoiré (Yuan et al., 2019) and FHDMi (He et al.) 2020) benchmarks. The results show that our dynamic demoiréing acceleration method, termed DDA, achieves obvious FLOPs reduction with PSNR and SSIM increases. For instance, DDA reduces 45.2% FLOPs of the ofte-of--hhean demoiréing network MBCNN (Zheng al. 2020 with 0.35 dB PNSR increase. Furthermore, the
2

---

Published as a conference paper at ICLR 2023
Moiré prior
Moiré image
Clean image
Models with different computation burden
Sub-image patches with varying moiré complexity
Restored sub-image patches
Figure The framework of our proposed dynamic demoiréing acceleration method (DDA).
accelerated DMCNN (Sun et al. 2018) leads to real-time image demoiréing the VIVO X80 Pro smartphone, with a tiny latency of 48.0ms when processing an image 1920x 1080 resolution.
This work addresses the problem of demoiréing network deployment resource-limited mobile devices. The key contributions of this paper include: (1) A novel framework accelerating image demoiréing networks in dynamic manner. (2) An effective moiré prior to identify the demoiréing complexity of a given image patch. (3) Performance maintenance and apparent acceleration on modern smartphone devices.
2 RELATED WORK
2.1 IMAGE DEMOIRÉING
Image demoiréing aims at removing moiré patterns captured images. Early work mainly focuses on manually designed algorithms with the aid of low-rank sparse matrix decomposition (Yang ett al.) 2017a, Liu et al., 2015). With the explosion and popularity of deep learning, extensive demoiréing networks are proposed in recent years to achieve moiré removal in an end-to-end man- ner (Sun et al. 2018 He et al., 2019). As a pioneering work, (Sun et al., 2018) proposed multi- scale network structure to remove moiré patterns at different frequencies and scales. (He et al. (019)) dived into designing specific learning schemes to resolve the unique properties of moiré patterns in- cluding frequency distributions, edge information and appearance attributes. (Zheng et al. (020)) reformulated the image demoiréing problem as moiré texture removal and color restoration, and proposed MBCNN (Zheng et al.) 2020) which consists of learnable bandpass filter to learn the fre- quency prior and a two-step tone mapping mechanism restore color information. FHDe2Net (He) et al.) 2020) uses global branch to eradicate multi-scale moiré patterns and local branch to reserve fine details. (Liu al., (2000) further proposed WavleNet handle demoiréing in the wavelet do- main. The same dilemma for the aforementioned networks is their huge computation burden, which greatly prohibits the practical deployment on mobile devices.
2.2 DYNAMIC NETWORKS AND SUPERNETS
Dynamic networks adapt the network structures or parameters w.E. different inputs (Kong et al., 2021 Huang al. 2017] Bolukbasi et al.) 2017). Due to the advantages accuracy performance and computation efficiency. dynamic networks have received increasing research interest in recent years. A comprehensive overview of dynamic networks can found at (Han et al. (202T). Supernets are type of dynamic network that reserves weight-shared sub-networks of multiple sizes within only one network, and randomly samples these sub-networks for training (Chen et al. 2022 Yang et al. 2021 Yu & Huang 2019). According to the constraints of available resources, different sub- networks with varying widths and resolutions can be adaptively chosen during the testing phase without introducing additional parameter burden. Inspired by these studies, our proposed DDA involves the supernet paradigm by dynamically allocating image patches with different demoiréing complexity to their corresponding sub-networks.
3

---

Published as a conference paper at ICLR 2023
Moiré Complexity Score
Figure 3: The sorted sub-image patches w.r. moiré complexity scored by our proposed moiré prior.
3
METHODOLOGGY
2 manifests the general framework of our dynamic demoiréing acceleration method (DDA). A moiré image firstly split into several sub-image patches, which are then reorganized into different groups in conformity with the patch moiré complexity. This is achieved by novel moiré prior that considers both color and frequency information of moiré patterns. We detail it in Sec. For the purpose of real-time image demoiréing on mobile devices, we train multiple networks of different complexity to process patches in different groups. In the training and testing phases, image patches with higher-complexity
fed to larger networks and patches with lower-complexity are dealt with smaller networks. Sec. Sec.B.2 gives the implementations. Finally, as an alternative for training separate networks that result in more parameters, we regard each one subnet of the vanilla demoiréing network (supernet), leading to a weight-shared training paradigm as depicted in Sec. 3.3
3.1 MOIRÉ PRIOR
The complexity degree of moiré patterns can be determined by human, however, it is lavish and laborious to manually define the complexity for all patches in every moiré image. Many former works dynamic networks (Han al. Kong et al., (2021) train an additional module adapt the network w.r.f different inputs, which, however, brings unexpected parameters and computations for its compositions of several convolutional fully-connected layers. Given our plan of deploying demoiréing networks on mobile devices, such a solution is not feasible, or at least not optimal.
We propose novel moiré prior measure the moiré complexity of an image in fast manner. The motivation for this prior comes from an in-depth observation on moiré images. As can be inferred from Fig. 3 moiré patterns vary a lot in frequency and colorfulness. Customarily, a perceptible moiré pattern is highlighted by either high frequency or rich color information. Therefore, prior reflecting both image frequency and colorfulness can be an efficacious method to model the intensity of moiré patterns. Denoting moiré image as X, we first decompose it into sub-image patches as (rr) For a specific patch æ; we use the Gaussian high-pass filter (Dogra & Bhalla 2014) with standard deviation of 5 for the Gaussian distribution to extract the frequency information F(x). To measure the patch colorfulness, we consider a linear combination of the mean and standard deviation of the pixel cloud in the RGB colour space (Hasler & Suesstrunk 2003):
C(x) o xG) + o² (0.5(ER + xG) FB)
+0.3 - xa) + (0.5(( + +a) - TR),
where v²()) and a--) are the mean and standard deviation functions, TR. DG. FB denote the R, G. B color channels, respectively. Here 0.3 is parameter found by (Hasler & Suesstrunk 2003) through maximizing the correlation between the experimental data and the colorfulness metric. Re- fer to (Hasler & Suesstrunk 2003) for more principles of measuring image colorfulness. Therefore, our proposed moiré complexity score using frequency and colorfulness priors is finally defined as: (2)
M(x) C(x) =(F(x)),
where pa-) is the mean function. Fig. shows that M (x) can be reliable metric for evaluating the patch moiré complexity. Notice that, without building any extra network module, the operations of our moiré prior become highly cheap, bringing negligible computation burden.
3.2 DYNAMIC DEMOIRÉING ACCELERATION
In image demoiréing, moiré- image X is expected restore moiré-fres ground-truth in natural scenes. A traditional demoiréing process is formulated using CNNs as:
Y F)
(3)
4

---

Published as a conference paper at ICLR 2023
where F standards for the demoiréing network with its parameters denoted O. As can be seen from Eq. (3), existing methods restore all areas of a moiré image with the same network F, which wastes excessive computation since the moiré complexity varies significantly across different areas of an image as aforementioned in Sec. T This violates our goal of real-time image demoiréing on mobile devices. A natural way for demoiréing acceleration is to reallocate computation costs according to the complexity of moiré area.
To that effect, we reorganize the sub-image patches { of X in an ascending order of their moiré complexity score defined in Eq. (2). and then split the ordered patches into M groups denoted as {G1. G2 .... ). Each group G; contains these image patches, moiré complexity scores of which range from the top-((i- 1)- N 1+1) to (i- 1) smallest among all. Then, we construct M JM different demoiréing networks {Fi) Te with parameters of different sizes as {0r)== and process each image patch E G. using the i-th network F; as:
= 0,|2 E.))
(4)
In our setting, the complexity of Fi is smaller than that of Fi+1 such that smaller-complex image patches can be handled by networks with low computation costs, and vice versa. Eq. ( dynami- cally accelerates the derivation of Eq. 3 by assigning computation costs in line with the degree of moiré complexity. Meanwhile, the recovery quality is still ensured as moiré centralized areas are re- stored using larger networks. Finally. the moiré-free output of our dynamic demoiréing acceleration method (DDA) is obtained by concatenating patch outputs of all networks:
Y concat (Fi (I; 11 E F2(; 22/2 x E G2), FM (z;; Oas E Gu)), where concat() concatenates the output patches to construct a moiré-free full image.
(5)
3.3 SUPERNET TRAINING
Though the aforesaid procedure benefits reduction of the overall computation costs, the challenge arises in respect of parameter burden when deploying our demoiréing method on mobile devices featured with short-supply memories. For simple case, setting the largest network FM as the vanilla demoiréing network F, additional parameters of ((O)( are introduced in total.
((O,))
To solve this, in place of training networks {F.)MM in isolation, further propose to use the supernet paradigm Yangg et al. 202T to train and infer all networks a parameter- sharedd manner. a supernet and its parameters θ
In detail, the vanilla demoiréing network F is regarded partly shared by F,. Supposing the network width of F; is W;, we inherit the first W; proportion of convolution filter weights to the subnet F,, denoted as [[W.). Consequently, the network F, becomes a subnet of F. Therefore, our moiré-free output is finally reformulated as:
= concat (F (a; @[Wi]læ F(r:) [[W2]| E G2), G2), (WM))) E Gm).
(6)
As consequence, image demoiréing be effectively accelerated without any additional parameter burden, which finally reaches our target for a real-time deployment on mobile devices.
EXPERIMENT
4.1 EXPERIMENT SETUP
4.1.1 DATASETS
There are three main public datasets for image demoiréing. LCDMooré dataset (Yuan al. 2019) from the AIM19 image demoiréing challenge consists of 10,200 synthetically generated image pairs including 10,000 training images, 100 validation images and 100 testing images 1024x 1024
resolution. (2) FHDMi dataset (He et al., 2020) contains 9,981 image pairs for training and 2,019
for testing with 1920x 1080 resolution. (3) The TIP2018 dataset (Huang et al. 2017) consists real photographs constructed by photographing images with 400 resolution from ImageNet (Deng) al. 2009) displayed on computer screens. In this paper, we conduct experiments on the LCDMoire and FHDMi datasets. We do not consider the TIP2018 benchmark since the resolution is too small to meet our target for image demoiréing on mobile devices from which the captured images generally have extremely high resolution with 1920x 1080 or higher.
5

---

Published as a conference paper at ICLR 2023
Table 1: Ablation study for applying different width list configurations in supernet training.
FHDMi
AIM
W
PSNR
SSIM
Params
PSNR
SSIM
Params
[0.4, )
22.81
0.8124
M0.61M
41.68
0.9869
M0.6TM
(0.25, 0.5, 0.75)
23.07
0.8766
11.88M
41.43
0,9852
11.88M
{0.1, 0.5, 0.9}
21.03
0.7988
13.15M
40.21
0.9791
13.15M
4.1.2 EVALUATION PROTOCOLS
We adopt the widely-used metrics of PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structure Sim- ilarity) to conduct a quantitative comparison for demoiréing performance. We also report the color difference between restored images and the clean images using the CIE DeltaE 2000 (Sharma al. 2003) measurement, denoted as AE. The Float Points Operations (denoted as FLOPs) and network latency per image VIVO X80 Pro smartphone with the Snapdragon 8 Gen chip of networks reported as the accelerating evaluation.
4.1.3 BASELINES
We choose to accelerate DMCNN (Sun et al. 2018) and MBCNN (Zheng et al. 2020) to verify the efficacy of our DDA method. DMČNN is pioneering network for image demoiréing with a multi- scale structure. MBCNN is a state-of-the-art demoiréing network, which consists of a learnable bandpass filter to learn the frequency prior and two-step tone mapping mechanism restore color information. We report the results of baseline models as well as their compact versions by slimming the network width based on our re-implementation on the PyTorch framework (Paszke al. 2019
4.1.4 IMPLEMENT DETAILS
Our implementation of DDA is based the PyTorch framework (Paszke al., 2019), with the
group number M = 3, width list W = {0.25, 25, 0.5, 0.75} on FHDMi and W {0.4, 0.5, 0.6} on LCDMoiré. We split the original images of LCDMoiré and FHDMi datasets into sub-image patches of 512x512 and 640x540, respectively. Then, we classify the sub-image patches into multiple groups with different moiré complexity using our proposed moiré prior. We train the supernet using Adam (Kingma & Ba 2014) optimizer. The initial learning rate and batch size are set to le-4 and in all experiments. During training, we iteratively extract batch of image pairs within a specific class of moiré complexity, which are used to train the subnet of corresponding width extracted from the supernet. For DMCNN, we give 200 epochs for training with the learning rate divided by 10 at the 100-th epoch and 150-th epoch. For MBCNN, learning rate by half if the decrease in the validation loss is lower than 0.001 dB for four consecutive epochs and stop training once the learning rate becomes lower than 1e-6. All experiments are on NVIDIA Tesla V100 GPUs.
follow (Zheng et al. 2020) to reduce the
4.2 PERFORMANCE ANALYSIS
In this section, we perform detailed performance analysis on the different components of our DDA including the supernet training paradigm and moiré prior.
4.2.1 SUPERNET TRAINING
We first conduct experiments to investigate how the hyper- parameters in the supernet training influ- ence the performance of DDA, w.r.t, the width list configuration W. The experiments are conducted on the FHDMi (He et al. 2020) and LCDMoiré (Yuan et al. (219)) datasets using MBCNN. We can observe from Tab. that a dispersed configuration W = 0.25,, 0.5, 0.75) performs better on the FHDMi dataset, while compact configuration W (0.4, 0.5, 0.6} works better on the LCDMoiré dataset. To explain, LCDMoiré is a synthetic dataset, where the moiré patterns distribute more balanced compared with FHDMi that is captured using embedded cameras. Generally speaking, more discrete width list guarantees our purpose of dynamically removing moiré patches of different complexity in real scenarios.

---

Published as a conference paper at ICLR 2023
Table 2: Ablation study for dy-
Table 3: Ablation study for restoring image patches from the same group for the moiré prior. with different widths.
Table 4: Ablation study
namic acceleration with/without supernet training.
Prior
PSNR
Method
PSNR
Params
Network
G₁
G2
Gs
C 22.44
w supernet
23.62
1L88M
MBCNN O. x
24.03
22.21
21.29
F
22.18
w/o supernet
23.15
30.16M
MBCNN-0.5X
24.12
22.02
20.22
C+F
22.42
MBCNN-0.25x
23.99
21.81
19.69
Ours
23.62
Table 5: Quantitative results for demoiréing acceleration on the FHDMi dataset.
Method
PSNR
SSIM
AE
FLOPs
FLOPs +
Params
Params
Latency
DMCNN
21.69
0.7731
6.67
699.160
0.0%
1.43M
0.0%
69.4ms
DMCNN-0.75 x
21.56
0.7704
6.81
476.62G
31.4%
0.99M
29.9%
55.2ms
DMCNN-0.5 x
21.11
0.7691
7.14
314.82G
54.8%
0.68M
52.1%
47.9ms
DMCNN-0.25 x x
20.63
0.7655
7.98
208.73G
70.1%
0.48M
66.6%
41.8ms
DDA
21.86
0.7708
6.55
333.39G
52.3%
0.99M
29.9%
48. Oms
MBCNN
23.27
0.8201
5.38
4.22T
0.0%
14.21M
0.0%
259.8ms
MBCNN-0.75 x
22.51
0.8113
6.11
3.05T
27.3%
11.88M
16.4%
192.4ms
MBCNN-0.5x
22.12
0.8077
6.32
2.07T
47.5%
9.92M
30.2%
147.2 ms
MBCNN-0025
21.83
0.7991
6.54
1.28T
60.8%
8.36M
41.2%
119.7ms
DDA
23.62
0.8293
5.21
2.13T
45.2%
11.88M
16.4%
147.
Besides, Tab. 2 compares the performance for accelerating MBCNN on LCDMoiré between super- net training and respectively training each sub-network. It can be seen that supernet training does lead to performance degradation and drastically reduces the parameter burden compared with simultaneously keeping multiple sub-networks. The result well demonstrates the effectiveness of our DDA for practical deployment.
4.2.2 MOIRÉ PRIOR
We further analyze the performance of the proposed moiré prior. The experiments conducted on the FHDMi dataset using MBCNN. We use networks of different widths to infer different groups of pictures with different complexity classified by our proposed moiré prior. The results for Tab. lab.3]s show that all widths perform similarly for the easiest group, while larger width significantly outperforms smaller width for the group with highest moiré complexity. Such results demonstrate the effective- ness of our proposed moiré prior operator, and also validate our point that using large networks to restore patches with low moiré complexity wastes massive computation resources.
At last, we investigate three variants of our proposed moiré prior including (1) only using high- frequency information (denoted as F), (2) only using colorfulness information (denoted as C), (3) adding two information scores instead of multiplication in Eq. (2) (denoted as C+F), As shown in Tab. variants result worse performance, which well demonstrates the efficacy proposed moiré prior that considers both color and frequency properties of moiré patterns. It is worth mentioning that C+F implies domination of the colorfulness measurement due to the fact that scores given by C generally two orders of magnitude larger than those of F in observation. As a result, C+F leads to a similar performance to C. In contrast, by multiplying both scores, our prior offers reliable moiré complexity for given image patch.
our
4.3 QUANTITATIVE COMPARISON
Tab.5 and Tab. reeport the quantitative results of our DDA for accelerating DMCNN and MBCNN. On FHDMi, DDA surprisingly improves the PSNR of MBCNN by 0.35 dB even with FLOPs reduction of 45.2%. We attribute such results to that the original baseline assigns the same network to restore the areas with very few or no moiré patterns, which may damage the original details of the image. Consequently, the poor performance barricades the usage of demoiréing networks in practical deployment. In contrast, DDA leverages the smallest network to restore these non-moiré areas, leading to a better global demoiréing effect. Besides, we demonstrate the effectiveness of DDA by comparing MBCNN accelerated by with several state-of-the of-ee-of-the-art demoiréing networks including MDDM (Cheng al. 2019), MopNet (He al. 2019), FHDe? (He et al. (020) on FHDMi dataset. As can be seen from Tab. DDA can outperform other networks regarding both complexity reduction and demoiréing performance. For instance, DDA surpasses FHDDe Net by 0.69
7

---

Published as a conference paper at ICLR 2023
Table 6: Quantitative results for demoiréing acceleration on the LCDMoiré dataset.
Method
PSNR
SSIM
AE
FLOPPs
FLOPs +
Params
Params +
Latency
DMCNN
34.58
0.9612
1.76
353.1IG
0.0%
1,43M
0.0%
35.2ms
DMCNN-0.
33.41
0.9589
1.84
242.24G
31.4%
0.99M
29.9%
28.0ms
DMCNN-0..
32.99
0.9604
1.90
159.67G
54.8%
0.68M
52.1%
24.2ms
DMCNN-0.25x
31.75
0.9547
2.27
105.42G
70.1%
0.48M
66.6%
21.1ms
DDA
34.19
0.9601
1.73
158.71G
55.1%
0.78M
45.4%
24.4ms
MBCNN
43.95
0.9909
0.69
2.14T
0.0%
14.2TM
0.0%
132.1mo
MBCNN-0.75x
41.67
0.9853
0.90
1.55T
27.3%
11.88M
16.4%
98. Im.
MBCNN-0.5x
41.31
0.9844
0.94
1.05T
47.5%
9.92M
30.2%
76.3ms
MBCNN-0.25x
40.78
0.9801
0.94
0.65T
60.8%
8.36M
41.2%
61.2ms
DDA
41.68
0.9869
0.85
1.09T
46.9%
10.61M
25.4%
75.2ms
Table 7: Performance comparison between MBCNN accelerated by DDA and state-of-the-ar demoiréing networks on the FHDMi dataset.
Method
DMCNN
MDDM
MopNet
FHDe*Net MBCNN
MBCNN-DDA
PSNR
21.69
20.83
22.76
22.93
23.14
23.62
SSIM
0.7731
0.7343
0.7958
0.7885
0,8201
0.8293
FLOPs
0.41T
0.97T
6,26T
11.41T
4.22T
2.13T
Params
2.37M
8.01M
12.40M
13.57M
14.21M
10.61M
Table 8: Performance comparison between MBCNN accelerated by DDA and state-of-the-art demoiréing networks on the LCDMoiré dataset.
Method
DMCNN
MDDM
MDDM
MopNet
MBCNN
MBCNN-DDA
PSNR 34.58
42.49
43.44
42.02
43.95
41.68
SSIM
0.9612
0.9940
0.9960
0.9872
0.9909
0.9869
FLOPs
476.62G
472.38G
440.44G
3.16T
2.14T
1.09T
Params 2.37M
8.01M
6.55M
12,40M
14.21M
10.61M
dB PSNR with even far fewer FLOPss (2.13T DDA and 11,41T for FHDe which shows the correctness and effectiveness of our perspective for reallocating computation costs in proportion to the moiré complexity of image patches.
As to LCDMoiré, compared with DMCNN-0.75 which simply infers the whole image, our DDA dynamically assigns computation resources with respect to moiré complexity patches, retaining better PSNR performance (34.19 dB for DDA and 33.41 dB for DMCNN-0.75) and more FLOPs duction (55.1% for DDA and 31.4% for DMCNN-O. 7575). Meanwhile, DDA achieves a noticeable la- tency reduction of 56.9ms for accelerating MBCNN (75.2ms for DDA and 132. Ims for the baseline), enabling a real-time image demoiréing on mobile devices. Nevertheless, a noticeable performance drop of PSNR is still observed (41.68 dB for DDA and 43.95 dB for the full model), and compari- son results with state-of-the-art networks including MDDM (Cheng et al. (2019). MDDM+ (Cheng al. 2021) and MopNet (He et al. 2019 in Tab. g also suggest relatively poor result of DDA than its performance on FHDMi. Here we argue that the LCDMoiré dataset is built on simulating the aliasing between CFA and the screen's 's LCD subpixel, which results in images with different distributions of moiré patterns compared with smartphone-captueed moiré images. Compared with smartphone-captured FHDMi with different moiré distributions, the moiré patterns LCDMoiré are much more uniform and their cropped moiré patches are of similar complexity. This explains the relatively poor performance of DDA on the LCDMoiré dataset. Note that our approach even im- proves the performance of baseline models with less computation burden on FHDMi and achieves superior performance in comparison with SOTA networks. Given our motivation for the practical de- ployment of image demoiréing in real cases, the efficacy of the proposed method is still affirmative.
4.4 QUALITATIVE COMPARISON
In addition to the quantitative results, Fig. further displays the visualization results of restored images on the FHDMi dataset. Results on the LCDemoiré dataset can be found in Appendix(A.2) As can be observed, uniformly performing the same accelerating rate for the whole image (MBCNN-
8

---

Published as a conference paper at ICLR 2023
t
33 C
CCC
gories.
meter efficient. fully of parameter-t..
of parameter--x.
al. [29] propose Sque gories.
(DT v al prupose Sque
weter officient. fully of parametet-cu.
include R al [9] propase Sque gories.
Di eeter efficient. fully' of parameter-ca
gories.
Di at al|280 propose Sque
include R al [30) propose Sque
gories.
of parameter-es.
ette efficient, fully
(D ecter efficient, fully
iuclude Ri
include Hi"
include R
Moiré Image
MBCNN
MBCNN-0.5x
DDA
Clear Image
Figure 4:
Visual quality comparison for accelerating MBCNN the FHDMi dataset. The red boxes show a zoomed in area for better observation.
0.5x) drastically degrades the performance as the areas with dense moiré patterns do not receive enough computation resources to be efficiently restored. In contrast, by dynamically allocating the computational resources, our DDA achieves promising demoiréing quality compared with the original network. The efficacy of our proposed DDA for accelerating demoiréing networks for practical application therefore well demonstrated.
LIMITATION AND FUTURE WORK
We further discuss the limitations of our DDA, which will our future focus. Firstly, DDA simply divides all images into equal-number patches in each class, laying some avenues for future research in devising image-aware classification priors. Besides, our limited computing facilities prevent us from accelerating other demoiréing networks with varying structures (Liu et al. 2020: He et al., 2019). More validations are expected to be performed to further demonstrate the efficayy of DDA.
CONCLUSION
In this paper, we have presented a novel dynamic demoiréing acceleration method (DDA) to reduce the huge computational burden of existing networks towards real-time demoiréing mobile de- vices. Our DDA is based on the observation that the moiré complexity is highly unbalanced across different areas of an image. On this basis, we propose split the whole image into sub-put patches, which are then regrouped according to their moiré complexities measured by a novel moiré prior that considers both the frequency and colorfulness information. Then, we use models with different sizes to restore patches in each group. In particular, larger networks are utilized to restore moiré cen- tralized areas to ensure the recovery quality while smaller networks are leveraged to restore moiré diluted areas to relieve computation burden. To avoid the additional parameter burden caused by re- taining multiple networks, we further leverage the supernet paradigm to jointly train the networks in parameter-sharee manner. Results on several benchmarks demonstrate that our method can effec- tively reduce the computation costs of existing networks with negligible performance degradation, enabling a real-time demoiréing on current smartphones.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported by the National Science Fund for Distinguished Young (No.62025603) the National Natural Science Foundation of China (No.62025603, No. U1705262, No. 62072386, No. 62072387, No. 62072389, No, 62002305, No.61772443, No. 61802324 and No. 61702136) and
Guangdong Basic and Applied Basic Research Foundation
REFERENCES
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning (ICML), pp. 527-536. PMLR, 2017.
Bohong Chen, Mingbao Lin, Kekai Sheng, Mengdan Zhang, Peixian Chen, Ke Li, Liujuan Cao, and Rongrong Ji. Arm: Any-time super-resolution method. In European Conference on Computer Vision (ECCV), 2022.
Xi Cheng, Zhenyong Fu, and Jian Yang. Multi-scale dynamic feature encoding network for im- age demoiréing. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). pp. 3488-3343. IEEE, 2019.
Xi Cheng, Zhenyong Fu, and Jian Yang. Improved multi-scale dynamic feature encoding network for image demoiréing. Pattern Recognition, 11::007970, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 248-255, 2009.
Ayush Dogra and Parvinder Bhalla. Image sharpening by gaussian and butterworth high pass filter. Biomedical and pharmacology journal, 7(2):777-713, 2014.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
David Hasler and Sabine E Suesstrunk. Measuring colorfulness in natural images. In Human vi- sion and electronic imaging VIII, volume 5007, pp. 87-95. International Society for Optics and Photonics, 2003.
Bin He, Ce Wang, Boxin Shi, and Ling- Yu Duan. Mop moire patterns using mopnet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2424-2432, 2019.
Bin He, Ce Wang, Boxin Shi, and Ling-Yu Duan. Fhde net: Full high definition demoireing network. In European Conference on Computer Vision (ECCV), pp. 713-729. Springer, 2020.
Gao Huang. Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 29900, 2014.
Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong. Classsr: A general framework to accelerate super-resolution networks by data characteristic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12016-12025, 2021.
Fanglei Liu, Jingyu Yang, and Huanjing Yue. Moiré pattern removal from texture images via low- rank and sparse matrix decomposition. In 2015 Visual Communications and Image Processing (VCIP), pp. 1-4. IEEE, 2015.
Lin Liu. Jianzhuang Liu, Shanxin Yuan, Gregory Slabaugh, Aleš Leonardis, Wengang Zhou, and Qi Tian. Wavelet- based dual-branch network for image demoiréing. In European Conference on Computer Vision (ECCV), pp. 86-102. Springer, 2020.
10

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8026-8037, 2019.
Gaurav Sharma, Wencheng Wu, and Edul N Dalal. The ciede2000 color-difference formula: Im- plementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, 30(1):21-30, 2005.
Yujing Sun, Yizhou Yu, and Wenping Wang. Moiré photo restoration using multiresolution con- volutional neural networks. IEEE Transactions Image Processing (TIP), 27(8)4110-4417, 2018.
Jingyu Yang, Fanglei Liu, Huanjing Yue, Xiaomei Fu, Chumping Hou, and Feng Wu. Textured image demoiréing via signal decomposition and guided filtering. IEEE Transactions on Image Processing (TIP), 26(7):3528-3544 2017a.
Jingyu Yang, Xue Zhang, Changrui Cai, and Kun Li. Demoiréing for screen-shot images with multi-channel layer decomposition. In 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1-4. IEEE, 2017b.
Taojiannan Yang, Sijie Zhu, Matias Mendieta, Pu Wang, Ravikumar Balakrishnan, Minwoo Lee, Tao Han, Mubarak Shah, and Chen Chen. Mutualnet: Adaptive convnet via mutual learning from different model configurations. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2021.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), pp. pp. 1811,
1803-
Shanxin Yuan, Radu Timofte, Gregory Slabaugh, and Aleš Leonardis. Aim 2019 challenge image demoireing: Dataset and study. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 2526-3533. IEEE, 2019.
Bolun Zheng, Shanxin Yuan, Gregory Slabaugh, and Ales Leonardis. Image demoireing with learn- able bandpass filters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2020.

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 MORE VISUALIZATION RESULTS FOR THE MOIRÉ PRIOR
Moiré Complexity Score
FOL
OW US
CONTENT
Moiré Complexity Score
Moiré Complexity Score
Figure 5: The sorted sub-image patches cropped from the FHDMi dataset according to moiré com- plexity scores given by our proposed moiré prior.
A.2 QUALITATIVE RESULTS ON THE LCDMOIRÉ DATASET
encoder
de the
osed approa
is the me
ck shirt.
re 3: Co
Mairé Image
Zoomed
MBCNN
MBCNN-0.5x
DDA
Clear Image
Figure 6: Visual quality comparison for accelerating MBCNN on the LCDMoiré dataset. The red
boxes show a zoomed in for better observation.
12

---

Published as a conference paper at ICLR 2023
Table 9: Training time comparison on the FHDMi and LCDMoiré datasets. We report NVIDIA
Tesla V100 GPU days.
Dataset
DMCNN
DMCNN-DDA
MBCNN
MBCNN-DDA
LCDMoiré 0.42
0.61
3.89
5.77
FHDMi
2.04
3.11
8.19
10.02
A.3 TRAINING TIME COMPARISON
In this section, we report the training time of MBCNN (Zheng 2020). DMCNN (Sun et al, 2018) and their accelerated version by DDA. The results in Tab.9 suggest heavier training con- sumption of DDA compared with the vanilla demoiréing networks. The additional training time stems from more training iterations per epoch in our supernet since the original datasets have been split into multiple patches. Nevertheless, we stress our goal this paper to perform a real-tima deployment with its predominant advantage at the inference efficiency.
13


Reviews:
[{"review_id": "cDA4jRhM3JH", "reviewer": "Reviewer_Y5JK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method of demoireing an image in real time on a mobile device in this body of work. They do this by dynamically selecting areas (patches inside image) by varying complexity and apply different networks to remove the moire pattern. ", "summary_of_the_review": "This paper proposes an innovative method to deal with moire patterns from images. The image is first subdivided and then a color and frequency metric (moire prior) estimates the complexity of the pattern and an appropriate network is applied to remove the moire pattern. The network complexity (supernet) is studied and it's effectiveness is shown in the ablation study. ", "strength_and_weaknesses": "Strengths: \n- The frame work for identifying different areas of image to remove the patterns (referred to as DDA) is quite novel \n- The ablation study shows that instead of only using frequency or color information, a combined metric shows better performance \n\nI wonder if the image can be segmented using a segmentation network to intelligently divide it into patches?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NUOl7GUPJNv", "reviewer": "Reviewer_9cpB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper attempts to propose a solution to clean moire which includes patterns of colored stripes with varied frequencies captured by a camera degrading the visual quality of images. Instead of using a large CNN to clean a whole input image, it targets image patches with lighter CNNs. In order to determine the width of the CNN to be used for each patch, a complexity score based on frequency and colorfulness is utilized. Also, to reduce the parameter burden of utilizing multiple CNNs at the time of deployment, a shared weight training scheme called supernet training is utilized. The resulting solution provides a trade-off for image quality and runtime latency, outperforming compared methods and enabling higher quality real-time mobile device deployment.", "summary_of_the_review": "A method to train a model for image demoireing that can be utilized with reduced run-time latency at the time of deployment is provided. I have some concerns about missing comparisons to the SOTA methods, both lacking a dataset and some well performing methods. I would not the paper an improvement and ready for acceptance before knowing the results of such comparisons. Besides, a few clarifications on the training process could improve the paper.", "strength_and_weaknesses": "- S1: Proposing a method to sort image patches in terms of their moire complexity.\n    \n- S2: Proposing a method to reduce the memory requirement of image demoireing while the drop in image quality of the reconstructed output is less than just simply reducing the network width.\n    \n- W1: The numbers in bold within the tables presenting quantitative results in tables 1,2, and 3 are not representing best results but just the proposed method's results. This could be misleading for the readers.\n    \n- W2: The comparison to SOTA methods is only provided for the FHDMi dataset and the same comparison for LCDMOIRE dataset is missing.\n    \n- W3: The MBCNN method has higher quantitative performance numbers for the LCDMOIRE dataset [2]. For example, it achieves a PSNR of 45.08 in table 6 of that paper. What is the reason for this performance reduction?\n    \n- W4: Some missing methods form SOTA comparisons that has achieved better quantitative results on the LCDMOIRE dataset include Islab-zju with PSNR of 44.07 as shown in table 1 of [1], MDDM with PSNR of 42.49 as shown in table 1 of [3], and MDDM+ with PSNR of 43.44 as shown in table 1 of [4]. It is also not clear without direct comparison if the proposed method achieves a better memory-quality trade off compared to the mentioned methods.\n\n[1] Yuan S, Timofte R, Slabaugh G, Leonardis A, Zheng B, Ye X, Tian X, Chen Y, Cheng X, Fu Z, Yang J. Aim 2019 challenge on image demoireing: Methods and results. In International Conference on Computer Vision Workshop (ICCVW) 2019.\n\n[2] Zheng B, Yuan S, Yan C, Tian X, Zhang J, Sun Y, Liu L, Leonardis A, Slabaugh G. Learning frequency domain priors for image demoireing. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.\n\n[3] Cheng X, Fu Z, Yang J. Multi-scale dynamic feature encoding network for image demoiréing. In International Conference on Computer Vision Workshop (ICCVW) 2019.\n\n[4] Cheng X, Fu Z, Yang J. Improved multi-scale dynamic feature encoding network for image demoiréing. Pattern Recognition. 2021 ;116:107970.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FUn41_3CxM", "reviewer": "Reviewer_Wo5f", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a image demoireing method that focus on its use on mobile phones. This is important as current methods are extremely costly, and cannot therefore be embedded in current devices. The authors propose to first, divide the image into patches, second, given a proposed Moire prior, classify the patches according to the level of Moire that is presented in them, and third, process ech of this patches depending on the Moire level. In orer to make this efficient and to not fight with the parameter burden; the authors propose to have a supernet -in particular a Vanilla Net-, which allows the share of weights for the subnetworks.", "summary_of_the_review": "The paper tackles an improtant problem in current mobile phones, and present an idea that works. Also, it is mostly well written. The only downside is the explanation from the Color Equations, which clearly show that the authors are not from the color community, This said, I believe the paper can be accepted, given that the authors properly correct the explanation about the color parts as explained in the weaknesses section, and add a color metric to the manuscript.", "strength_and_weaknesses": "Strenghts:\n- The idea is interesting, and addresses a real problem in current mobile devices. Dividing the patches according the level of Moiré is clearly and advantage, even for obtaining better results -as shown in Tables 1 and 2, because it avoids the over-whitening of the clear parts of the images.\n\n- As already mentioned, the results are state-of-the-art or very close to it, with a reduction of FLOPs of around 50%.\n\n- Also the paper is mostly well written (although I would put section 4.3.1 just after 4.1.4).\n\nWeaknesses:\n- The authors write about the color part of the Moiré prior (Eq.1) in a way that shows clearly that they have not put enough attention on understanding the color part. They just select one equation from another paper and give a light -to not say worng- explanation of it. Let me detail this. Authors say that the used a measure in the CIELAB color space... but the equation they present is in RGB color space. I agree when studying the equation, that it is definitelu inspired in an opponent color space -from which CIELAB is an example-, because of the relations applied between the R,G, and B channels... However, the CIELAB color space is a non-linear transform from RGB, and in the equation presented here, there is not a non-linearity! I strongly recommend the authors to rewrite this part, stating that they apply a Colorfulness formula on the RGB color space, that is inspired in the chromatic -because luminance is clearly ignored- part of color opponent spaces.\n\n- Regarding to the previous comment (Eq. 1), even if the formula comes from another paper, authors need to briefly explain why there is a relation of 0.3 to 1 in the standard deviation and the mean.\n\n-  In Eq.3, authors need to clarify who is $\\mu$. Is it the mean after computing the high-pass filter? Also, if that is the case, I am surprised about how well both measures combine... as there is a direct multiplication of both. Are they therefore on the same order of magnitude? That should be stated. If not, one may lead to thing that if one is much more larger than the other, will be clearly dominating the prior, and that will mean than the other term is actually meaningless.\n\n- Finally, as the paper are basing their Moiré Prior in some color measure, I believe they should at least add one color metric to their results. I recommend CIE DeltaE.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A45wWho5yu", "reviewer": "Reviewer_LYcM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach for real-time demoire on mobile devices.  Demoireing is an interesting (and hard) problem of removing moire patterns from images, which can easily appear due to high frequency patterns in the scene interfering with the camera color filter array.  It is hard because the moire patterns span a large set of frequencies and colors.  The paper proposes to divide the image into patches, and for each patch assess its moire complexity.  The complexity of the demoire network is adaptive based on the moire complexity; with easier to handle patches receiving less computation compared to harder ones.  This way, the approach can provide the computation where it’s needed, saving unnecessary computations on easy patches.  This is implemented using a supernet approach, and shown to save computation.  Experimental results show improved PSNR and runtime on the FHDMi dataset, and slightly lower PSNR but improved runtime on the LCDMoire dataset compared to MBCNN.  However the method outperforms DMCNN on both datasets.", "summary_of_the_review": "In summary, this paper is addressing an important and difficult problem of demoire.  The supernet approach, coupled with the way to find moire-contaminated patches is interesting and has some (limited) novelty.  The faster results are appreciated by this reviewer, and real-time performance may have important use cases, however these are not articulated in the paper.  The paper seems overly critical of previous work given its results underperform MBCNN on one dataset and are 31% faster than DMCNN on another dataset.  Nonetheless, the paper reflects an advance on the problem on fast demoire, and the paper is the first to explicitly address this problem.  The paper is held back due to clarity issues, which impact reproducibility. ", "strength_and_weaknesses": "Demoireing of images is an interesting and hard problem, recently receiving attention in the computer vision community particularly through the use of deep learning techniques.  As argued by the paper, it is common to take photographs of screens using a smartphone, but the resulting image may be corrupted by moire patterns.  They can occur anytime high frequency patterns interfere with the camara color filter array.  This paper addresses the challenge of real-time demoire as existing approaches have focused primarily on demoire performance and not speed.  The method adopts the supernet strategy to dynamically process patches based on their moire complexity.  The more complex patches receive more computation compared to simpler ones.  The moire complexity is estimated by combining frequency and color estimators together into a score applied to patches.  The score is novel although its components are fairly well established.  The paper shows improvements in speed are possible using the supernet approach.\n\nStrengths:\n\n1.\tThe paper is the first to address real-time moire.  The paper doesn’t really describe why real-time performance is needed however.  Conceivable for photographs, if it takes up to 500 ms to demoire an image most users won’t notice as there is a lag between taking the photo and reviewing the captured photo.  Perhaps the real-time moire is important for video processing.  Perhaps the paper could articulate why real-time more is important, although it seems likely that real-time demoire has more use cases than a slower demoire approach.\n\n2.\tThe method to characterise the patches in terms of their moire complexity is interesting.  The ranking in Figure 2 is compelling, with the severity of the moire pattern increasing from left to right.  The examples in Figure 2 seem to be mostly grayscale however; it would be more convincing if they were more colorful as color is one of the measurements in the scoring criteria.\n\n\n3.\tThe paper appears to be the first to apply a supernet approach to demoire.  This provides a practical approach to reduce computational complexity to perform fast demoire, by applying the computation where it is most needed.  \n\nWeaknesses:\n\n1.\tThe paper seems to overstate the problem given the relative improvement produced by the proposed technique.  For example, in the abstract, the paper describes existing networks “causing massive computation burden”.  In Table 1, DMCNN requires 69.4ms latency whereas DDA requires 48.0ms.  DDA is certainly an improvement, but if DMCNN is a massive computation burden one might wonder if DDA also could be described similar as it is only 31% faster.  Perhaps the language could be toned down.\n\n2.\tPage 2, the paper claims that existing methods bring about side effect, such as over-whitened image contents.  However, the results don’t seem to show this, and indeed MBCNN seems to have better PSNR on LCDMoire than the proposed method.  It would be more convincing if this claim was substantiated somehow, for example in the results section. \n\n3.\tThe paper has novelty in the way it scores patches in terms of their moire complexity and then uses a supernet architecture, however it relies on components derived from other papers. \n\n4.\tSome clarity issues raise hold the paper back.  Clarity is discussed in the next section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster