Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a method of demoireing an image in real time on a mobile device in this body of work. They do this by dynamically selecting areas (patches inside image) by varying complexity and apply different networks to remove the moire pattern. 
strength_and_weaknesses: Strengths: 
- The frame work for identifying different areas of image to remove the patterns (referred to as DDA) is quite novel 
- The ablation study shows that instead of only using frequency or color information, a combined metric shows better performance 

I wonder if the image can be segmented using a segmentation network to intelligently divide it into patches?

clarity,_quality,_novelty_and_reproducibility: - The idea is well presented and clearly written
- Code was provided 


summary_of_the_review: This paper proposes an innovative method to deal with moire patterns from images. The image is first subdivided and then a color and frequency metric (moire prior) estimates the complexity of the pattern and an appropriate network is applied to remove the moire pattern. The network complexity (supernet) is studied and it's effectiveness is shown in the ablation study.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper attempts to propose a solution to clean moire which includes patterns of colored stripes with varied frequencies captured by a camera degrading the visual quality of images. Instead of using a large CNN to clean a whole input image, it targets image patches with lighter CNNs. In order to determine the width of the CNN to be used for each patch, a complexity score based on frequency and colorfulness is utilized. Also, to reduce the parameter burden of utilizing multiple CNNs at the time of deployment, a shared weight training scheme called supernet training is utilized. The resulting solution provides a trade-off for image quality and runtime latency, outperforming compared methods and enabling higher quality real-time mobile device deployment.
strength_and_weaknesses: - S1: Proposing a method to sort image patches in terms of their moire complexity.
    
- S2: Proposing a method to reduce the memory requirement of image demoireing while the drop in image quality of the reconstructed output is less than just simply reducing the network width.
    
- W1: The numbers in bold within the tables presenting quantitative results in tables 1,2, and 3 are not representing best results but just the proposed method's results. This could be misleading for the readers.
    
- W2: The comparison to SOTA methods is only provided for the FHDMi dataset and the same comparison for LCDMOIRE dataset is missing.
    
- W3: The MBCNN method has higher quantitative performance numbers for the LCDMOIRE dataset [2]. For example, it achieves a PSNR of 45.08 in table 6 of that paper. What is the reason for this performance reduction?
    
- W4: Some missing methods form SOTA comparisons that has achieved better quantitative results on the LCDMOIRE dataset include Islab-zju with PSNR of 44.07 as shown in table 1 of [1], MDDM with PSNR of 42.49 as shown in table 1 of [3], and MDDM+ with PSNR of 43.44 as shown in table 1 of [4]. It is also not clear without direct comparison if the proposed method achieves a better memory-quality trade off compared to the mentioned methods.

[1] Yuan S, Timofte R, Slabaugh G, Leonardis A, Zheng B, Ye X, Tian X, Chen Y, Cheng X, Fu Z, Yang J. Aim 2019 challenge on image demoireing: Methods and results. In International Conference on Computer Vision Workshop (ICCVW) 2019.

[2] Zheng B, Yuan S, Yan C, Tian X, Zhang J, Sun Y, Liu L, Leonardis A, Slabaugh G. Learning frequency domain priors for image demoireing. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.

[3] Cheng X, Fu Z, Yang J. Multi-scale dynamic feature encoding network for image demoiréing. In International Conference on Computer Vision Workshop (ICCVW) 2019.

[4] Cheng X, Fu Z, Yang J. Improved multi-scale dynamic feature encoding network for image demoiréing. Pattern Recognition. 2021 ;116:107970.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The writing is easy to follow. The method is clearly discussed, except the use cases of previous works such as supernet training for which it is not clear how the parameters were shared or how much time would it consume to train in such a manner compared to individual small CNN models or a full image CNN.
    
- Quality: There are some concerns about the quality of the reconstructions as discussed in W2, W3, and W4 above.
    
- Novelty: The proposed approach utilizes previous methods for the shared training and achieving components of the moire complexity metric. However, it could still be considered novel for the application in demoireing.
summary_of_the_review: A method to train a model for image demoireing that can be utilized with reduced run-time latency at the time of deployment is provided. I have some concerns about missing comparisons to the SOTA methods, both lacking a dataset and some well performing methods. I would not the paper an improvement and ready for acceptance before knowing the results of such comparisons. Besides, a few clarifications on the training process could improve the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a image demoireing method that focus on its use on mobile phones. This is important as current methods are extremely costly, and cannot therefore be embedded in current devices. The authors propose to first, divide the image into patches, second, given a proposed Moire prior, classify the patches according to the level of Moire that is presented in them, and third, process ech of this patches depending on the Moire level. In orer to make this efficient and to not fight with the parameter burden; the authors propose to have a supernet -in particular a Vanilla Net-, which allows the share of weights for the subnetworks.
strength_and_weaknesses: Strenghts:
- The idea is interesting, and addresses a real problem in current mobile devices. Dividing the patches according the level of Moiré is clearly and advantage, even for obtaining better results -as shown in Tables 1 and 2, because it avoids the over-whitening of the clear parts of the images.

- As already mentioned, the results are state-of-the-art or very close to it, with a reduction of FLOPs of around 50%.

- Also the paper is mostly well written (although I would put section 4.3.1 just after 4.1.4).

Weaknesses:
- The authors write about the color part of the Moiré prior (Eq.1) in a way that shows clearly that they have not put enough attention on understanding the color part. They just select one equation from another paper and give a light -to not say worng- explanation of it. Let me detail this. Authors say that the used a measure in the CIELAB color space... but the equation they present is in RGB color space. I agree when studying the equation, that it is definitelu inspired in an opponent color space -from which CIELAB is an example-, because of the relations applied between the R,G, and B channels... However, the CIELAB color space is a non-linear transform from RGB, and in the equation presented here, there is not a non-linearity! I strongly recommend the authors to rewrite this part, stating that they apply a Colorfulness formula on the RGB color space, that is inspired in the chromatic -because luminance is clearly ignored- part of color opponent spaces.

- Regarding to the previous comment (Eq. 1), even if the formula comes from another paper, authors need to briefly explain why there is a relation of 0.3 to 1 in the standard deviation and the mean.

-  In Eq.3, authors need to clarify who is $\mu$. Is it the mean after computing the high-pass filter? Also, if that is the case, I am surprised about how well both measures combine... as there is a direct multiplication of both. Are they therefore on the same order of magnitude? That should be stated. If not, one may lead to thing that if one is much more larger than the other, will be clearly dominating the prior, and that will mean than the other term is actually meaningless.

- Finally, as the paper are basing their Moiré Prior in some color measure, I believe they should at least add one color metric to their results. I recommend CIE DeltaE.
clarity,_quality,_novelty_and_reproducibility: - The paper is very clear in most of the parts. As stated before, my only complain is that I believe section 4.3.1 should be put before, in a way that allows presenting some supernet results already in Tables 1,2,3. 

- I am satisfied by the quality of the work.

- The novelty is good. Even if it seems quite a straighforward idea; it was not proposed before, and that is actually a very positivie thing.

- The authors seems quite straigforward to reproduce, although I would strongly encourage the authors to release their code -there are not comment on that in the manuscript-.


summary_of_the_review: The paper tackles an improtant problem in current mobile phones, and present an idea that works. Also, it is mostly well written. The only downside is the explanation from the Color Equations, which clearly show that the authors are not from the color community, This said, I believe the paper can be accepted, given that the authors properly correct the explanation about the color parts as explained in the weaknesses section, and add a color metric to the manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for real-time demoire on mobile devices.  Demoireing is an interesting (and hard) problem of removing moire patterns from images, which can easily appear due to high frequency patterns in the scene interfering with the camera color filter array.  It is hard because the moire patterns span a large set of frequencies and colors.  The paper proposes to divide the image into patches, and for each patch assess its moire complexity.  The complexity of the demoire network is adaptive based on the moire complexity; with easier to handle patches receiving less computation compared to harder ones.  This way, the approach can provide the computation where it’s needed, saving unnecessary computations on easy patches.  This is implemented using a supernet approach, and shown to save computation.  Experimental results show improved PSNR and runtime on the FHDMi dataset, and slightly lower PSNR but improved runtime on the LCDMoire dataset compared to MBCNN.  However the method outperforms DMCNN on both datasets.
strength_and_weaknesses: Demoireing of images is an interesting and hard problem, recently receiving attention in the computer vision community particularly through the use of deep learning techniques.  As argued by the paper, it is common to take photographs of screens using a smartphone, but the resulting image may be corrupted by moire patterns.  They can occur anytime high frequency patterns interfere with the camara color filter array.  This paper addresses the challenge of real-time demoire as existing approaches have focused primarily on demoire performance and not speed.  The method adopts the supernet strategy to dynamically process patches based on their moire complexity.  The more complex patches receive more computation compared to simpler ones.  The moire complexity is estimated by combining frequency and color estimators together into a score applied to patches.  The score is novel although its components are fairly well established.  The paper shows improvements in speed are possible using the supernet approach.

Strengths:

1.	The paper is the first to address real-time moire.  The paper doesn’t really describe why real-time performance is needed however.  Conceivable for photographs, if it takes up to 500 ms to demoire an image most users won’t notice as there is a lag between taking the photo and reviewing the captured photo.  Perhaps the real-time moire is important for video processing.  Perhaps the paper could articulate why real-time more is important, although it seems likely that real-time demoire has more use cases than a slower demoire approach.

2.	The method to characterise the patches in terms of their moire complexity is interesting.  The ranking in Figure 2 is compelling, with the severity of the moire pattern increasing from left to right.  The examples in Figure 2 seem to be mostly grayscale however; it would be more convincing if they were more colorful as color is one of the measurements in the scoring criteria.


3.	The paper appears to be the first to apply a supernet approach to demoire.  This provides a practical approach to reduce computational complexity to perform fast demoire, by applying the computation where it is most needed.  

Weaknesses:

1.	The paper seems to overstate the problem given the relative improvement produced by the proposed technique.  For example, in the abstract, the paper describes existing networks “causing massive computation burden”.  In Table 1, DMCNN requires 69.4ms latency whereas DDA requires 48.0ms.  DDA is certainly an improvement, but if DMCNN is a massive computation burden one might wonder if DDA also could be described similar as it is only 31% faster.  Perhaps the language could be toned down.

2.	Page 2, the paper claims that existing methods bring about side effect, such as over-whitened image contents.  However, the results don’t seem to show this, and indeed MBCNN seems to have better PSNR on LCDMoire than the proposed method.  It would be more convincing if this claim was substantiated somehow, for example in the results section. 

3.	The paper has novelty in the way it scores patches in terms of their moire complexity and then uses a supernet architecture, however it relies on components derived from other papers. 

4.	Some clarity issues raise hold the paper back.  Clarity is discussed in the next section.

clarity,_quality,_novelty_and_reproducibility: The paper’s clarity could be improved in places.  Some of these points may relate to things not discussed in the paper and impact reproducibility:

1.	Some patches will have no moire patterns.  How does the supernet deal with these?  Perhaps there could be a “passthrough” option for patches that have no moire, they require no processing at all by the demoire network.

2.	The network architecture of DDA isn’t so clear.  How does the supernet split the computation for demoire?  It appears the supernet is also applied to MBCNN, but it wasn’t clear to this reviewer how this is achieved, as MBCNN is a multi-scale architecture and not so easy to split into different computational blocks.  Perhaps the author rebuttal can comment on this.

3.	What are the parameters used to compute Equation 2?  For example, the parameters of the Gaussian high pass kernel?  Without describing these it because difficult to reproduce the paper.

4.	It wasn’t clear if the source code and pretrained models will be released should the paper be accepted.

5.	After Equation 5, the paper states that output patches are concatenated to reconstruct the full image.  In this reviewer’s experience, doing this type of processing may result in artifacts at patch boundaries, because each patch was processed independently by the network.  Are such artifacts present in the proposed results?  How to handle these artifacts?

6.	In Table 2, MBCNN performs best on LCDMoire in terms of PSNR and SSIM.  It would be best to make it bold in this table.  Usually a bold font is used to denote the best performing method.


The main originality comes from the supernet approach and how patches are characterized.  The supernet appears in other literature.  The characterization of patches is interesting and has some novelty, although the components in the score are known.  The paper has some novel elements but may read more as a systems paper bringing together known concepts into a practical working algorithm.

Smaller issues: 

•	Throughout the paper, please use “complexity” when discussing computation.  For example, in the abstract, please change “higher-complex” to “higher-complexity”; and “lower-complex” to “lower-complexity”.  Please note this issue appears several times in the paper, not just the abstract.

•	Page 1, the phrase “on offline mobile devices” seems unusual.  In what sense are the mobile devices offline?  Note most smartphones have accelerators capable of running neural networks.  Perhaps the word “offline” can be removed.  

•	Page 1, please change “contaminated by the moire.” to “contaminated by the moire pattern.”

•	Page 2, please change “Detailedly,” to “In detail,”

•	Page 2, please change “resource-hungry” to “resource-limited”

•	Page 3, please change “pioneer work” to “pioneering work”.  Note this issue also appears on Page 6.

•	Page 4, please change “by human” to “by a human”

•	Page 4, please change “or full-connected” to “or fully-connected”

•	Page 4, please change “not the optimal” to “not optimal”

•	Throughout the paper, please change “zooming-in” to “zoomed in”

•	Table 2 uses the term LCDMoire dataset, but Table 4 calls it AIM.  Please choose one for consistency.

•	Page 9, please change “nowadays smartphones” to “current smartphones”

summary_of_the_review: In summary, this paper is addressing an important and difficult problem of demoire.  The supernet approach, coupled with the way to find moire-contaminated patches is interesting and has some (limited) novelty.  The faster results are appreciated by this reviewer, and real-time performance may have important use cases, however these are not articulated in the paper.  The paper seems overly critical of previous work given its results underperform MBCNN on one dataset and are 31% faster than DMCNN on another dataset.  Nonetheless, the paper reflects an advance on the problem on fast demoire, and the paper is the first to explicitly address this problem.  The paper is held back due to clarity issues, which impact reproducibility.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors proposed an efficient image demoireing method that is friendly to mobile devices. In particular, a dynamic mechanism is applied to allocate computation costs adaptively based on the complexity of image patches and the strength of moire patterns. 

Strengths: 
(1) The idea is simple but efficient, and the engineering implementation is non-trivial. 
(2) Experimental results are convincing.

Weaknesses:
(1) The authors should further polish this paper and fix typos.
(2) The authors should add more analytic content and discuss the potential of the proposed method to other learning tasks, which will helps to attract more researchers in this community. 
justification_for_why_not_higher_score: (1) The method is simple and efficient, but its generalizability is not fully discussed.
(2) Although the idea is interesting, the application itself (i.e., demoireing) is not an attractive topic in ICLR.
justification_for_why_not_lower_score: As aforementioned, the methodology is interesting and efficient, and the engineering part of this work is solid in my opinion.
Decision: Accept: poster