Title: IODA: Instance-Guided One-shot Domain Adaptation for Super-Resolution
Venue: NeurIPS 2024
Accepted: yes
Keywords: one-shot domain adaptation

super resolution

domain adaptation
Abstract: The domain adaptation method effectively mitigates the negative impact of domain gaps on the performance of super-resolution (SR) networks through the guidance of numerous target domain low-resolution (LR) images. However, in real-world scenarios, the availability of target domain LR images is often limited, sometimes even to just one, which inevitably impairs the domain adaptation performance of SR networks. We propose Instance-guided One-shot Domain Adaptation for Super-Resolution (IODA) to enable efficient domain adaptation with only a single unlabeled target domain LR image. To address the limited diversity of the target domain distribution caused by a single target domain LR image, we propose an instance-guided target domain distribution expansion strategy. This strategy effectively expands the diversity of the target domain distribution by generating instance-specific features focused on different instances within the image. For SR tasks emphasizing texture details, we propose an image-guided domain adaptation method. Compared to existing methods that use text representation for domain difference, this method utilizes pixel-level representation with higher granularity, enabling efficient domain adaptation guidance for SR networks. Finally, we validate the effectiveness of IODA on multiple datasets and various network architectures, achieving satisfactory one-shot domain adaptation for SR networks. Our code is available at https://github.com/ZaizuoTang/IODA.


Peer reviews:
comments: SUMMARY:
The author uses CLIP to assist in extracting the representation of target domain samples and implements a one-shot domain adaptation framework.

STRENGTHS:
The introduction of CLIP improves the performance of one-shot domain adaptation.

WEAKNESSES:
Although the method is effective, it does not appear to be the first time that CLIP has been used for domain adaptation. Moreover, this work seems to merely combine Alpha-CLIP with SR.In addition, the training scenario for CLIP may involve cross-domain scenarios of super-resolution (SR). However, the author did not explain this issue.

QUESTIONS:
Why can Alpha-CLIP enhance the performance of SR when other versions of CLIP cannot?

LIMITATIONS:
The proposed method exhibits slow inference speed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework of efficient instance-guided one-shot domain adaptation (abbr. IODA) with only one unlabeled target domain LR image for addressing image super-resolution (SR) issues. On top of that, it designs an instance-guided target domain distribution expansion strategy to expand the diversity of domain distribution, thus enhancing one-shot DA performance. Extensive ablation studies across multiple datasets and networks have shown the effectiveness of IODA.\n\nSTRENGTHS:\n1. The paper provides a new perspective for solving the problem of image super-resolution based on the domain adaptation method, especially in resource-constrained situations.\n\n2. IODA achieves efficient domain adaptation by using a single unlabeled low-resolution (LR) image of the target domain. Moreover, the instance-guided target domain distribution expansion strategy prevents the pattern collapse by expanding the diversity of domain distribution. These facilitate IODA to carry out practical applications in the real world.\n\nWEAKNESSES:\n1. In order to clearly illustrate the methodology of this paper, the description of the specific implementation of the image-guided domain adaptation and instance-guided target domain distribution extension strategies needs to be further supplemented and refined, in particular with respect to the associated constraints.\n\n2. The evaluation of SR results cannot be limited to pixel-oriented PSNR and SSIM, it is equally important for the evaluation of visual perceptual effects, especially in real-world applications. In addition, visual demonstrations should be provided in the main text, not just in an appendix.\n\nQUESTIONS:\n1. During model training, is the selection of LR images for unlabeled target domains is random? Or does it need to be based on some criteria that are not stated in this manuscript?\n\n2. In practice, is it also necessary to add some screening criteria or conditional restrictions to the distribution expansion strategy to limit its scope? If the selection is random, how to ensure its representativeness?\n\nLIMITATIONS:\nWhile IODA provides inspiration for subsequent solutions to the task of domain adaption-based SR, it may encompass too many existing approaches such as CLIP, Alpha-CLIP, SAM, and it would further enhance the significance and value of this work if some specialized designs or optimizations could be proposed for the current research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to one-shot domain adaptation for super resolution. The key idea is to use the CLIP directional vector between low resoultion source and target domain images to guide the SR image generation in the target domain. They further use occlusion masks to further increase the performance of the model by providing pre-trained Alpha-CLIP with different region-range masks, enabling it to generate Alpha-CLIP spatial features focused on different areas of the image, thereby expanding the diversity of the target domain distribution.\n\nSTRENGTHS:\n- The paper has a well-organized structure and expresses its ideas clearly.\n- Ablation analysis is provided in the paper to study the effect of various components of the pipeline on the downstream performance.\n- The idea is pretty interesting and new and the motivation is sound.\n\nWEAKNESSES:\n- Given the high time complexity of the approach, the performance gains over the baseline seems to be small.\n- The SR performance of the model needs to be compared against stronger SR baselines to better analyze the performance gains.\n- More visualization analysis is needed in the paper to better support the performance gains.\n\nQUESTIONS:\n- Can you compare the performance of your models against the other domain adaptation-based SR approaches like [12] and [13]?\n- Can you provide a more comprehensive visualization analysis to better support you performance claims?\n- Can you provide a comparison of the efficiency between this method and other methods? A comparison that considers both efficiency and PSNR/SSIM would be more reasonable.\n\n\n\n\n\n[12] Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longcun Jin, and Hengjie Song. Unsuper304 vised real-world image super resolution via domain-distance aware training. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13380–13389, 2021.\n13] Wei Wang, Haochen Zhang, Zehuan Yuan, and Changhu Wang. Unsupervised real-world super307 resolution: A domain adaptation perspective. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4298–4307, 2021.\n\nLIMITATIONS:\nThe main limitations of this paper is the relatively low training efficiency that would impose certain limitations in practical scenarios. Although the idea looks nice, but as mentioned in the paper, it takes approx 10 mins to generate a SR version of a single image. Therefore, I'm not completely convinced about its complexity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses one-shot domain adaptation (OSDA) in the field of super-resolution (SR). It leverages the fact that the content remains unchanged during super-resolution to propose an image-guided domain adaptation method, ensuring consistency by aligning the direction between the source domain and the target domain. The authors highlight the difficulty in learning the target domain's distribution during OSDA in SR and propose increasing data samples using random masking, similar to MAE. They also utilize SAM and Alpha Clip to obtain instance-aware representations. The method trains to align the direction between the source and target domains using the representation obtained from Alpha Clip. Various experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n- They propose an OSDA method for real-world applications of super-resolution (SR).\n- They effectively utilize foundation models to address the domain adaptation (DA) challenges in the SR task.\n\nWEAKNESSES:\nThere seems to be an insufficient survey of domain adaptation methods in the SR task, specifically those outlined in references [1-4]. Major revision is needed to emphasize the necessity and originality of the proposed method, based on an analysis of these existing methods. Particularly, since [4] addresses test-time adaptation, it is essential to highlight the advantages of one-shot domain adaptation over test-time adaptation. Additionally, the experimental tables predominantly report performance improvements over the baseline, but a comparison with existing domain adaptation methods is also required.\n\n[1] Wang, Wei, et al. \"Unsupervised real-world super-resolution: A domain adaptation perspective.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[2] Wei, Yunxuan, et al. \"Unsupervised real-world image super resolution via domain-distance aware training.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[3] Xu, Xiaoqian, et al. \"Dual adversarial adaptation for cross-device real-world image super-resolution.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[4] Deng, Zeshuai, et al. \"Efficient test-time adaptation for super-resolution with second-order degradation and reconstruction.\" Advances in Neural Information Processing Systems 36 (2023): 74671-74701.\n\nQUESTIONS:\nPlease refer to the Weaknesses section\n\nLIMITATIONS:\nThey've addressed the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}