Title: Towards Syn-to-Real IQA: A Novel Perspective on Reshaping Synthetic Data Distributions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Blind Image Quality Assessment; Data Distribution Reshaping; Synthetic Data
Abstract: Blind Image Quality Assessment (BIQA) has advanced significantly through deep learning, but the scarcity of large-scale labeled datasets remains a challenge. While synthetic data offers a promising solution, models trained on existing synthetic datasets often show limited generalization ability. In this work, we make a key observation that representations learned from synthetic datasets often exhibit a discrete and clustered pattern that hinders regression performance: features of high-quality images cluster around reference images, while those of low-quality images cluster based on distortion types. Our analysis reveals that this issue stems from the distribution of synthetic data rather than model architecture. Consequently, we introduce a novel framework SynDR-IQA, which reshapes synthetic data distribution to enhance BIQA generalization. Based on theoretical derivations of sample diversity and redundancy's impact on generalization error, SynDR-IQA employs two strategies: distribution-aware diverse content upsampling, which enhances visual diversity while preserving content distribution, and density-aware redundant cluster downsampling, which balances samples by reducing the density of densely clustered areas. Extensive experiments across three cross-dataset settings (synthetic-to-authentic, synthetic-to-algorithmic, and synthetic-to-synthetic) demonstrate the effectiveness of our method. The code is available at https://github.com/Li-aobo/SynDR-IQA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Blind Image Quality Assessment method employing a synthetic distortion to real distortion framework. The proposed method uses distribution-aware upsampling to obtain diverse reference contents, and density aware redundant cluster downsampling to reduce excessive redundant samples in the dataset. Experiments demonstrate that these strategies improve the performance (incrementally) across many IQA datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The only novelty I see in this work is that it proposes a refinement methodology to improve diversity and reduce redundancy in the training dataset. This strategy (although designed in a very ad-hoc manner) does provide some performance benefits especially for cross dataset generalization.\n\nWeakness\n1. Related Work is incomplete. The authors have missed out key references which employ distortion classification for BIQA task\n\na. W. Kim, A.-D. Nguyen, S. Lee, and A. C. Bovik, “Dynamic receptive field generation for full-reference image quality assessment,” IEEE Trans. Image Process., vol. 29, pp. 4219–4231, 2020.\n\nb. Madhusudana, P. C., Birkbeck, N., Wang, Y., Adsumilli, B., and Bovik, A. C. Image quality assessment using\ncontrastive learning. IEEE Transactions on Image Processing, 31:4149–4161, 2022.\n\nc. Saha, A., Mishra, S., and Bovik, A. C. Re-iqa: Unsupervised learning for image quality assessment in the wild. In\nProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5846–5855, 2023\n\n2. I find it hard to connect the proposed method with Theorem 3.1. The conclusion from the theorem is pretty generic and there is no BIQA specific concept. That theorem is applicable to any clustering scheme.\n\n3. The Algorithm 1,2 and 3 look quite ad-hoc and not very intuitive. The authors use many heuristics in the algorithm without proper justification. For example in Algorithm 1 why minimum, maximum and median is used at different places? For algorithm 2 why distortion intensities of levels 1, 3, and 5 are used for all distortions? Shouldn't that be distortion specific?\n\n4. The performance improvement is often incremental when compared to the current state-of-the-art. Also many IQA datasets have been skipped from reporting\n\na. Z. Ying, H. Niu, P. Gupta, D. Mahajan, D. Ghadiyaram, and A. Bovik, “From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2020, pp. 3575–3585.\n\nb. Y. Fang, H. Zhu, Y. Zeng, K. Ma, and Z. Wang, “Perceptual quality assessment of smartphone photography,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2020, pp. 3677–3686.\n\n5. In the statement \"Notably, to reduce the increase of redundant samples and prevent additional label noise, only distortion intensities of levels 1, 3, and 5 are implemented.\" What exactly are these levels? These levels can have different meaning for different distortions?\n\nQUESTIONS:\n1. It is a bit confusing when authors interchangeably use synthetic dataset and synthetic distortions. Synthetic dataset can mean different things such as animation images, images generated through generative models etc., whereas in this context it primarily refers to natural images which are corrupted with synthetic distortions\n\n2. Is there a way to evaluate Algorithm 1 alone? The goal was to enrich training data by increasing distribution diversity. Can we qualitatively or quantitatively evaluate if the proposed scheme is actually doing this?\n\n3. Provide more intuitions on how Algorithm 1, 2 and 3 were designed. At a high level all these methods look like a bunch of 'if' conditions\n\n4. The authors mention \"only distortion intensities of levels 1, 3, and 5 are implemented\". It would be interesting to know if all distortion levels were included and performance was evaluated. This will also throw light on how each distortion level influences the overall quality related feature learning.\n\nLIMITATIONS:\nThe authors acknowledge the limitations in generalization capability between synthetic and authentic distortions. I believe there is one more limitation which is based on the assumption \"similar content under the same distortion conditions should result in similar quality degradation\".", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that that synthetic IQA datasets tend to form discrete, clustered feature spaces, high-quality images are clustered by reference, low-quality by distortion type, and medium-quality samples generally scatter inconsistently which undermines regression performance on authentic, real-world images. \n\nTo overcome this, the papers propose a new method called SynDR-IQA, constituting of DDCUp to selectively up-sample underrepresented content and DRCDown to prune over-dense clusters. Results show that the proposed method improves over compared methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors identify a very important problem in the BIQA space. The authors identify synthetic distortions often cluster unnaturally in feature space and argue that can hurt regression-based IQA models.\n2. Relatively good number of experiments presented in this paper, along with some ablations. \n\nWeaknesses:\n1. Experiments use a supervised pre-trained ResNet-50 (and some expts with one Swin transformer variant). The paper does not justify why using a ImageNet pretrained network is a reliable estimate of content representation of \"real world\" images? It is strange that in 2025 the authors did not consider using more recent pre-trained vision encoders like CLIP/DINO/DINOv2/SigLIP or even self supervised imagenet pre-trained encoders like MoCO/SimCLR/JEPA/BYOL. \n2. There are inherent biases associated with using a pre-trained encoders. \n3. Algorithm complexity/Runtime : There is no discussion on algorithmic complexities or runtime requirements for Algo 1-3. Also there are no intuitions on how the algorithms are designed, especially algorithm 1. For example, the authors claim in the paper  \"original content distribution while also being sufficiently distinct from each other (lines 5 and 7),\" there seems to be missing inspiration why and how this design was reached.\n4. Some of the results reported in Table 3 show really small gains (cross database). It would be great if some sort of statistical significance test is performed to confirm of definite improvement. One option would be repeat the training for both the baseline and proposed 10 times or similar. \n5. Also the authors should also compare their method with the popular family of un/self-supervised methods developed specially for IQA like CONTRIQUE, Re-IQA, ARNIQA etc.\n\nQUESTIONS:\n1. Effect of Encoders: More recent pre-trained vision encoders like CLIP/DINO/DINOv2/SigLIP could be tried.\n2. Please comment on the algorithmic complexity/Runtime of the proposed algorithms.\n3. The results in Table 3 could be made more convincing with addition of statistical significance results. \n\nThere are major concerns I have. Addressing these partialy or fully can help change my decision on the paper.\n\nLIMITATIONS:\nSome limitations are discussed. There are no expected potential negative societal implications of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the generalization gap in Blind Image Quality Assessment (BIQA) between synthetic and real-world distortions. The authors propose SynDR-IQA, a framework to reshape synthetic data distributions by increasing content diversity and reducing sample redundancy. Two strategies are introduced: (1) Distribution-aware Diverse Content Upsampling (DDCUp) and (2) Density-aware Redundant Cluster Downsampling (DRCDown). A theoretical generalization bound is derived to justify the design choices. The method is evaluated on standard cross-dataset BIQA benchmarks and demonstrates improvements over prior work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Targets a very important problem of limited generalization of synthetic datasets prevalent in BIQA\n\n- The paper provides a theoretically grounded explanation for the observed failure mode, using a generalization bound for clustered data.\n\n- The proposed techniques show measurable improvements over recent baselines on several cross-domain benchmarks.\n\n- The ablation and visualization studies support the hypothesis.\n\nWeakness\n\n- Both proposed techniques (DDCUp and DRCDown) are relatively simple data augmentation and filtering heuristics. There is little innovation in model design or learning methodology.\n\n- The pseudo-labeling strategy in DDCUp assumes that visually similar content leads to similar distortion behavior, which may not generalize beyond synthetic datasets. No evaluation is done to test this assumption under failure conditions.\n\n- What is the reason for suboptimal performance of the approach in LIVEC and KONIQ-10k datasets as shown in Table 1 ?\n\n- The paper lacks comparison of data downsampling approach with existing approaches such as clustering or data pruning baselines\n\nQUESTIONS:\n• How sensitive is DDCUp to the threshold used for selecting pseudo-label neighbors? Would performance degrade if content distribution shifts?\n\n• Why was DRCDown not compared with other sampling strategies (e.g., k-means undersampling, core-set selection, curriculum filtering)?\n\n• Can you provide a variance analysis on key benchmark improvements?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of synthetic-to-real generalization in BIQA. The authors make a key observation that models trained on synthetic data tend to produce discrete, clustered feature representations that hinder generalization. Through theoretical analysis, they attribute this limitation to insufficient content diversity and sample redundancy. To overcome these challenges, they propose SynDR-IQA, a data-centric framework with two novel strategies (DDCup and DRCDown) that reshapes the training data distribution to improve generalization. Extensive experiments across multiple cross-dataset settings validate the framework's effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper provides a novel and insightful perspective on why synthetic-to-real generalization fails in IQA, providing rigorous theoretical analysis of how sample diversity and redundancy affect generalization bounds.\n2. The proposed DDCup and DRCDown strategies directly address the identified clustering problems through a well-motivated, data-centric approach that requires no architectural changes.\n3. Consistent improvements across multiple cross-dataset evaluation settings demonstrate the effectiveness and broad applicability of the framework.\n4. The paper is clearly written and well-organized.\n\nWeakness\n1. The proposed DDCup and DRCDown strategies rely on several hyperparameters whose sensitivity is not thoroughly analyzed.\n2. More visualization results on other synthetic and real-world datasets should be provided to better support the main claims.\n3. The UMAP visualization for the model \"Trained directly on KADID-10k\" should also be added to Figure 3 to better illustrate the limitation of the original synthetic dataset.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about improving the generalizability of no-reference image quality assessment methods that are trained on limited data. The authors provide a key observation that representations learned from synthetic data tend to present a discrete and clustered pattern, and this issue is due to the distribution of synthetic data, especially the given distortion types. To solve this problem, the paper proposes a novel framework called SynDR-IQA to reshape the synthetic data distribution. Specifically, after theoretical analysis, the framework distinctly upsample diverse content and downsample the redundant cluster to balance the distribution. The abundant cross-dataset evaluation demonstrates the impressive generalizability of the proposed SynDR-IQA.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The figure 1 well illustrates the limitation of training BIQA methods using synthetic data. The visualization on large-scale KADID-10K provided convincing evidence of the key challenge of distribution imbalance.\n2. The paper writing is clear and easy to understand. The introduction gives a clear entry into the current challenge of data distribution of synthetic data and elicits the explicit solution to reshape the distribution.\n3. Understandable theoretical analysis and extensive experimental results. The cross-dataset evaluation are based on multiple settings, including syn-to-real test.\n\nWeakness:\n1. Lack of comparison with more training data. The paper presents a potential solution to improve the generalizability of BIQA methods using distribution reshaping. However, directly using a larger training dataset can also solve the problem. The proposed SynDR-IQA is only effective if it can achieve similar performance compared to the same model trained on a much larger dataset, which means the SynDR-IQA can save a significant amount of native training data. In my opinion, an ablation experiment can be added that compares the SynDR-IQA and using a larger dataset to make the framework more convincing. Please note that it is not mandatory to complete the experiment during rebuttal since training with large dataset is very time-consuming. I recommend the authors can add this experiment in their final version.\n2. Figure 1 lacks some image samples, making it somewhat abstract. The figure 2 contains some samples that make it more vivid.\n3. No examples of content upsampling. Providing some original training set and corresponding selected reference image for upsampling can be very helpful to understand.\n\nQUESTIONS:\n1. Can Figure 1 add some image samples? The Figure 1 only presents abstract discrete points, maybe providing a few corresponding image samples can make the figure more understandable, especially the images with high and low quality.\n2. How the content upsampling actually works? Does it just select additional images with related content from another dataset (KADIS-700K)? In this way, can a larger pool of reference images (maybe from diverse large-scale datasets) be used for selection?\n3. Is the content upsampling time-consuming? It seems selecting reference images from a large dataset can be hard to implement and time-consuming.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}