Title: Latent Harmony: Synergistic Unified UHD Image Restoration via Latent Space Regularization and Controllable Refinement
Venue: NeurIPS 2025
Accepted: yes
Keywords: Ultra-high-definition

Latent Regularization

Image Restoration

Fidelity-Perception Trade-off
Abstract: Ultra-High Definition (UHD) image restoration struggles to balance computational efficiency and detail retention.
While Variational Autoencoders (VAEs) offer improved efficiency by operating in the latent space, with the Gaussian variational constraint, this compression preserves semantics but sacrifices critical high-frequency attributes specific to degradation and thus compromises reconstruction fidelity. 
% This compromises reconstruction fidelity, even when global semantics are preserved.
Consequently, a VAE redesign is imperative to foster a robust semantic representation conducive to generalization and perceptual quality, while simultaneously enabling effective high-frequency information processing crucial for reconstruction fidelity.
To address this, we propose  \textit{Latent Harmony}, a two-stage framework that reinvigorates VAEs for UHD restoration by concurrently regularizing the latent space and enforcing high-frequency-aware reconstruction constraints. 
Specifically, Stage One introduces the LH-VAE, which fortifies its latent representation through visual semantic constraints and progressive degradation perturbation for enhanced semantics robustness; meanwhile, it incorporates latent equivariance to bolster its high-frequency reconstruction capabilities. 
Then, Stage Two facilitates joint training of this refined VAE with a dedicated restoration model.
This stage integrates High-Frequency Low-Rank Adaptation (HF-LoRA), featuring two distinct modules: an encoder LoRA, guided by a fidelity-oriented high-frequency alignment loss, tailored for the precise extraction of authentic details from degradation-sensitive high-frequency components; and a decoder LoRA, driven by a perception-oriented loss, designed to synthesize perceptually superior textures. These LoRA modules are meticulously trained via alternating optimization with selective gradient propagation to preserve the integrity of the pre-trained latent structure. This methodology culminates in a flexible fidelity-perception trade-off at inference, managed by an adjustable parameter 
$\alpha$.
Extensive experiments demonstrate that \textit{Latent Harmony} effectively balances perceptual and reconstructive objectives with efficiency, achieving superior restoration performance across diverse UHD and standard-resolution  scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Latent Harmony, a two-stage VAE-based framework for UHD image restoration, addressing the trade-off between computational efficiency and detail retention. In Stage One, they introduce LH-VAE, which enhances latent space robustness through semantic constraints and degradation perturbations while improving high-frequency reconstruction via latent equivariance. In Stage Two, they integrate HF-LoRA, a High-Frequency Low-Rank Adaptation module with encoder-decoder components optimized for fidelity and perceptual quality, trained via alternating gradient propagation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The motivation is summarized in detail and explained deeply.\n2. The experiments are rich and can prove the effectiveness of the proposed method.\nWeakness:\n1. The authors do not propose an innovative architecture, but only design multiple loss functions and training strategies.\n2. The frequent appearance of the \"xx's \" writing style in the article is not standardized.\n\nQUESTIONS:\n1. Although Figure 1 is presented, it is not explained in the manuscript.\n2. Figure 3 shows the entire method overview, but I can't tell what type of degradation the displayed figure shows. Furthermore, the fidelity of the restored images corresponding to different α values shows a positive correlation. Can the author explain this phenomenon?\n3. In the first stage of training, the authors propose the PDPS strategy to enhance the robustness of VAE against degradation in the latent space. Among them, p0, p1, and p2 control different perturbations, respectively, but the authors do not provide their values or reasons.\n4. Eq 4 and 5 represent two loss functions. However, for Eq 4, the author emphasizes that it is a distance metric, while Eq 5 is an L1 norm. What are the differences between the two equations?\n5. In fact, the authors adopt the VAE-based image restoration pipeline, but most of the methods compared in the experiment are based on Transform or diffusion methods. It is necessary to explain the advantages of VAE compared with these methods.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Latent Harmony, a two-stage framework for UHD image restoration that enhances the balance between efficiency, reconstruction fidelity, and generalization. In the first stage, a specially designed LH-VAE is trained using progressive degradation perturbation, semantic alignment via DINOv2 features, and latent space equivariance to build a robust and generalizable latent representation. In the second stage, the authors introduce high-frequency-guided low-rank adaptation which fine-tunes the encoder for fidelity and the decoder for perceptual quality using separate high-frequency losses. Extensive experiments show that the method achieves state-of-the-art performance with lower computational cost and strong generalization across various degradation types.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n* The proposed two-stage pipeline is well-structured and experimentally validated on both UHD and standard-resolution restoration tasks.\n* Ablation studies are comprehensive and demonstrate the impact of each component.\n* The results show strong quantitative performance with reduced FLOPs and parameter counts, making a compelling case for practical efficiency.\n\n**Weaknesses**:\n* Despite empirical gains, the architectural novelty is incremental—main components like VAE regularization, LoRA fine-tuning, and perceptual loss design are all based on existing paradigms.\n* The contribution may be limited by reliance on a strong pretraining backbone (DINOv2), which may obscure whether improvements stem from the framework or the visual prior itself.\n* Most components (e.g., PDPS, use of DINOv2 for semantic alignment, LoRA for tuning) are adaptations of known techniques rather than fundamental innovations.\n* The proposed HF-LoRA modules do not offer architectural novelty beyond standard LoRA tuning, and are primarily differentiated by their loss functions and placement.\n\nQUESTIONS:\n1. The proposed HF-LoRA tuning strategy uses alternating optimization of encoder (FHF) and decoder (PHF) modules. This is an interesting design, but the supplementary does not report comparisons with joint tuning. It is important to know whether the alternating scheme is empirically superior, or simply a safeguard against interference. Please provide evidence or reasoning for the alternating scheme's effectiveness. For example, does joint tuning degrade latent structure or reduce performance?\n\n2. Several key components—DINOv2 for latent alignment, LoRA for efficient fine-tuning—are derived from prior work. While the integration is novel, it is important to clarify how much of the gain stems from this architecture versus the strength of the DINOv2 prior and the generality of LoRA. Have you conducted experiments with weaker backbones (e.g., vanilla ViT or ConvNeXt) or without the LoRA separation? Additional baselines in the supplementary would help isolate the contribution of the proposed synergy.\n\n3. The progressive degradation perturbation strategy is a central component of Stage 1, yet its exact scheduling policy, degradation diversity, and sensitivity to design choices remain under-explained. While the supplementary material shows some degradation types used, it is unclear how critical the progression (vs. random sampling) is to the learned latent space. Please clarify the rationale behind the PDPS schedule and whether ablation against simpler augmentation strategies (e.g., single-step or random mix) has been performed. This would help disentangle how much benefit comes from PDPS itself versus general data augmentation.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a high-resolution image restoration method called Latent Harmony. It aims to address a issue: how to ensure computational efficiency while retaining details when inpainting ultra high resolution images such as 4K. To achieve this goal, the paper proposes a two-stage approach to handle the existing defects of VAEs, optimizing the VAEs by introducing an architecture sensitive to high-frequency details, followed by fine-tuning through a LoRA (Low-Rank Adaptation) module guided by high-frequency information.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The proposed method has a relatively lightweight computational complexity and parameter count compared to similar approaches, making it suitable for handling high-resolution content.  \n- The proposed modules enable the inpainting model to adapt to various types of degradation.  \n- Experiments show that the method can achieve superior results in metrics such as PSNR and SSIM.  \n\nWeakness\n- Figure 4 is best accompanied by a comparison of more detailed regions, such as local textures.\n- It is recommended to give a separate comparison for the restoration of text-type content in Figure 4.\n\nQUESTIONS:\n- In Section 3, the authors conduct extensive analysis of the VAE. What is the configuration of the VAE with stronger reconstruction capability? What are the design and training differences between the two baseline models? Can it be proven that low-frequency or high-frequency information in the low-dimensional latent space is strongly correlated with the low/high-frequency information in the corresponding model results in the pixel domain?  \n- Since this paper aims to solve high-resolution image restoration (e.g., UHD resolution), why are the metric calculations in Tables 1 and 2 only applicable to 256×256 resolution?\n- Different restoration tasks have varying dependencies on low/high frequencies, but the method's design still focuses on recovering high-frequency information. Intuitively, its generalization to multi-type degradation tasks relies on Equation 3, which is equivalent to moving the so-called \"second-order degradation\" from the image domain to the latent space. Has the method made specific designs for low-frequency selection or frequency adaptation, or analyzed the effectiveness of each proposed stage for each task?\n\nLIMITATIONS:\nThe limitation of proposed method is discussed in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Latent Harmony, a two-stage unified framework for UHD image restoration that addresses key limitations of Variational Autoencoders (VAEs) in balancing semantic generalization and high-frequency detail reconstruction. Stage 1 develops a regularized VAE (LH-VAE) using progressive degradation perturbation, semantic alignment (via DINOv2), and latent equivariance to learn a robust and frequency-balanced latent space. Stage 2 applies a novel high-frequency-guided LoRA (HF-LoRA) mechanism, separating fidelity- and perception-oriented fine-tuning paths for the encoder and decoder, respectively.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: This work presents a  VAE-based framework for UHD image restoration. The authors first justify their motivation through statistical analysis, then detail the loss function and network architecture. Extensive experiments validate each component, demonstrating a complete methodology.\n\nClarity: UHD image restoration combined with VAEs is natural and well-motivated, and the paper presents a clearly structured framework. To enhance readability, the dimensions of certain figures (e.g., Figures 1 and 4) could be further optimized.\n\nSignificance: The study constitutes a valuable engineering investigation of VAE-based approaches, although many design elements (Latent Space Representation and LoRA) follow an incremental improvement paradigm.\n\nOriginality:  Its novelty remains limited as it fails to provide new fundamental insights for UHD image restoration. The observed generalization capabilities appear to derive primarily from the inherent properties of VAEs.\n\nQUESTIONS:\n1. While the two-stage design is central, the interplay between the Stage 1 VAE latent structure and Stage 2 LoRA tuning (especially how much the LoRA fine-tuning modifies or respects the original latent semantics) is not clearly quantified.  Please provide quantitative or visual analysis (e.g., latent feature similarity, latent traversals) to show how the LoRA updates preserve or adapt the semantic structure from Stage 2.\n\n2. In Section 5.1, the authors claim that their method \"effectively removes degradations while preserving intricate background textures.\" However, the qualitative results in Figure 4 do not convincingly support this claim. In fact: The visual improvements over other baselines (e.g., HAIR, UHDprocesser) appear marginal at best;  Some restored outputs from your method still exhibit blurring, noise, or residual artifacts, especially in complex background regions;\n\n3.In Tables 1 and 2, only one other UHD-capable method (UHDprocesser ) is included for direct 4K full-size evaluation. However, recent works such as DreamUHD, and ERR have demonstrated UHD restoration capabilities, though perhaps indirectly or under modified setups.\n\nLiu, Yidi, et al. \"DreamUHD: Frequency Enhanced Variational Autoencoder for Ultra-High-Definition Image Restoration.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 6. 2025.\nZhao, Chen, et al. \"From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n4.Could you explain how Param and FLOPs are decreased in Table 5(c)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}