Enhancing Images with Coupled Low-Resolution and Ultra-Dark Degradations: A Tri-level Learning Framework

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 11:53:51

Abstract: Due to device constraints and lighting conditions, captured images frequently exhibit coupled low-resolution and ultra-dark  degradations. Enhancing the visibility and resolution of ultra-dark images simultaneously is crucial for practical applications.  Current approaches often address both tasks in isolation or through simplistic cascading strategies, while also relying heavily on empirical and manually designed composite loss constraints, which inevitably results in compromised training efficacy, increased artifacts, and diminished detail fidelity.  To address these issues, we propose TriCo, the first to adopt a Tri-level learning framework that explicitly formulates the bidirectional Cooperative relationship and devises algorithms to tackle coupled degradation factors.  In the optimization across Upper (U)-Middle (M)-Lower (L) levels, we model the synergistic dependencies between illumination learning and  super-resolution tasks  within the M-L levels. Moving to the  U-M levels, we introduce hyper-variables to automate the learning of beneficial constraints for both learning tasks, moving beyond the traditional trial-and-error pitfalls of the learning process. Algorithmically, we establish a Phased Gradient-Response (PGR) algorithm as our training mechanism, which facilitates a dynamic, inter-variable gradient feedback and ensures efficient and rapid convergence.  Moreover, we present the Integrated Hybrid Expert Modulator (IHEM), which merges inherent illumination priors with universal semantic model features to adaptively guide pixel-level high-frequency detail recovery. Extensive experimentation validates the framework's broad generalizability across challenging ultra-dark scenarios, outperforming current state-of-the-art methods across 4 real and synthetic benchmark datasets over 8 metrics (e.g., 5.8\%$\uparrow$ in PSNR, 26.6\%$\uparrow$ in LPIPS, and 13.9\%$\uparrow$ in RMSE).

Main paper: 
 related work: 
 Enhancing Low-Light Images. LLE's goal is to make images engulfed in darkness visible. Early works generally concentrated on leveraging handcrafted priors and empirical insights for LLE, such as Retinex model [13, 17] for separate treatment of illumination and reflection. Recent advancements have been seen with models based on convolutional neural networks, addressing these fundamental challenges [9, 26, 28] . Typically, such techniques always rely on manually selecting complex losses. Instead, we introduce the trilevel automated strategy to pinpoint beneficial constraint feedback, diverging from the reliance on empirical hyperparameter tuning.Normal/Low-light Image Super-Resolution. Normal-light SR task generates high resolution images from low resolution inputs under standard lighting conditions. Recently, a large number of methods based on convolutional neural networks have emerged to continuously refresh the performance [11, 21, 42] . With the growing popularity of transformer-based technologies, many leading-edge methods [3, 4, 22, 53] have been developed for super-resolution enhancement, including SwinIR [22] , Restormer [46] , the recently proposed SRFormer [53] and HAT [3] . Drawing on this, we fuse semantic cues from universal models [15, 48] with illumination attributes for modulation, meticulously steering the detail restoration of reflectance features and maintaining color consistency.Moreover, recent forays into super-resolution focused on lowlight imagery, have yet to yield satisfactory results [5, 10, 32] . For instance, Cheng et al. [5] proposed a light-guided and cross-fusion U-Net, featuring an intensity estimation unit, targets uneven-light image super-resolution. Yet, its sole reliance on pixel shuffling for resolution enlargement introduces notable color distortion and a lack of clarity in structure. A potential reason is that previous approaches did not account for the coupled collaboration between the two tasks, treating them in isolation. Hence, we employ hierarchical optimization to model and solve this multi-tiered coupled task.Bi-level Optimization. Bi-level Optimization is the hierarchical mathematical program, where the feasible region of upper-level task is restricted by the solution set mapping of lower-level task and the two task are mutually reinforced [16] . Subsequently, the bi-level optimization framework has been investigated in view of many important applications in the fields of machine learning and computer vision e.g., hyper-parameter optimization [12, 29] , multi-task and meta learning [29, 35] , neural architecture search [45, 54] , and image processing and analysis [24, 25] . Motivated by the above observations, We explicitly consider the collaborative relationship between super-resolution and brightness adjustment tasks, constructing a novel perspective with tri-level optimization for modeling and solving.  
methodology: 
 Our TriCo aims to transform extremely dim, low-resolution images x 𝑙𝑙 into luminance-friendly, super-resolution counterparts y 𝑛𝑠 . In the following, we firstly delve into the architecture of our model, followed by the details of the proposed learning strategy.Acknowledging the consensus that direct enlarging of dark images can result in the loss of details, noise amplification, and artifacts, we meticulously crafted an illumination-guided integrated network. This network does not merely cascade brightness adjustment with resolution enhancement modules; instead, it adopts a more profound integration approach to ensure a holistic improvement. As depicted in Fig. 2 , our network adopts a top-down integration approach, seamlessly incorporating an illumination adjustment subnetwork for initial brightness enhancement and a feature-level refinement sub-network for resolution upscaling. A pivotal bridging component (i.e., Integrated Hybrid Expert Modulator, IHEM) is introduced within this structure, leveraging illumination and semantic priors as cues for guided refinement. This architecture is summarized into two phases: (i) the Learning Interpolated Illuminance (LII), focusing on adjusting the initial luminance, and (ii) the Learning Feature Refinement (LFR) for super-resolution, dedicated to enhancing and enlarging the image details at the feature level.From LII to LFR. LII performs the initial mapping from "lowlight" to "normal-light". Based on Retinex theory, the normalized illumination map satisfies the inequality within the dynamic range: 0 ≤ x 𝑙𝑙 ≤ u ≤ 𝐼 . Thus, LII is designed to construct an interpolation mapping to estimate the illumination map u. Finally, the initial reflectance map v is obtained by applying element-wise division to u. The initial reflectance map v is then input to the LFR sub-network (i.e., 𝑁 𝑓 𝑟 ) for fine-grained feature modulation, ensuring that the upsampling process generates more high-frequency details:EQUATIONwhere the interpolation factors 𝜶 and 𝜷 are generated by the underlying Unet-style illumination regulator 𝑁 𝑖𝑟 and satisfy the constraints within the unit interval, with their sum equaling 1. 𝝎 𝒊 and 𝝎 𝒔 are network parameters of 𝑁 𝑖𝑟 and 𝑁 𝑓 𝑟 , respectively. Finally, we introduce a dynamic grid up-sampling module [7] to enlarge the image dimensions. For specific architectural details of 𝑁 𝑖𝑟 and 𝑁 𝑓 𝑟 , please refer to the Supplementary Material. The uniqueness of LII lies in its reliance solely on a single luminance loss1 for unsupervised learning, eliminating cumbersome training with multiple stages and losses. Also, we feed v into a pre-trained large-scale base semantic model (i.e., SAM [15, 48] ) to extract multi-layer semantic features:𝑁 𝑠𝑎𝑚 [v|Θ * 𝑠𝑎𝑚 ] = [𝑓 [1] 𝑠 , • • • , 𝑓 [𝑜 ] 𝑠 , • • • , 𝑓 [ 𝐽 ]𝑠 ], 𝑜 = 0, • • • , 𝐽 . We note that the multi-scale illuminance features and semantic features can serve as expert cues containing degradation priors (i.e., exposure and color information of different local areas). Therefore, we design the Integrated Hybrid Expert Modulator (IHEM) to modulate reflectance features layer-by-layer within the LFR sub-network's decoder, guiding the generation of high-frequency texture details.IHEM. As illustrated in Fig. 3 , the structural details of the IHEM are showcased. Denote each layer's reflectance feature in the decoder as 𝑓 [𝑜 ] 𝑟 is subsequently updated via the transposed Semantic-Induced Response (S-InR, 𝑊 S-InR ):EQUATIONwhere 𝜏 1 represents a learnable scaling factor that adjusts the magnitude of the product of K𝑟 and Q𝑠 . ⊗ denotes the element-wise Similarly, we output multi-layer illumination features 𝑓 EQUATIONwhere 𝜏 2 is a learnable scaling factor. Subsequently, 𝑊 I-InR is processed by 𝐹 𝐹 𝑁 , yielding the doubly modulated reflection feature f 𝑟 [𝑜 ] = 𝐹 𝐹 𝑁 𝑊 I-InR . EQUATIONwhere P 𝑙 (•) denotes the solution set, with D 𝑙𝑙 and D 𝑢𝑙 representing the lower and upperlevel datasets, respectively. The hierarchical formulation explicitly delineates the collaborative training modality between 𝑁 𝑖𝑟 and 𝑁 𝑓 𝑟 . This collaboration is heavily contingent upon the judicious selection of lower and upper level objectives, ensuring that the sub-networks can reciprocally foster enhancement and positive feedback. This raises a pivotal question: How can we automate the assignment of high hyperparameters that significantly foster positive influences on the learning tasks? Delving deeper into this inquiry, we transcend the confines of the hierarchical optimization framework and venture into an expanded horizon-establishing a nested optimization problem that encompasses both lower and upper levels, aimed at the autonomous learning of beneficial constraints for two learning tasks.Tri-level Constraint Modeling. Evidently, to automate the determination of constraints that significantly influence the learning tasks through weight allocation, we introduce a novel concept, the hyper-variable 𝝇. This hyper-variable, along with the two preceding variables 𝝎 𝒊 and 𝝎 𝒔 , forms a new set of constraint relationships, thereby constituting a hierarchical optimization problem based on where 𝝎 * 𝒊 and 𝝎 * 𝒔 represent the best-reponse for a given 𝝇. These three variables are highly interdependent and dynamically influence each other throughout the training process. This modeling approach offers significant advantages: firstly, it explicitly defines the mathematical relationships between multiple variables, allowing for dynamic feedback during training, thereby enhancing training efficiency. Secondly, it automates the determination of constraints' positive feedback, overcoming the reliance on manual hyper-parameter tuning based on empirical knowledge, thereby reducing the need for extensive manual intervention.Moving forward, we devise a Phased Gradient-Response (PGR) algorithm that iterates from the upper to the lower layers, serving as the training strategy. Specifically, we define a comprehensive function as the weighted sum of multiple losses related to the hypervariables 𝝇, addressing various specific attributes (e.g., brightness, color, exposure, smoothness, and content) pertinent to multi degradation restoration tasks. The total loss is defined as follows:L 𝑡𝑜𝑡𝑎𝑙 (𝝇, 𝝎 𝒊 , 𝝎 𝒔 ; {D}) = 𝑁 ∑︁ 𝑢=1 𝝇 𝑢 • L 𝑢 (𝝇, 𝝎 𝒊 , 𝝎 𝒔 ), L 𝑢 ∈ T, (6)where the hyper-variable is denoted as 𝝇 := {𝝇 𝑢 } 𝑁 𝑢=1 ∈ R 𝑁 . T represents the loss selection set. Please refer to Sec. 3.4 for T. To prevent ambiguous solutions during training and safeguard against overfitting, we introduce a regularization constraint term for 𝝇, thereby reformulating the total loss as:L 𝑡𝑜𝑡𝑎𝑙 (𝝇, 𝝎 𝒊 , 𝝎 𝒔 ; {D}) = 𝑁 𝑢=1 1 2𝝇 𝑢 • L 𝑢 (𝝇, 𝝎 𝒊 , 𝝎 𝒔 ) + ln(1 + 𝝇 2 𝑢 ).The training set is divided into proportions denoted by 𝜂, thus the upper and lower levels are abstractly defined as: Φ 𝑢𝑙 := L 𝑡𝑜𝑡𝑎𝑙 (𝝇, 𝝎 𝒊 , 𝝎 𝒔 ; D 𝑢𝑙 ), Φ 𝑙𝑙 := L 𝑡𝑜𝑡𝑎𝑙 (𝝇, 𝝎 𝒊 , 𝝎 𝒔 ; D 𝑙𝑙 ). Next, we decompose the problem into two stages of hierarchical optimization to solve the tri-level coupled problem step by step. B 𝜔 𝑠 ← 1 2𝜆 ( 𝜕Φ 𝑙𝑙 (𝝎 𝒊 ,𝝎 + 𝒔 ) 𝜕𝜔 𝑖 - 𝜕Φ 𝑙𝑙 (𝝎 𝒊 ,𝝎 - 𝒔 ) 𝜕𝜔 𝑖 ) 15: 𝜔 𝑖 ← 𝜔 𝑖 -𝒐 𝒖 𝜕Φ 𝑢𝑙 (𝝎 𝒊 , ω ) 𝜕𝜔 𝑖 + 𝒐 𝒍 B 𝜔 𝑠 16:% % lower-level variable probe:17:𝝎 𝒔 ← 𝝎 𝒔 -𝒐 𝒍 𝜕Φ 𝑙𝑙 (𝝎 𝒊 ,𝝎 𝒔 ) 𝜕𝝎 𝒔18: end while Gradient-Response Algorithm. Following the first-order gradient algorithm based on hierarchical optimization [24] , we compute the composite upper gradients based on the best-response from the lower optimization. We first calculate the upper-level gradient:EQUATION)For simplicity, we define the lower-level variables as 𝝎 := {𝝎 𝒊 , 𝝎 𝒔 }.The second term, the coupled gradient, is denoted as A 𝜔 . Subsequently, based on a single-step gradient descent to approximate the best-response, we calculate the finite difference approximation [23] for the coupled gradient A 𝜔 asA 𝜔 = 1 2𝜆 ( 𝜕Φ 𝑙𝑙 (𝝇,𝝎 + ) 𝜕𝝇 - 𝜕Φ 𝑙𝑙 (𝝇,𝝎 -) 𝜕𝜍 ),where𝝎 ± ← 𝝎 ± 𝜆 𝜕Φ 𝑙𝑙 (𝝇,𝝎 ) 𝜕𝝎, and 𝜆 denotes a constant learning rate. For the second phase, a similar derivation to the first phase is implemented. Given the optimal hyper-variable 𝝇 * obtained from the first stage, we compute the upper-level gradient with respect to the variable 𝝎 𝒊 :∇ 𝝎 𝒊 Φ 𝑢𝑙 𝝎 𝒊 , 𝝎 𝒔 = 𝜕Φ 𝑢𝑙 𝝇 * 𝝎 𝒊 ,𝝎 * 𝒔 (𝝎 𝒊 ) 𝜕𝝎 𝒊 + B 𝜔 𝑠 , where B 𝜔 𝑠 = 𝜕Φ 𝑢𝑙 𝝇 * 𝝎 𝒊 ,𝝎 * 𝒔 (𝝎 𝒊 ) 𝜕𝝎 𝒔 ∇ 𝝎 𝒊 𝝎 * 𝒔 (𝝎 𝒊 ).Ultimately, the optimization process across both stages is amalgamated to form our training strategy, which is summarized in Alg. 1.As illustrated in Fig. 2 , we propose a set of five specific loss objectives constituting a candidate space that encapsulates the model's constraints on brightness, color, exposure, smoothness, and content attributes, denoted as the set T, as follows:• Self-regularized luminance loss: To ensure that the generated reflectance aligns with the luminance attributes of largescale natural ImageNet dataset [8] in a consistent distribution, we design L 𝑠𝑟𝑙 :EQUATIONwhere v𝑐 signifies the operation of computing the mean across channels. Channel means and standard deviations are 𝝁 𝑐 = [0.EQUATIONwhere ℎ, 𝑤, 𝑐 are the height, width, and channel count. • Semantic perceptual loss: We utilize a perceptual loss function to maintain semantic congruence between y 𝒏𝒔 and z 𝒏𝒉 :EQUATIONwhere 𝑗 indicates the j-th feature extraction layer, which includes layers from conv1, • • • , conv5. • Structural similarity loss: We employ the SSIM loss L 𝑠𝑠𝑖𝑚 to maintain the structural similarity between y 𝒏𝒔 and y 𝒏𝒉 . • Smoothness loss: We incorporate a total variation metric [33] to reduce noise and enhance image smoothness:EQUATIONwhere 𝜋 = {𝑅, 𝐺, 𝐵}, ∇ ℎ and ∇ 𝑣 are the horizontal and vertical gradient operators, respectively. 
experiments 4.1 experimental settings: 
 Datasets and Metrics. We evaluated the benchmark performance of all compared methods across four datasets: 1) RELLISUR [1]2 , 2) DarkFace [44] 3 , 3) Dark-Zurich [34] , and 4) Cityscapes [6] . Due to space constraints, please refer to the Supplementary Material for details on the data preparation of four datasets. In the evaluation phase, we employ five full-reference metrics to assess the performance, namely PSNR [2] , SSIM [40] , Learned Perceptual Image Patch Similarity (LPIPS) [50] , Root Mean Square Error (RMSE), Feature-based Similarity Index (FSIM) [49] . Additionally, we introduce three no-reference assessments, namely Natural Image Quality Evaluator (NIQE) [31] , Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [30] and MetaIQA [55] , to evaluate non-paired metrics. Implementation Details. We adhere to the tri-level learning strategy as outlined in Alg. 1 for our network training, with the total number of iterations set to 150,000. We utilize the Adam optimizer with beta values configured at [0.9, 0.999]. The initial learning rates for the upper and lower layers of the two stages are set to 𝜸 𝒖 = 1𝑒-4, 𝜸 𝒍 = 2𝑒-4, 𝒐 𝒖 = 1𝑒-4 and 𝒐 𝒍 = 1𝑒-4, respectively. A cosine annealing restart strategy is implemented for cyclic learning rate scheduling. The dataset {D} is partitioned into {D 𝑢𝑙 } ∪ {D 𝑙𝑙 } and at a distribution ratio of 1 : 5. Experiments are conducted Compared Methods. To substantiate the efficacy of our proposed methodology, we conduct a comprehensive comparison with a diverse array of SOTA methods in LLE and SR. Specifically, we meticulously benchmark against 3 emblematic LLE techniques, namely ZeroDCE [20] , SCI [28] , and LLFormer [37] , alongside 10 SR algorithms, which include 9 under normal lighting conditions-SRResNet [18] , RDN [51] , SRFBN [21] , PAN [52] , SwinIR [22] , MIRNet [47] , Restormer [46] , SRFormer [53] , and HAT [3] -and one dedicated to low-light scenarios, LCUN [5] . Notably, the enhancement results on the RELLISUR dataset for the sole low-light SR method, LCUN, were furnished by the authors themselves. To ensure a fair comparison, we retrain the publicly available codes of all competing methods on the training set of the RELLISUR dataset. We opt for HAT as the subsequent magnification model, cascading it with three distinct LLE methods. It is noteworthy that, given Ze-roDCE and SCI operate in an unsupervised manner, we train them on the ×1 low-light RELLISUR dataset for the initial "brightening" task. Moreover, considering HAT serves as a post-processing module for super-resolving the enhanced images, it is trained on the ×2 and ×4 normal-light RELLISUR datasets, designated as ‡ HAT, to differentiate from the original HAT configuration.Following Alg. 1, we undertake a tri-level automated training regimen to sequentially optimize the three variables. Fig. 7 illustrates the convergence trend of the outermost variable and the overall loss function throughout the iterations. During the S1 cycle in Alg. 1 (refer to steps 1 to 8), the upper-level variable {𝝇 𝑢 } 5 𝑢=1 evolves from an initial unit vector to eventually converge to [0.473, 2.177, 0.181, 0.901, 0.195]. This convergence elucidates that the constraints positively influencing the learning task, as hypothesized by our algorithm, are indeed effective. It is possible to autonomously identify which constraints significantly foster a positive impetus for the learning task through adaptive weight allocation. Notably, the top three constraints-perceptual constraint, smoothness constraint, and reconstruction constraint-play a pivotal role in augmenting performance, underscoring the efficacy of our proposed approach in leveraging these constraints for enhanced learning outcomes.Evaluation on RELLISUR. Tab. 1 presents the quantitative results for low-light super-resolution tasks at ×2 and ×4 scales on the RELLISUR dataset. While cascading strategies prove effective, the improvement is not drastic. Relative to the second-best method, our approach achieves significant enhancements across all metrics (e.g., a 5.8% increase in PSNR, a 2.3% boost in SSIM, a 21.0% leap in LPIPS, a 13.9% advancement in RMSE, and a 0.6% rise in FSIM). The substantial improvements in LPIPS and RMSE underscore our method's capability to refine textures and robustly adapt to various extreme low-light conditions. Fig. 4 showcases a visual comparison on RELLISUR for simultaneous brightness adjustment and ×2 upscaling. The majority of the compared methods suffer from significant noise and blur issues, especially observable in SwinIR, LCUN, and SRFormer. Some cascaded approaches exhibit severe color bias and insufficient brightness enhancement. In contrast, our method is capable of producing images with vivid luminance and excellent restoration of high-frequency structural details. Signal plots intuitively confirm the consistency between our method and the reference images at the pixel intensity level. Evaluation on Real Nighttime Scenarios. We assess the generalization performance of the entire benchmark suite under two real-world challenging scenarios: dimly lit urban streetscapes at night and nocturnal open highway scenes. The quantitative comparison results for both challenging scenarios are presented in Fig. 6 , respectively. On the DarkFace dataset, we computed three non-paired metrics to evaluate the quantitative scores. Particularly noteworthy is our performance on the MetaIQA metric, where we achieved a 10.5% improvement over the second-best method on DarkFace. This underscores our method's efficacy in effectively restoring a variety of degradations such as noise, blur, and underexposure. Please refer to the Supplementary Material for qualitative and quantitative comparisons on the DarkFace dataset. For comparisons on Cityscapes, see the Supplementary Material as well.Robustness across Diverse Darkness. Fig. 5 conducts a robustness analysis across varying levels of darkness. Five distinct levels of low exposure are generated by adjusting exposure time, resulting in corresponding dark images (e.g., -2.5EV, -3.0EV, -3.5EV, -4.0EV, and -4.5EV). Our method maintains consistent enhancement across various levels of darkness. This is visually corroborated by the probability density histograms, which demonstrate a uniform consistency distribution across the five different levels of darkness, highlighting the high robustness of our model to inputs under varying levels of darkness. Furthermore, it is noteworthy that the RMSE scores in Tab. 1 also underscore the significant generalization capability of our method across various darkness levels.Computational Efficiency. To evaluate model efficiency, we present the parameters, FLOPs, inference time, and FPS of compared SOTA methods in Tab. 2. The evaluations are performed on a single 2080 Ti GPU using images of size 128 × 128. Excluding the parameters of the frozen SAM model, our network has a parameter count of less than 1.4MB. In conclusion, our network achieves a favorable balance between performance and efficiency.Effectiveness of IHEM. When the IHEM module is removed, as seen in Config.3 of Tab. 3, there is a noticeable performance drop-approximately 2.2% in PSNR and 2.4% in SSIM-compared to the optimal model, Config.0. Fig. 9 presents the ablation results with the feature visualizations facilitated by IHEM. We visualized the features before and after the IHEM process in the last three decoding layers. Upon comparison, the features prior to IHEM appear more sparse and scattered, with a distinct lack of textural detail (see the discernible regions within the dashed circles: the car wheels). Post-IHEM features, however, exhibit a more abstract and semantically rich visual representation. This indicates that IHEM fosters a greater focus on capturing higher-level semantics.Analysis of the Solution Algorithm. We conduct the ablation study to quantify the impact of the proposed algorithm components, with comparative results detailed in Tab. 3, from Config.0 to Config.3. Omitting the S1 strategy alone leads to a performance degradation of approximately 2% in PSNR and 2.2% in SSIM compared to the best-performing model, Config.0. Similarly, removing only the S2 strategy results in a reduction of about 0.9% in PSNR and 1.2% in SSIM relative to Config.0. This delineation underscores the critical importance of synergistically integrating S1 and S2 strategies to achieve the superior performance set forth by Config.0. Due to space constraints, Supplementary Materials include ablation studies (i.e., LII, LFR, loss functions, etc.). 
conclusion and remarks: 
 This investigation delves into the intricate realm of brightening and magnifying ultra-dark images, a pursuit fraught with practical complexities due to the dual dilemmas of low resolution and profound darkness. Our tailored TriCo, adopts a tri-level learning strategy that intertwines the tasks of illumination enhancement and super-resolution. By fostering the collaborative learning, TriCo effectively negates the historical deficiencies of isolated or simplistic task handling, yielding superior clarity and artifact reduction.Broader Impacts. TriCo's strategic innovation extends beyond the ultra-dark super-resolution challenge, advocating for a broader investigation into joint low-level visual and high-level semantic tasks under adverse conditions, which can elevate the development of effective training strategies for a range of coupled tasks. 
: 
 and SR methods [3, 53] applied both independently and in a cascaded manner to inputs with coupled degradations.autonomous driving [19, 27, 36, 38] . Due to inherent limitations in imaging devices and constraints posed by environmental lighting conditions, captured data frequently exhibits coupled degradations characterized by low resolution and extreme darkness. Imaging devices may struggle to capture clear details in poorly lit environments, resulting in images of low resolution; concurrently, insufficient ambient lighting exacerbates the darkness of the images, making content difficult to discern. This paper addresses the integrated challenge of enhancing brightness and increasing resolution in ultra-dark images plagued by these intertwined degradations.Capturing images in ultra-low-light settings introduces a plethora of challenges that amplify the complexity of this joint task, including uneven exposure resulting in highly irregular lighting, diminished contrast, color inaccuracies, and an overflow of artifacts. Standard image Super-Resolution (SR) techniques [11, 21, 39, 41] , crafted with modular techniques for image resolution enhancement under normal-lighting scenarios, cannot be straightforwardly adapted to enhance the luminance and resolution of images captured in lowlight conditions. Indeed, the direct application of these techniques would inevitably amplify hidden noise, blur, and artifacts present in darkness, leading to unnatural edges and textures and deviating from the primary goal of super-resolution. Contrarily, recent Low-Light Enhancement (LLE) methods [19, 26, 43] , while capable of brightening, fall short in concurrently amplifying resolution and authentically enhancing high-frequency details. This prompts a further inquiry: Can LLE and SR be effectively combined in a simple "A+B" cascaded format to achieve the desired outcomes? Upon evaluation, we ascertain that this direct "A+B" does not address the entangled degradation factors at the data level, possibly akin to a "A×B" degraded form, with ongoing shortcomings in enhancing brightness and rendering texture details.As illustrated in Fig. 1 , we present a visual comparison of two cutting-edge LLE methods -SCI [28] and LLFormer [37] -alongside two normal-light SR techniques, HAT [3] and SRFormer [53] .A closer examination reveals that employing HAT and SRFormer independently falls short in recapturing fine details, producing blurred artifacts. Similarly, cascaded LLE⇒SR approaches (e.g., SCI⇒HAT, LLFormer⇒HAT) also fail to restore adequate brightness, exacerbating noise and structural distortions when magnified. In stark contrast, our proposed method generates natural and authentic exposure and color fidelity, alongside improved structural clarity and texture detail. In addition to the methods previously mentioned, a few recent studies have emerged focusing on superresolution within low-light scenes [5, 10] . They tend to rely on simple brightness corrections and resolution scaling on synthetic datasets, leading to poor generalization in real-world scenarios. Thus, we summarize the two primary shortcomings limiting the efficacy of existing methods: (i) The failure to recognize the intricacies of degradation-coupled data, which extends beyond a simplistic additive enhancement model. This overlooks the intrinsic bidirectional cooperation necessary for joint processing. (ii) A heavy reliance on empirical network design and manual aggregation of losses. Such methods disregard the guiding principles of physical image formation, and overlook the crucial role that suitably chosen loss constraints play in facilitating cooperative learning between intertwined tasks.Stemming from these insights, this paper seeks to explore a tri-level optimization perspective that formulates the cooperative relationships and devises a corresponding solution strategy. We propose TriCo, aiming to automate the optimization of these weighted constraints with hyper-variable and the coupling dependencies between two entangled tasks, striving to achieve a unified enhancement. Specifically, we initiate the process with an illumination interpolation mapping inspired by Retinex theory, yielding a brightened reflectance that serves as the foundation for subsequent featurelevel super-resolution enhancement. We leverage universal foundational semantic model priors and illumination features under the self-regularized luminance constraint to provide dual guidance for the super-resolution process, specifically targeting the compensation of high-frequency details. On the algorithmic front, we have crafted a phased gradient-response algorithm as our training mechanism, meticulously designed to synergize the optimization of three key variables while offering dynamic gradient feedback throughout the training phase, thereby ensuring streamlined training efficiency and rapid convergence. In summary, our contributions are fourfold:• We propose TriCo, the first to introduce a Tri-level optimization perspective that explicitly models the bidirectional Cooperative relationship of illumination learning and superresolution, formulating a solution to synergistically brighten and enlarge images afflicted with coupled low-resolution and ultra-dark degradations. • We establish an Upper (U)-Middle (M)-Lower (L) level nested formulation, which in its M-L level, explicitly delineates the collaborative dependency of two entangled tasks. In the U-M level, we integrate hyper-variables to autonomously enforce positive constraint feedback, thus dismantling the reliance on manual trial-and-error intervention. • We propose a Phased Gradient-Response (PGR) algorithm as the training mechanism, designed to synergistically optimize three variables while providing dynamic gradient feedback, thus achieving efficient training and rapid convergence.• We propose an Integrated Hybrid Expert Modulator (IHEM) that acts as a conduit between illumination prior cues (i.e., intrinsic attributes) and generic semantic model features (i.e., SAM), facilitating an adaptive pixel-level guidance for the restoration of high-frequency details.Extensive experimentation validates the framework's broad generalizability and performance advantages across 4 real and synthetic benchmark datasets over 8 metrics (e.g., 5.8%↑ in PSNR, 26.6%↑ in LPIPS, and 13.9%↑ in RMSE). 
 
 Appendix: Please refer to the self-regularized luminance loss in Eq. (8).https://vap.aau.dk/rellisur/https://flyywh.github.io/CVPRW2019LowLight/ 


Decision: This submission has been reviewed by three experts, the final ratings from whom are all positive. Considering the quality of the submission, the comments from the reviewers, and the rebuttal from the authors, the paper can be accepted by the conference.

Reviews: review: paper_summary: paper_summary: Captured images often suffer from low-resolution and ultra-dark degradation due to device and lighting constraints. Enhancing visibility and resolution simultaneously is crucial, but current methods typically address these tasks separately or through simplistic cascading approaches. They rely heavily on manual loss constraints, leading to compromised efficacy, increased artifacts, and diminished detail fidelity. To overcome these challenges, the authors introduce TriCo, the first Tri-level learning framework. It explicitly considers the bidirectional cooperative relationship between tasks and devises algorithms to address coupled degradation factors. TriCo optimizes across Upper (U)-Middle (M)-Lower (L) levels, modeling synergistic dependencies between illumination learning and super-resolution tasks. Hyper-variables are introduced in the U-M levels to automate constraint learning, reducing reliance on trial-and-error. We employ a Phased Gradient-Response (PGR) algorithm for training, enabling dynamic gradient feedback and rapid convergence. Additionally, the Integrated Hybrid Expert Modulator (IHEM) combines illumination priors with semantic features to guide high-frequency detail recovery at the pixel level.
strength_weakness: strength_weakness: The paper proposes a novel approach to deal with coupled low-resolution and ultra-dark degradations.

The reported results show competitive performance.
limitations: limitations: 1.	It is better to add more descriptions in Fig. 3, Fig. 8 and Fig. 9, enabling readers to grasp the content without having to refer to the main text.
2.	The proposed Illumination-Induced Attention is similar with Illumination-Guided Multi-head Self-Attention in Retinexformer.
3.	From the results shown in Figure 6, it appears that the method proposed in this paper does not handle the highlighted areas very well.
4.	The ablation study section lacks in-depth analysis. In addition, it is necessary to include ablation experiments regarding the loss function.

[1] Retinexformer: One-stage retinex-based transformer for low-light image enhancement, CVPR 2023.
paper_summary: paper_summary: This paper proposes a new method, namely TriCo, for image joint low-light enhancement and super-resolution. Three technical contributions are claimed.

(i) An Upper (U)-Middle (M)-Lower (L) level nested formulation. The M-L level explicitly delineates the collaborative dependency of two entangled tasks. The U-M level integrates hyper-variables to autonomously enforce positive constraint feedback, thus dismantling the reliance on manual trial-and-error intervention.

(ii) A Phased Gradient-Response (PGR) algorithm as the training mechanism, designed to synergistically optimize
three variables while providing dynamic gradient feedback, thus achieving efficient training and rapid convergence.

(iii) An Integrated Hybrid Expert Modulator (IHEM) that acts as a conduit between illumination prior cues (i.e., intrinsic attributes) and generic semantic model features (i.e., SAM)
strength_weakness: strength_weakness: (i) The writing is clear, good, and easy to follow, especially the mathematical notations in the method part. The presentation is well-addressed. I like the style of the figures in this paper.

(ii) The performance seems solid. The improvement of the proposed TriCo over the state-of-the-art method is about 1 dB. This is very significant. The visual results also demonstrate that the proposed TriCo can reconstruct more high-frequency details and structural contents.

(iii) The ablation study part is also very sufficient to validate the effectiveness of the proposed techniques.
limitations: limitations: (i) The main results are not very convincing. The authors claim that their methods are very effective in high-resolution low-light enhancement. But some important experiments are missing. For example, it would be better if the results on the NTIRE 2024 datasets were provided. The NTIRE 2024 provides 6Kx4K images to evaluate high-resolution low-light enhancement. Also, what about the performance on LOL datasets?

(ii) Many recent highly related works are missing. for example, there are no comparisons with ICCV 2023 works? e.g., Retinexformer [1]?

[1] Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement. In ICCV 2023.

(iii) What are the advantages of joint super-resolution and low-light enhancement? I mean, we can do these two tasks by two methods, one for super-resolution and one for low-light enhancement, right? More explanation about the motivation should be given.
paper_summary: paper_summary: The paper introduces TriCo, a Tri-level learning framework for improving visibility and resolution in ultra-dark images. It overcomes current limitations by using a tri-level learning approach, hyper-variables, and some proposed blocks. TriCo outperforms existing methods, making it effective for ultra-dark scenarios.
strength_weakness: strength_weakness: 1. The experimental results are good, outperforming existing super-resolution and low-light methods. It demonstrates a robust generalization capability in real-world scenarios.
2. A novel framework is introduced that fosters a collaborative improvement of super-resolution and low-light enhancement tasks.
limitations: limitations: 1. The paper writing is hard to follow, and the naming of blocks appears overly complex. For instance, the Integrated Hybrid Expert Modulator is essentially a standard cross attention transformer block, which has been previously utilized in low-light image enhancement research[1].
2. Typically, low-light enhancement models have fewer parameters. The paper employs the large-scale SAM feature for performance improvement but lacks relevant ablation studies. It remains unclear whether the SAM feature usage has significantly contributed to performance enhancement.
3. The tri-level learning framework incorporates an adaptive balance of loss function weights. It would be interesting to explore if other adaptive methods[2] could yield similar results.

[1] Wu, Yuhui, et al. "Learning semantic-aware knowledge guidance for low-light image enhancement." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Sener, Ozan, and Vladlen Koltun. "Multi-task learning as multi-objective optimization." Advances in neural information processing systems 31 (2018).