Perceptual-Distortion Balanced Image Super-Resolution is a Multi-Objective Optimization Problem

Authors: qiwen  zhu
yanjie  wang
liqun  chen
jiahuan  zhou
sheng  zhong
shilv  cai
luxin  yan
xu  zou
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 12:51:32

Abstract: In this paper, we introduce a novel approach to single-image super-resolution (SISR) that balances perceptual quality and distortion through multi-objective optimization (MOO). Traditional pixel-based distortion metrics like PSNR and SSIM often fail to align with human perceptual quality, resulting in blurry outputs despite high scores. To address this, we propose the Multi-Objective Bayesian Optimization Super-Resolution (MOBOSR) framework, which dynamically adjusts loss weights during training. This reduces the need for manual hyperparameter tuning and lessens computational demands compared to AutoML. Our method conceptualizes the relationship between loss weights and image quality assessment (IQA) metrics as black-box objective functions, optimized to achieve an optimal perception-distortion Pareto frontier. Extensive experiments demonstrate that MOBOSR surpasses current state-of-the-art methods in both perception and distortion, significantly advancing the perception-distortion Pareto frontier. Our work lays a foundation for future exploration of the balance between perceptual quality and fidelity in image restoration tasks. Source codes and pretrained models are available at: https://github.com/ZhuKeven/MOBOSR.

Main paper: 
 introduction: 
 Super-resolution (SR) plays a crucial role in multimedia, increasingly integrated into devices to enhance the viewing experience of low-resolution images and videos, not only improve visual quality but also reducing bandwidth and storage needs for media streaming.In the single-image super-resolution (SISR) field, both model architecture and computational efficiency have received considerable attention, resulting in impressive achievements. The majority of these studies report and compare results using PSNR and SSIM [55] , both of which are pixel-based metrics. Thus, SR task is actually treated as a regression problem, using regression losses to train, such as L1 Normalization (Mean Absolute Error), L2 Normalization (Mean Squared Error), Charbonnier loss [22] , and Huber loss [23] . Pixel-based regression losses are favored for their simplicity and computational efficiency. However, as image downscaling results in significant loss of high-frequency details, SR poses an ill-posed problem. Pixel-based regression loss functions struggle with recovering these high-frequency details [26, 40] , often producing blurred images, despite achieving high PSNR and SSIM [55] scores.While PSNR or SSIM [55] are convenient for computation and comparison, they do not adequately reflect the perceptual quality of images and may even have a negative correlation with human perception (see Figure 1 , 'Ours-a' has higher PSNR than 'Ours-c' while 'Ours-a' is more blurry on the contrary). The most accurate reflection of human perceptual quality is naturally human-opinionscores, but due to its irreproducibility and high cost, it is challenging to use for fair comparative analysis [4] . Consequently, Full Reference Image Quality Assessment (FR-IQA) metrics like LPIPS [58] have been developed to better align with human perception while being reproducible and computationally efficient. To enhance the perceptual quality of SR models, researchers commonly employ perceptual loss [19] or GAN [16] for training, which, while enhancing image sharpness, may also result in the introduction of noise [1] validation set. Metrics are normalized, so a higher bar indicates better performance for the corresponding metric. This indicates there is an inherent conflict between perceptual and regression losses (e.g. [L 1 , L 2 , L 𝐹 𝐹𝑇 , L ∇ , L 𝑆𝑆𝐼 𝑀 ] contribute more to distortion metrics (PSNR, SSIM [55] , and LR-PSNR), while[L 𝐿𝑃𝐼 𝑃𝑆 , L 𝜙 2 , L 𝜙 3 , L 𝜙 4 , L 𝜙 5 ] contribute more to perceptual metric (LPIPS [58])).and artifacts (see Figure 1 , 'SPSR' [31] holds better LPIPS [58] than 'Ours-b', but 'SPSR' [31] causes much more artifacts).Therefore, many researchers combine regression and perceptual losses to strive for an optimal balance between pixel accuracy and visual perception [10, 14, 15, 26, 28, 37, 40, 47, 50, 53] . However, as shown in Figure 2 , there is an inherent conflict between perceptual loss and regression losses, as well as between perceptual quality and distortion metrics [5] , leading to a compromise.Several methods attempts to balance perception and distortion by manually setting the weights of different losses during training [15, 24, 38, 41, 53] . Concurrently, some approaches involve training two structurally identical networks: one focuses on regression loss to optimize distortion metrics, while the other emphasizes perceptual loss to enhance perceptual metrics. The parameters of these networks are subsequently blended through interpolation, employing manually determined weights to finely adjust the trade-off between perception and distortion within the final model [47, 52, 53] . In a novel strategy, Wang et al. [50] incorporated conditional branches into the network, which are regulated by a scalar value to produce varying trade-offs. Furthermore, certain methods selectively apply different losses to different regions or frequency components of an image [10, 14, 26, 28, 37, 40, 51] . This entails utilizing perceptual loss or GAN for areas with intricate textures to maximize perceptual quality, while reserving pixel regression loss for regions with simpler textures to maintain fidelity. While these artworks have garnered notable success, they have not yet explored the optimal balance boundary between perception and distortion. Furthermore, they either necessitate laborious hyperparameter tuning or impose additional computational demands during inference phase.To intuitively illustrate the conflict between perception and distortion, we conducted a series of simple experiments to quantify the contributions of various loss functions to different metrics, as depicted in Figure 2 . It was observed that loss functions with a higher contribution to distortion metrics inversely contributed less to perceptual metrics, and vice versa. This conflicting characteristic prompted us to consider: Whether Multi-Objective Optimization (MOO) could be a viable solution to this dilemma?MOO is an optimization strategy designed to simultaneously address multiple objectives. Due to potential contradictions among these objectives, it is often challenging to find a single solution that maximizes or minimizes all objectives concurrently. Therefore, the aim of MOO is to identify a set of solutions that offer the best compromise among the different objectives (or the so-called optimal balance boundary), culminating in what is known as the Pareto frontier. On this frontier, any improvement in one objective comes at the cost of compromising at least one other objective. Multi-Objective Bayesian Optimization (MOBO), a widely utilized algorithm within the MOO framework, is particularly adept at optimizing black-box objective functions that are gradient-inaccessible and costly to evaluate.This paper introduces a novel application of MOBO to address the challenge of balancing perceptual quality and distortion in SISR models. We conceptualize the relationship between loss weights and IQA metrics as black-box objective functions to be optimized. In the training process, the performance metrics of SR models guide the optimization algorithm to refine the loss weights for the next epoch, thereby creating a feedback loop that enhances model efficacy with each iteration. This approach circumvents the computational burden associated with AutoML, where each optimization iteration requires a full model training, and does not incur any additional computational or storage costs during inference.In summary, the contributions of this work are threefold:• We pioneer the application of MOBO to address the challenge of balancing perception quality and distortion in SISR models.  
related works: 
 Our work employs MOBO to address the balance between perception and distortion in SISR models. The following sections will provide an overview of SISR methods aimed at addressing this balance, followed by a concise introduction to MOO methods.Since Dong et al. [11] first applied CNN to the task of SISR, this field has been predominantly driven by deep learning advancements. SISR models have continuously improved in performance. Notably, ESPCN [42] has replaced interpolation with sub-pixel convolution and deferred scaling to the end of the model, significantly reducing computational demands. EDSR [27] has achieved enhanced performance by omitting Batch Normalization [18] layers, which, despite their effectiveness in other tasks, proved detrimental to SR tasks. Furthermore, SwinIR [25] has constructed SR models based on Swin Transformer [29] , establishing a new SOTA. These methods employ regression loss for training, achieving high PSNR and SSIM [55] scores but failing to recover high-frequency details, resulting in blurred images. Johnson et al. [19] introduced perceptual loss which utilizes feature maps extracted from a pre-trained deep convolutional network (such as VGG [43] ) as the input for the loss function, significantly enhancing the perceptual quality of super-resolution models. SR-GAN [24] pioneered the application of GAN [16] to SR models, enabling the generation of sharp details. These two seminal works have inspired a plethora of perception-oriented SR approaches. Beyond perceptual [19] and GAN [16] losses, a plethora of other loss functions have been incorporated into SISR models. A category of these introduces additional constraints, including: frequency domain constraints [6, 10, 15, 28, 44, 49] , which utilize the spectral maps generated by FFT, DCT or Wavelet transformations to compute loss; gradient constraints [6, 30, 31, 46, 47, 54] , which calculate loss using the gradient maps of images; and low-resolution consistency constraints [34, 46] , which compute loss by comparing the downsampled super-resolved images with the original lowresolution images, thereby reducing the likelihood of generating incorrect textures. The application of these losses can further enhance certain metrics of SISR models, as shown in Figure 2 .However, while perceptual loss [19] and GAN [16] can enhance perceptual quality, they also introduce noise or artifacts and may even produce incorrect textures. To balance perceptual quality and distortion in SISR, most methods involve manually adjust the weights of different loss functions [15, 24, 38, 41, 53] , necessitating extensive hyperparameter tuning to achieve superior outcomes, yet without guaranteeing optimality. Vu et al. [47] , along with Wang et al. [52, 53] , have demonstrated that a balance between perception quality and distortion can be attained through network parameter interpolation. This method mitigates the need for extensive network training for each weight by limiting the process to just two networks, thereby substantially reducing resource consumption. However, it is important to note that this approach still does not guarantee optimal results. In a similar way, Wang et al. [50] introduced conditional branches within the network architecture, modulated by a scalar parameter, to facilitate a spectrum of tradeoffs. However, this approach introduces additional computational load during inference phase.Rad et al. [40] categorized images into background, edges, and objects, applying different loss functions to each category. Fritsche et al. [14] employed separate loss functions for the high-frequency and low-frequency components of images. Liang et al. [26] divided images into numerous patches, classifying them into three categories-solid colors, texture-rich like trees or hair, and those with strong edge transitions-and applied different loss functions accordingly. Liu et al. [28] segmented images based on Spectral Bayesian Uncertainty before applying various loss functions. Park et al. [37] train an auxiliary network using the LPIPS [58] Map as supervisory signal to predict a discrete set of predefined loss weight combinations, aiming to maximum perceptual quality. These methods either fail to achieve the optimal balance, necessitate additional network training, or increase computational load during inference. Decomposing images before super-resolution and then recombining them may also potentially introduce more noise or artifacts.MOO is a process that seeks to find the optimal balance among several often conflicting objectives. If the user has a clear preference for each objective, a weighted sum approach can be employed to transform it into a single-objective optimization problem. Conversely, if the user's preferences are not well-defined, MOO techniques are necessary to determine the Pareto frontier.In the field of MOO, the two main methodologies are multiobjective evolutionary algorithms (MOEA) and multi-objective Bayesian optimization (MOBO). MOEAs, such as NSGA-II [9] , SPEA2 [59] , MOEA/D [57] , SMPSO [35] etc., are well-regarded for their ability to generate a diverse set of Pareto-optimal solutions through population-based approaches. They excel in exploring the solution space, which is crucial for capturing the trade-offs among conflicting objectives. However, MOEAs can be computationally demanding, especially when scaling to problems with high-dimensional objective spaces or when requiring a high-resolution Pareto front.MOBO approaches extends Bayesian Global Optimization (BGO) [20] from single-objective to multi-objective contexts. It typically uses Gaussian Processes (GP) for modeling the objective functions and utilizes heuristic acquisition functions, such as Expected Hypervolume Improvement (EHVI) [7, 8, 13, 48] , to identify the subsequent data point for evaluation. MOBO is highly sample efficient and is particularly well-suited for optimizing black-box objective functions which are expensive to evaluate. Thus, MOBO is especially well-suited for implementing our proposed SISR approach that balances perceptual quality and distortion. 
method 3.1 preliminary: 
 MOO refers to simultaneously optimizing multiple objective functions 𝒇 (𝒙) ⊂ R 𝑀 over a bounded search space X ∈ R 𝑑 , aiming to find solutions that optimize all objectives. However, these objectives often conflict with each other, making it challenging to find a single solution that optimizes all objectives simultaneously. Instead, we typically obtain a compromise among multiple objectives. A common approach is to identify the Pareto front. The primary goal of SISR is to employ a model, e.g. a neural network, denoted as 𝐺 𝜃 and parameterized by 𝜃 , to transform a lowresolution input image 𝐼 𝐿𝑅 , which is typically obtained through bicubic downsampling from a high-resolution image 𝐼 𝐻𝑅 , into a super-resolved image 𝐼 𝑆𝑅 with the same resolution as 𝐼 𝐻𝑅 :EQUATION)The goal is to make 𝐼 𝑆𝑅 as close as possible to 𝐼 𝐻𝑅 , and to achieve this, we minimize a loss function:EQUATIONHowever, relying solely on a single loss function may not lead to an optimal model, as different loss functions have different advantages and drawbacks. Therefore, we linearly combine multiple losses:EQUATIONThe gradient of this combined loss with respect to 𝜃 is:EQUATIONTherefore, by adjusting the weights 𝜔 𝑖 of each loss L 𝑖 , we can control the corresponding magnitude of the loss gradient and, consequently, its impact on the IQA metrics 𝑸 𝜃 = [𝑃𝑆𝑁 𝑅 𝜃 , . . . , 𝐿𝑃𝐼𝑃𝑆 𝜃 ] of SR model. There exists a deterministic relationship between 𝑸 𝜃 and loss weights 𝝎 = [𝜔 1 , . . . , 𝜔 𝑁 ]:EQUATIONSolving for 𝒇 allows us to achieve optimal balance between perceptual quality and distortion by setting the best loss weights. However, since 𝒇 is an unknown black-box function and these objectives may conflict with each other, MOBO will be employed for its estimation.Initially, we randomly and uniformly sample a certain number of initial weights within the search space 𝝎 ∈ 𝛀 𝑑 , train the SR model using the sampled weights, and then compute the IQA metrics of the SR model to obtain the observed dataset D = {(𝑸 𝑖 , 𝝎 𝑖 )} 𝑇 𝑖=1 . Subsequently, we use the observed data D to fit a probabilistic surrogate model. Surrogate models typically employ Gaussian processes (GP), which simulates the probability distribution of the true objective function 𝒇 based on the observed data D:EQUATIONwhere 𝝁 is the mean function, and 𝑲 is the covariance function. In MOBO, a GP is fitted for each objective.Given the high cost of evaluating the objective function and the low cost of evaluating the surrogate model, BO utilizes acquisition functions to heuristically select the next data point for evaluating and training, thereby minimizing the number of evaluations. Acquisition functions assess the potential performance of new data point based on the distribution predicted by the surrogate model. The most commonly used acquisition function is the Expected Improvement (EI):EQUATIONwhich can achieves a good balance between exploration and exploitation. When extended to multi-objective scenarios, the Expected Hypervolume Improvement (EHVI) is used as the acquisition function:EQUATION) Through the acquisition function, we iteratively select the next most valuable point for evaluation and update the surrogate model: By repeating these steps, BO can achieve the global optimum of the black-box objective function with a minimal number of evaluation steps. The pseudocode for the overall optimization process is outlined as Algorithm 1.EQUATION 
experiments 4.1 implementation details: 
 Model Architecture. The ESRGAN [53] architecture was employed as the SR model to evaluate the approach, due to its excellent performance and popularity in recent perception-related SR research. The widespread adoption of ESRGAN [53] and its proposed RRDB [53] backbone ensures a comprehensive and equitable comparison of our methodology.Loss Functions. The loss functions we employed include:• L1 Loss:EQUATION)• L2 Loss:EQUATION)• FFT Loss:EQUATION)• Gradient Loss:EQUATIONwhere ∇ ℎ and ∇ 𝑣 are the Sobel gradient operators in the horizontal and vertical directions. • Perceptual Loss [19] :EQUATIONTable 1 : Comparison of MOBOSR with other artworks on 7 datasets. The best and second-best results are highlighted in bold and underline, respectively. The symbols ↑ and ↓ indicate that higher or lower values of the metric are preferable. To the best of our knowledge, for fair comparisons, all publicly available methods (utilize RRDB [53] as the backbone, and aim to address the balance between perceptual quality and distortion) are selected. This includes SPSR [31] , RRDB+LDL [26] , CAL-GAN [36] and SROOE [37] . In this table, we use the data point labeled as Our-c in Figure 1 for comparison. Except for SROOE [37] , our proposed MOBOSR consistently outperforms existing methods by a large margin on all metrics, particularly demonstrating a significant advantage in LR-PSNR with over 5dB improvements. Compared to SROOE [37] , we achieve superior performance on all distortion metrics with similar LPIPS [58] , even though we only train on the smaller DIV2K [1] EQUATIONwhere ⇓ 3 is the bicubic downsampling operator. • GAN Loss [16] :The discriminator loss is:EQUATIONwhere 𝐷 represents the discriminator network. The generator loss is:EQUATION• SSIM [55] Loss:EQUATIONwhere 𝜇 𝐼 𝑆𝑅 and 𝜇 𝐼 𝐻 𝑅 are the mean values of 𝐼 𝑆𝑅 and 𝐼 𝐻𝑅 , 𝜎 𝐼 𝑆𝑅 and 𝜎 𝐼 𝐻 𝑅 are the standard deviations of 𝐼 𝑆𝑅 and 𝐼 𝐻𝑅 , 𝜎 𝐼 𝑆𝑅 𝐼 𝐻 𝑅 is the covariance between 𝐼 𝑆𝑅 and 𝐼 𝐻𝑅 , 𝐶 1 and 𝐶 2 are small constants to stabilize the division. • LPIPS [58] Loss:EQUATIONwhere 𝐹 represents the set of intermediate layers in the chosen neural network, 𝜙 𝑙 is the normalized outputs of layer 𝑙 for images, 𝜔 𝑙 are the official weights determined by Zhang et al. [58] .Datasets. The DIV2K [1] training set (800 images) was employed for training. For testing, we utilize a collection of datasets including Set5 [3] , Set14 [56] , DIV2K [1] validation set, BSD100 [32] , Urban100 [17] , General100 [12] , and Manga109 [33] . We followed the previous work [26, 31, 36, 53] to focus only on ×4 super-resolution to train and evaluate the effectiveness of our SR model. Specifically, we Training Details. We randomly sample 16 patches from each image as a batch, the patch size of LR is 32 × 32. To ensure stability in the SR model training progress and MOBO optimization process, we initially pre-train for 250 epochs using manually set loss weights from ESRGAN [53] , defined as:EQUATIONwhere[𝜔 1 , 𝜔 𝜙 5 , 𝜔 𝐺𝐴𝑁 ] = [1𝑒 -2, 1, 5𝑒 -3]. Subsequently, we commence loss weight optimization using our MOBO method. The learning rate during pre-training is fixed at 1𝑒 -4, and for optimization with MOBO, the initial learning rate is set at 5𝑒 -5 and is halved every 250 epochs. The optimizer employed is Adam [21] .Our SR model is implemented by PyTorch [39] on NVIDIA RTX 3090 GPUs, and MOBO is realized through the Ax platform [2] .Methods utilize RRDB [53] as the backbone, and aim to address the balance between perceptual quality and distortion are selected for fair comparisons. To this end, to the best of our knowledge, all five publicly available artworks are used for comparison: ESRGAN [53] , SPSR [31] , RRDB+LDL [26] , CAL-GAN [36] and SROOE [37] . All of which utilize RRDB [53] as their backbones. It is noteworthy that SROOE [37] is trained on the larger DF2K dataset (3450 images), which consists of the DIV2K [1] training set (800 images)and Flickr2K [45] (2650 images). ESRGAN [53] , on the other hand, is trained on the even larger DF2K-OST dataset (13774 images), which includes DF2K (3450 images) and OST [51] (10,324 images). Additionally, the SPSR [31] network incorporates an extra gradient branch, and SROOE [37] introduces an auxiliary network to predict a discrete set of predefined loss weight combinations. These two methods introduce additional computational load during inference. We incorporated all the loss functions introduced in Section 4.1 and optimized their weights using the MOBO strategy. Our optimization objectives were PSNR and LPIPS [58] . The loss function is formulated as:EQUATIONAnd the optimization formula is:EQUATIONAs demonstrated in Table 1 , except for SROOE [37] , our approach not only demonstrates superior performance in the full-reference perceptual metric (LPIPS [58] ) but also significantly surpasses other methods in distortion metrics (PSNR and SSIM [55] ) and consistency metric (LR-PSNR). Compared to SROOE [37] , we achieve Table 2 : Ablation study on 7 datasets compares MOBO-optimized loss weights to ESRGAN [53] manually-set loss weights. Both models use the official ESRGAN [53] train code and are trained on the DIV2K [1] training set with the same settings, except that the loss weights for MOBOSR are optimized by MOBO. The best results are highlighted in bold. MOBOSR consistently outperforms ESRGAN [53] by a large margin in the same setting. This comparison intuitively demonstrates the effectiveness of the core idea of our work: automatic loss weight optimization. superior performance on all distortion metrics with similar LPIPS [58] , even though we only train on the smaller DIV2K [1] training set. This indicates that our approach not only achieves excellent perceptual metrics but also mitigates the artifacts and false textures introduced by perceptual loss [19] and GAN [16] , thereby ensuring consistency with the LR source and the pixel accuracy with the HR image. In summary, our method consistently outperforms others in both distortion and perceptual quality, demonstrating that the integration of MOBO facilitates the attainment of the Pareto frontier (optimal balance boundary) for perception quality and distortion. Furthermore, prior to our work, manually setting such a multitude of loss weights is impractical, which consequently limits the incorporation of additional loss functions into the SISR model. While our method can efficiently and autonomously determine the optimal weights for loss functions, without introducing any additional computations during the inference and significantly extending the training duration (time consuming about SR model and MOBO during training are detailed in the supplementary material).Visual comparisons of Ours-[a,b,c] (as defined in Figure 1 ), as depicted in Figure 4 , clearly reveal the inherent contradiction between perception quality and distortion. This is evidenced by the gradual deterioration in PSNR and the corresponding improvement in LPIPS [58] on the Pareto frontier. A higher PSNR typically results in a more blurred image, albeit with reduced noise and artifacts. Conversely, a better LPIPS [58] score indicates a sharper image, which, however, tends to introduce noise and artifacts. Compared to other methods, our approach generates fewer noise and artifacts without excessively blurring the image. Additionally, it exhibits more accurate reconstruction in certain areas, better preserving the original structure of the objects in image.To demonstrate the effectiveness of using MOBO to optimize loss weights compared to manually set loss weights, we conduct a comparative experiment. Our setup is identical to that of ESRGAN [53] , with the exception that the loss weights are optimized by MOBO.Our optimization formula is expressed as:EQUATIONThe loss weights for ESRGAN [53] are adopted from the weights utilized by the authors in the original publication: [𝜔 1 , 𝜔 𝜙 5 , 𝜔 𝐺𝐴𝑁 ] = [1𝑒 -2, 1, 5𝑒 -3]. As demonstrated in Table 2 , our approach significantly surpasses ESRGAN [53] across all metrics, which is retrained on the DIV2K [1] training set using the official released codes. This experiment clearly demonstrates the substantial advantages of integrating MOBO for automatically optimizing loss weights in the SR model over the manual tuning of loss weights.From the comparison between 'MOBOSR (Ours)' in Table 2 and 'Ours-c' in Table 1 , it is evident that incorporating a greater variety of loss functions significantly benefits the SISR model, provided that the weights for these losses are optimally set. 
conclusion: 
 In this research, we propose the Multi-Objective Bayesian Optimization Super-Resolution (MOBOSR). To effectively address the balance between perceptual quality and distortion, we introduce multi-objective Bayesian optimization into the single-image superresolution model, dynamically adjusting the weights of various loss functions during training. Our method is rigorously validated through comprehensive experiments, demonstrating its effectiveness and versatility in both distortion and perceptual quality. Given its potential to be applied to nearly all domains of image restoration and enhancement, we believe our method can offer valuable insights and contributions to the community. 
: 
 Figure 1 : Left: Comparisons on the DIV2K [1] validation set, our method not only significantly surpasses others in distortion metrics (PSNR) but also holds an advantage in perceptual quality (LPIPS [58] ). [a, b, c] represent three different sample points on the perception-distortion Pareto frontier to illustrate the trade-off between distortion and perceptual quality. Right: From the quantitative and visual comparison of this single image, it is evident from our results, labeled as Ours-[a, b, c], that as PSNR decreases and LPIPS increases along the Pareto frontier, the images become sharper. However, this sharpness is accompanied by the introduction of noise and artifacts. In contrast to other methods, our method generates fewer artifacts and avoids excessive blurring, which can be attributed to our model achieving an optimal balance between perceptual quality and distortion. 
 
 Appendix:  


Decision: The work proposes a Single-image super-resolution (SISR) algorithm that uses a multi-objective optimization (MOO) method.  The method  balances perceptual quality with distortion by dynamically adjusting loss weights during training, reducing the need for manual hyperparameter tuning and lessening computational demands. The idea is novel and interesting. The paper is nicely written and the experimental validation is sufficient, showing the performance of the method both quantitatively and visually. 

Reviews are in general very positive and the authors have answered most of the reviewers comments and suggestions. I think this is a good and novel contribution to ACM MM.

Reviews: review: paper_summary: paper_summary: This paper investigates the use of different loss functions to create a pareto frontier for SISR based on a model called ESRGAN.
The work is well established although uses a strange conceptualization:
-	The use of a wrong definition for SISR models, that aim to create a higher resolution image from a lower resolution image. The paper confuses this with the typical testing model.
-	Consideres a specific metric as very special (LPIPS) instead of addopting a generic model with a large diversity of metrics.
Furthermore, the paper seems to ignore that the only reliable evaluation model is based on subjective evaluation, and not in metrics. Without a proper validation of a metric a SISR model evaluation is always somehow not reliable. Hece, considering a trained metric as a source of true quality is critical for the evaluation. A metric is only reliable after a subjective evaluation that tests it in the context it is being used.
strength_weakness: strength_weakness: See above
limitations: limitations: Remarks:

1 - Why the authors call LPIPS a perceptual metric? Usually, SSIM is considered a perceptual metric. Most of the developed metrics are considered perceptual. I do not understand why the authors consider LPIPS so unique for SISR.
If the authors want to use LPIPS for super-resolution evaluation need to validate its use first, typically with a subjective study. Otherwise, every is highly speculative. PSNR, on the other hand, provides a measure of the signal similarity. A PSNR below 30 is usually low quality.

2 - The formulation of eq. 2 is problematic. This is based in the principle that the high resolution image is available. However, that is not true in a real application (see remark 5 for further understanding of the problem in the 3.2 section definition of the problem.



3	- The abstract should summarize the paper. Instead, it is introductory. This is not an appropriate abstract.

4-	The authors use MOBO in the abstract without defining it.

5-	The problem formulation in 3.2 is not accurate. The paper specifies “…SISR model is to employ a neural network…”. However, we can do the same with other methodologies different from NN, whatever is the performance. Please correct.
Furthermore, this definition explains the typical methodology used in testing. However, in real applications the method intends to transform the Low-resolution image into a higher resolution. For testing purposes, the high resolution image is reduced to low resolution and the higher resolution obtained with the SISR method is evaluated compared with the high resolution.

6-The abstract should be reviewed to better reflect a summary of the paper. I could not understand what the paper is about by reading this badly formulated abstract.
paper_summary: paper_summary: Regression losses fail to recover high-frequency details, resulting in blurred images. Meanwhile, perceptual loss introduces noise or artifacts and may even produce incorrect textures. This paper proposes the Multi-Objective Bayesian Optimization Super-Resolution (MOBOSR) framework to adjust the loss weights of both regression and perceptual losses to address their inherent conflict and combine their advantages, allowing the framework to achieve a balance between perceptual and distortion trade-off.
strength_weakness: strength_weakness: This is an interesting paper that applies bayesian optimization to find the Pareto frontier between perception and distortion. And the effectiveness of the method is validated through extensive experiments with SOTA SR models.
limitations: limitations: 1. It's highly discouraged to use complete sentence as paper title. I suggest revising the title.
2. In Fig.1, the scatter plot on the left and the thumbnails on the right have inconsistent coordinates. For example, point c on the scatter plot is located at (30, 0.1), but on the right, it is denoted as (28.09, 0.0463). I should say Fig.1 is a bad teaser figure. Because it not only fails to clearly highlight the advantages of the proposed model but also creates unnecessary confusion. 
4. Bayesian optimization should yield outputs with as high PSNR as possible and as low LPIPS as possible, but the result is that as the training epochs increase, the model tends to prioritize high PSNR while sacrificing LPIPS (from Fig.1 in main text and Fig.2 in supplementary material). Can this be explained reasonably? Or is this a possible improvement to be addressed in future work?
5. Tab.1 and Tab.2 both involve ESRGAN and the same datasets, but why are the data different? Experimental settings need to be clarified.
6. Does the MOBOSR achieve the optimal solution at (or around) 476 epochs for all datasets? If not, how do you find the optimal training epoch for new data? This needs clarification.
paper_summary: paper_summary: In this manuscript, the authors explore the trade-off between image distortion and perception in reconstructed image. Building upon this, the authors introduce multi-objective optimization (MOO) into the Single Image Super-Resolution (SISR) domain, aiming to reduce the need for manual adjustment of hyperparameters by dynamically adjusting the weights of loss functions during training.
strength_weakness: strength_weakness: 1)The article represents the first attempt to apply MOBO to loss function adjustment in the SISR domain, addressing the challenge of trade-off between perceptual quality and distortion.
2)The proposed method of dynamically adjusting the weights of loss functions is significantly more resource-efficient compared to AutoML hyperparameter search methods.
3)The article conducts comprehensive experiments and maintains a smooth logical flow in its narrative.
limitations: limitations: 1)Whether through loss function adjustment or other strategies, the concept of perception-distortion trade-off has been attempted in some previous works on image super-resolution. However, the authors did not adequately review and discuss these methods. Some relevant articles include:
i) Park, S. H., Moon, Y. S., & Cho, N. I. (2023). Perception-oriented single image super-resolution using optimal objective estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1725-1735).
ii)Deng, X., Yang, R., Xu, M., & Dragotti, P. L. (2019). Wavelet domain style transfer for an effective perception-distortion tradeoff in single image super-resolution. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3076-3085).
2)The article "Perception-oriented single image super-resolution using optimal objective estimation" builds its network based on RRDB, similar to this manuscript, and adopts a strategy of dynamically adjusting loss functions to achieve a balance between perception and distortion. The authors need to provide comparisons with this algorithm, including performance analysis, differences in strategy usage, and the advantages of this article.
3)The authors should include discussions on the limitations of this work and future prospects.
paper_summary: paper_summary: This paper introduced multi-objective optimization to single-image super-resolution to balance perceptual quality and distortion. Specifically, they developed a method for dynamically adjusting loss weights during model training, thereby obviating the need for hyperparameter tuning and significantly reducing the computational resources required in comparison to AutoML hyperparameter search methods. This research expands the perceptual-distortion Pareto frontier of the single-image super-resolution domain, offering new insights into the tradeoffs between perceptual quality and distortion in various image restoration tasks.
strength_weakness: strength_weakness: Super-resolution is a research hotspot in image processing community. This paper investigates an important and interesting problem for super-resolution, that is, how to balance perceptual quality and distortion? The tradeoff problem also exists in other image enhancement and restoration tasks. This work may inspire similar works and facilitate the developments of image enhancement and restoration.  
(1)	The motivation of this paper is reasonable and convincing.  
(2)	The theoretical analysis and problem formulation of this paper are clear and easy to understand.
limitations: limitations: The detailed limitations of this paper are as follows.  
(1) Authors should provide and discuss the further applications of the proposed method. This helps to enlarge the contribution and impact of this paper.   
(2) Both the related work and comparison algorithms do not contain work in 2024. Authors should re-survey their academic community to make sure if they missed recent relevant work.   
(3) The generalization ability of the proposed method should be tested.