RefScale: Multi-temporal Assisted Image Rescaling in Repetitive Observation Scenarios

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:58:48

Abstract: With the continuous development of imaging technology and the gradual expansion of the amount of image data, how to achieve high compression efficiency of high-resolution images is a challenge problem for storage and transmission. 
Image rescaling aims to reduce the original data amount through downscaling to facilitate data transmission and storage before encoding, and reconstruct the quality through upscaling after decoding, which is a key technology to assist in high-ratio image compression. 
However, existing rescaling approaches are more focused on reconstruction quality rather than image compressibility. 
In repetitive observation scenarios, multi-temporal images brought by periodic observations provide an opportunity to alleviate the conflict between reconstruction quality and compressibility, that is, the historical images as reference indicates what information can be dropped at downscaling to reduce the information content in downscaled image and provides the dropped information to improve the image restoration quality at upscaling.
Based on this consideration, we propose a novel multi-temporal assisted reference-based image rescaling framework (RefScale).
Specifically, a referencing network is proposed to calculate the similarity map to provide the referencing condition, which is then injected into the conditional invertible neural network to guide the information drop at the downscaling stage and information fusion at the upscaling stage. Additionally, a low-resolution guidance loss is proposed to further constrain the data amount of the downscaled LR image. Experiments conducted on both satellite imaging and autonomous driving show the superior performance of our approach over the state-of-the-art methods.

Main paper: 
 introduction: 
 With the rapid advancements in imaging technology, high-resolution (HR) images and videos carry more visually pleasing details, delivering great benefits to human visual entertainment. However, the huge amount of data has resulted in significant challenges related to data storage and transmission. Therefore, a high effective compression method is urgently required to alleviate the conflict between limited bandwidth and high data volume, which is crucial for the storage, transmission, and management of image data, and even for technologies like Internet of Things and Cloud Computing.In order to achieve high-ratio compression, the rescaling-based coding, which contains rescaling module and compression module to form a Downscaling-Compression-Upscaling pipeline, stands out from other methods due to its low cost and high efficiency. Specifically, image downscaling is quite indispensable for storing and transferring large-size images, as the reduction of spatial resolution naturally removes part of the spatial information redundancy and significantly release the data amount. Additionally, when the content needs to be viewed, upscaling techniques are used to restore image details to their original resolution or adapt them to screens with varying resolutions. However, high-frequency details inevitably get lost during downscaling, posing a great challenge for reverse image reconstruction, which are generally referred to as "ill-posed" problems [4, 21, 36, 45] .To tackle the inverse problem of downscaling and upscaling, image rescaling has been studied mainly in three categories [42] . The first category is composed of methods that downscale images with fixed kernels, and upscale the LR images with image superresolution (SR) techniques [3, 35, 49] (Figure 1(a) ). Although advanced SR methods can restore some high-frequency textures, the crucial information lost during downscaling is hard to be fully recovered. With the insight that proper downscaling designs are influential in preserving beneficial information in LR images, the second category attempts to preserve the critical information for inverse restoration by optimizing both processes within a unified framework [9, 16, 33] (Figure 1(b) ). They have achieved remarkable visual reconstruction quality, but the authenticity of generated textures cannot be guaranteed. To fulfill the requirement of faithful image recovery, invertible neural network (INN) [2, 10, 32] based methods were proposed to build downscaling and upscaling into an invertible process [17, 43] (Figure 1(c) ). Up till now, these rescaling methods have greatly boosted the quality of reconstructed HR images, but little attention has been paid to the data amount of their LR counterparts, which may be overly informative for storage and transmission.This research aims to address the two challenges simultaneously: minimizing the amount of information conveyed in the downscaled images while maximizing the quality of the upscaled images. Inspired by the recent reference-based SR methods [24, 44, 50, 52] , we propose leveraging multi-temporal images as historical reference to achieve this goal. Specifically, reference images could guide the removal of redundant information that is similar to them in the downscaling process and provide the dropped information from the reference in the upscaling process to compensate for the loss of details. This strategy can be deployed in repetitive observed scenarios, such as satellite imaging with repetitive observations at specific locations or autonomous driving with repeated excursions along predetermined paths, for reducing their data storage and transmission costs [1, 24, 41, 44, 52] .In this paper, we propose a multi-temporal assisted referencebased framework for image rescaling (RefScale) (Figure 1(d) ) to alleviate the conflict between low information content and high visual quality by exploiting the relevance between the HR image and its historical reference. On one hand, during downscaling, we expect to drop as much mutual information that is comparable to reference images as possible, while preserving individual information that cannot be recovered from the reference. Additionally, the mutual information supplied by the reference can provide guidance for HR image reconstruction. To achieve this, we build a referencing network (Ref-Net) that collects the relevance between the HR and reference image and propose a novel conditional INN that embeds the mutual information into a latent variable that follows a specified distribution while retaining the individual information in the LR image. The HR image is then reconstructed based on the LR input, the condition, and random samples of latent variables. We employ a bottleneck structure and a quantization module to generate the similarity map with low bandwidth expense for recording the relevance. A similarity-based LR guidance loss is also designed to guide the redundancy elimination during training.In summary, the main contributions of this paper are: 1) We introduce multi-temporal images as reference in the repetitive observation scenarios for image rescaling to achieve a joint optimization of compression-friendly downscaling and high-quality upscaling as dropping the mutual information between the reference and the current images helps to reduce information content in the downscaled image whilst the reference will bring back the dropped information for image reconstruction.2) A novel rescaling framework called RefScale is proposed to integrate the reference in an invertible rescaling network, through the designs of referencing condition generation to indicate the similarity between the reference and the current image, and the conditionbased downscaling and upscaling network to conditionally drop and recover the mutual information in the rescaling process.3) Extensive experimental results demonstrate that our approach performs state-of-the-art results on both remote sensing and driving scenarios for both high reconstruction quality and low information content in LR images. 
related work 2.1 image rescaling: 
 Image rescaling refers to the resizing of a digital image, including image downscaling and upscaling. Previous work typically treated the two processes separately. The widely used downscaling approaches employ high-frequency filters, such as bilinear and bicubic. Later methods [11, 23, 37] attempted to preserve more structure and details in downscaled images for higher visual quality. Kopf et al. [11] proposed a content-adaptive method to optmize the shape and locations of downsampling kernels. Oeztireli et al. [28] optimized downscaled images with structural similarity index (SSIM). Weber et al. [37] preserved visually significant details by emphasizing the distinctive pixels, while Liu et al. [23] introduced a gradientratio prior to preserving salient edges. On the other hand, the upscaling process is typically achieved by image SR techniques. Since Dong et al. [3] proposed the convolutional neural networks (CNNs)-based single image SR (SISR), various effective SR models [5, 14, 35] have been proposed for deeper models and higher accuracy, some of which developed more powerful modules, e.g., residual connections [5, 14] , dense connections [6, 35] for deeper models and higher accuracy. Suffering from blurry results, some methods [27, 46] employed generative adversarial networks (GANs) to produce visually more perceptible results. More recently, normalizing flow [25, 48] and diffusion models [29] have been introduced to SISR and achieved superior performance.Recognizing the potential relevance between image downscaling and upscaling, recent studies [9, 16, 33] developed rescaling models that are jointly optimized for both processes and achieved more vivid reconstruction. Kim et al. [9] first proposed an autoencoderbased method, where the encoder and decoder simulate the downscaling and upscaling procedures, respectively. Li et al. [16] aimed to learn a compact-resolution image that is both visually pleasing and informative compared to HR images. Sun et al. [33] learned a content-adaptive downscaling kernel to maintain the structure of the HR input. More recently, INN-based methods [17, 43] were proposed to explicitly model the lost information during downscaling, resulting in remarkable reconstruction quality. IRN [43] was proposed to model the entire rescaling procedure using a bijective INN. HCFlow [17] further modeled the high-frequency component conditioned on the generated low-frequency component. Though these methods achieved outstanding reconstruction quality, they also introduced more information to LR images, which significantly increases the burden on data storage and transmission.The emergence of deep learning has led to significant advancements in computer vision tasks. However, there is still a lack of prior information learned from large-scale training data for restoration-related tasks such as image SR and image inpainting [19, 22, 34] . To break this dilemma, some methods have introduced reference images to provide auxiliary information. Reference-based image SR (RefSR) methods [8, 24, 31, 44, 50, 52] transfer high-frequency details from the reference image to the super-resolved image and exhibit promising results over SISR. Image inpainting methods [7, 15, 51, 53] take reference images as guidance for realistic texture and structure inference, effectively alleviating artifacts and unreasonable contents caused by large holes. The reference also frees users from the laborious interaction process for image generation [13, [38] [39] [40] . While reference-based image processing algorithms call for more specific application scenarios, they have shown to be vastly superior to approaches with no reference. 
proposed method 3.1 problem formulation: 
 Let 𝐼 𝐻𝑅 ∈ R 3×𝑊 ×𝐻 be the HR image, and 𝐼 𝐿𝑅 ∈ R 3× 𝑊 𝑠 × 𝐻 𝑠 be the corresponding LR image, where 𝑊 and 𝐻 denote the width and height, respectively, and 𝑠 is the downscaling factor. Conventionally, 𝐼 𝐿𝑅 is obtained using a predefined downscaling kernel, such as bicubic. Recent rescaling studies aim to jointly learn and optimize the downscaling operation 𝑓 (•) and the upscaling operation 𝑔(•) with respect to the objective of superior quality of the downscaled image Î𝐿𝑅 and the reconstructed HR image Î𝐻𝑅 , given by:EQUATION)where 𝜆 is a weight that balances the two terms.In addition to the image quality constraints, our objective is to reduce the information content (IC) contained in Î𝐿𝑅 to save on storage and transmission costs. To this end, we introduce a term in the objective function that constrains the information entropy of Î𝐿𝑅 and encourages dropping more details during downscaling, expressed as:EQUATIONwhere 𝐸 (•) measures the IC in Î𝐿𝑅 and 𝜆 𝑒 is the weight. The newly introduced item encourages dropping more details during downscaling, which hinders the reconstruction process. Thus, the key to resolving the problem is to fully use the information contained in the reference image.In this paper, we introduce a reference image 𝐼 𝑟𝑒 𝑓 ∈ R 𝐶 ×𝑊 ×𝐻 and exploit the correlations between 𝐼 𝐻𝑅 and 𝐼 𝑟𝑒 𝑓 for the image rescaling task. The proposed reference-based rescaling framework (RefScale) is depicted in Figure 2 , which consists of two sub-networks: a referencing network (Ref-Net) to generate a referencing condition from the HR image and the reference image, and then a conditional invertible neural network (INN) adopted the reference condition to perform image downscaling and upscaling respectively. A similarity-guided LR guidance loss is proposed to further constrain the IC of the downscaled image.In particular, at the downscaling stage, the Ref-Net calculates a similarity map 𝑆 𝑚𝑎𝑝 from 𝐼 𝐻𝑅 and 𝐼 𝑟𝑒 𝑓 , which is subsequently used to generate the referencing condition feature 𝐹 𝑐 . 𝑆 𝑚𝑎𝑝 is also used to guide the formation of the LR guidance loss L 𝐿𝑅 , under the consideration that the information can be more eliminated at similar regions while more information should be kept at dissimilar regions. Then, with the guidance from 𝐹 𝑐 and L 𝐿𝑅 , the conditional INN processes 𝐼 𝐻𝑅 to generate a less informative LR image Î𝐿𝑅 and embed the mutual information of 𝐼 𝐻𝑅 and 𝐼 𝑟𝑒 𝑓 into a latent variable In the following, we illustrate the key modules of the proposed RefScale, including: 1) the referencing network generating the referencing condition, 2) the LR guidance using the referencing similarity to constrain the IC of LR, and 3) the conditional INN to transfer between HR and LR images under the referencing condition and the LR guidance.The Ref-Net captures the correlations between 𝐼 𝐻𝑅 and 𝐼 𝑟𝑒 𝑓 and generates a similarity map 𝑆 𝑚𝑎𝑝 for LR guidance and a condition feature 𝐹 𝑐 for conditional INN.Extraction of Similarity Map 𝑆 𝑚𝑎𝑝 and Condition Feature 𝐹 𝑐 . As depicted in Figure 2 , 𝐼 𝐻𝑅 and 𝐼 𝑟𝑒 𝑓 are processed by the same encoder, and the output features 𝐹 𝐻𝑅 and 𝐹 𝑟𝑒 𝑓 are used to calculate 𝑆 𝑚𝑎𝑝 by the pixel-wise cosine distance operation:EQUATIONwhere (𝑖, 𝑗) is spatial position index. The referencing condition feature 𝐹 𝑐 is obtained by combining 𝐹 𝑟𝑒 𝑓 with 𝑆 𝑚𝑎𝑝 :EQUATIONwhere ⊙ denotes element-wise multiplication.Shrinkage of Similarity Map 𝑆 𝑚𝑎𝑝 . It should be noted that 𝐼 𝐻𝑅 is not available at the upscaling end, which means that 𝑆 𝑚𝑎𝑝 needs to be preserved and transmitted to the downscaling end to generate the same condition with the accessible reference image 𝐼 𝑟𝑒 𝑓 . To achieve this, we employ a bottleneck structure and a quantization module to reduce the size of 𝑆 𝑚𝑎𝑝 . First, 𝐹 𝐻𝑅 and 𝐹 𝑟𝑒 𝑓 are processed by downsampling layers to obtain a lower resolution for calculating 𝑆 𝑚𝑎𝑝 . Second, during the calculation of 𝐹 𝑐 , 𝑆 𝑚𝑎𝑝 is processed by upsampling layers to match the spatial size to 𝐹 𝑐 . Then, the quantization module is used to convert 𝑆 𝑚𝑎𝑝 from floating point to integer to further reduce the data size. As quantification is not differentiable, we introduce uniform noise of 𝑈 (-0.5, 0.5) during training to simulate this operation and quantify the similarity map when verifying our model.The mutual information extracted by Ref-Net tells the conditional INN that what can be discarded and recovered. In this section, we present a LR guidance to tell the conditional INN how to discard them, targeting reducing the IC of the LR image in similar regions with the reference while keeping details in dissimilar regions.Similarity-based LR Image Composition. Following the rescaling studies that utilize a bicubic-based downscaled image to guide LR image generation, we keep the bicubic as a detail-informative guide for dissimilar regions since existing methods can already reconstruct bicubic images well. Additionally, considering similar information between image pairs primarily pertains to structural features, we propose a new guide for similar regions, namely a Gaussian blurred downscaled image, which is less informative but still visually recognizable. It is processed by a downscaling and Gaussian blurring pipeline, which results in a significant loss of details and reduces the amount of information content. This new kind of LR guidance is adopted for constraining the downscaled image by a similarity-based LR guidance loss, which is formulated as follows:EQUATIONwhere 𝐼 𝐿𝑅 𝑏𝑖𝑐 and 𝐼 𝐿𝑅 𝑏𝑙𝑢𝑟 are bicubic-based and blur-based downscaled images, respectively. This loss is to drive Î𝐿𝑅 to discard more information where the HR image is similar to the reference but keeps irrecoverable details in Î𝐿𝑅 .To accomplish this goal of integrating the referencing condition from the reference, we propose a conditional invertible transformation 𝑓 (•):𝐼 𝐻𝑅 𝑐 ↔ [𝐼 𝐿𝑅 , 𝑧],where 𝑐 is the mutual information between the HR and reference images, and 𝑧 is a latent variable. The probability of HR images conditional on the mutual information is expressed as:EQUATIONHere, we assume that 𝐼 𝐿𝑅 is independent with 𝑧 as in IRN [43] and expect to exclude the information of 𝑧 from the conditional probability of 𝐼 𝐿𝑅 given 𝑐 to exclude the information of 𝑧. To this end, we approximate it as a multivariate Gaussian distribution with a mean of Ĩ𝐿𝑅 , which is a LR image that excludes mutual information. Furthermore, we formulate 𝑝 (𝑧|𝑐) as a standard multivariate Gaussian distribution, then the model can be defined as:EQUATIONwhere Σ is a diagonal covariance matrix with all diagonal elements close to zero. In practice, we employ the downscaled LR image as Ĩ𝐿𝑅 and the referencing condition 𝐹 𝑐 as 𝑐.Architecture of Conditional INN. We design a conditional INN with a newly introduced conditional affine coupling transform [10] , as illustrated in Figure 2 . The proposed conditional INN consists of a squeeze block and a stack of conditional flow blocks (CFB). The squeeze block exchanges spatial size for channel numbers to align the spatial dimension with the LR image. The CFB, which includes an Actnorm layer, an invertible convolution layer, and a conditional EQUATIONEQUATIONReferencing Condition Injection in CFB. 𝐹 𝑐 is injected into the conditional affine coupling layer of CFB to contribute to the process of discarding IC. The data flow of conditional INN progressively drops the information that can be obtained from 𝐼 𝑟𝑒 𝑓 based on 𝐹 𝑐 . Specifically, in the conditional affine coupling layer, the input hidden feature flow 𝐹 ℎ is split into two sub-features 𝐹 ℎ𝑎 and 𝐹 ℎ𝑏 along the channel axis, and then both undergo the affine transform to decide what information should be discarded under the constraint of 𝐹 𝑐 :EQUATIONEQUATIONwhere 𝜓 (•) and 𝜙 (•) are learnable functions and [ Fℎ𝑎 , Fℎ𝑏 ] are the output features. 𝐹 ℎ𝑏 and 𝐹 𝑐 are concatenated and fused by 𝜓 (•) and 𝜙 (•) to generate the parameters of affine transform, which is then applied to 𝐹 ℎ𝑎 to discard the information that can be recovered using 𝐹 𝑐 and 𝐹 ℎ𝑏 .Theoretically, INN can be trained by minimizing the negative loglikelihood loss. However, this loss cannot provide strong supervision for image reconstruction. Following IRN [43] , we constrain both the reconstruction quality of Î𝐻𝑅 and the distribution of 𝑧. Moreover, we use the similarity-based LR guidance on Î𝐿𝑅 to ensure that the information discarded during downscaling can be recovered from the reference. The objectives of HR reconstruction and distribution matching are described below.HR Reconstruction. We employ the widely used L 1 loss to measure the difference between the reconstructed HR image and the ground truth, as specified below:EQUATION)Distribution Matching. The objective of distribution matching is to encourage the generated latent variable to confirm a specific distribution:EQUATIONwhere 𝑞(𝐼 𝐻𝑅 ) and 𝑝 (𝑧) are the distributions of HR image 𝐼 𝐻𝑅 and latent variable 𝑧, respectively, 𝑓 𝑧 (•) is the partial transformation of mutual information and 𝐶𝐸 (•) is cross entropy function. We assume that 𝑝 (𝑧) follows a Gaussian distribution, thus the distribution matching can be easily calculated by L 2 regularization on the latent variable 𝑧.Total Loss. We optimize our model by minimizing the total loss composed by HR reconstruction loss L 𝐻𝑅 , LR guidance loss L 𝐿𝑅 , and distribution matching loss L 𝑑𝑖𝑠𝑡𝑟 :EQUATIONwhere 𝜆 𝐻𝑅 , 𝜆 𝐿𝑅 and 𝜆 𝑑𝑖𝑠𝑡𝑟 are the associated weights. 
experiments 4.1 experimental setup: 
 Datasets. To validate the effectiveness of RefScale, we utilize a dataset from the remote sensing scenes with readily available reference images. In addition, we evaluated on a driving scenes with repeated driving along a predetermined rout to demonstrate the extensibility of the proposed method. 1) Remote sensing (RS) dataset with historical images from periodical revisiting. We collected these 8-bit RGB images from the SPOT-5 satellite from various cities and timestamps, and their original resolution ranges from 1878×1400 to 6264×3456. We collected images of different scenes, such as railway station, port, airport, suburb, mountain and city center. Besides, the reference images are selected from different seasons or different years to guarantee the reliability of verification. To construct the training set, we crop them into non-overlapping patches of size 512×512 size, resulting in 6,690 image pairs of HR image and its reference. For the test set, we use 58 uncropped images with five historical images taken at different times serving as the reference images. 2) ACDC-snow dataset [30] with reference images from revisiting routes at different weather conditions. It is a natural image dataset consisting of cityscape driving images captured from the same routes but with different conditions, i.e., one image captured in sunny day and another in a snowy day. It contains 1,000 images and is split into train set, validation set and test set for roughly 4:1:5 proportion, and the resolution of each image is 1920×1080.Baseline Methods. We utilized three categories of image resizing methods as our baselines for comparison: 1) General SR methods, including six SISR methods, i.e., Bicubic, EDSR [20] , RCAN [49] , NLSN [26] , LDL [18] , LTE [12] , and two RefSR methods, i.e., TTSR [44] and 𝐶 2 Matching [8] , with bicubic interpolation for downscaling, 2) jointly optimized rescaling methods, TAD&TAU [9] , and 3) INNbased rescaling methods, including IRN [43] and HCFlow [17] .Evaluation Metrics. We employ PSNR and SSIM for the comparison of reconstruction quality. We also adopt spatial information (SI) [47] as a metric to measure the image complexity of LR images. SI has been proved to be strongly positively correlated with JPEGbased image complexity measures. Specifically, it can be calculated by:EQUATIONwhere 𝑠 ℎ and 𝑠 𝑣 denote data of Y channel filtered with horizontal and vertical Sobel kernels, respectively. Additionally, we compress LR images and similarity maps and use bit per pixel (bpp) as an auxiliary metric for SI. the image as a long-term reference does not need to be transmitted, namely not to be included in the calculation of bpp.Implementation Details. We implement the proposed RefScale with Pytorch and optimize it using Adam with 𝛽 1 = 0.9, 𝛽 2 = 0.999, and a learning rate of 2 × 10 -4 . During training, the HR images and corresponding reference images are cropped to 128 × 128 patches and augmented with the same random flips and rotations. The batch size is set to 8. The loss weights 𝜆 𝐻𝑅 , 𝜆 𝐿𝑅 , and 𝜆 𝑑𝑖𝑠𝑡𝑟 are set to 1, 4, 1 for 2× rescaling, and 1, 16, 1 for 4× rescaling, respectively. We employ RRDB [35] as the backbone of Ref-Net and initialize it by pre-training on natural images.Quantitative Results. To ensure a fair comparison, we fine-tuned all the baseline methods on both RS and ACDC-snow datasets. As shown in Table 1 , our method achieves superior performance on both datasets, with increments in PSNR and SSIM and decreases in Qualitative Results. To illustrate the details of the downscaled and upscaled images, we visualize some samples from both datasets at a 4× scale. As shown in Figure 3 , our method produces sharper and more realistic HR images, which contain more details and are more faithful to the ground truth than those reconstructed by baseline methods, since reference images provide rich auxiliary information. Moreover, our model leverages the reference image to guide the downsampling process and discard mutual information, which results in downscaled images that are smoother and contain fewer details than other methods.Visualisation of Similarity Map. To depict how Ref-Net captures the correlations, we present the similarity map in Figure 5 . The map shows that our method can capture similar information from reference images that share a similar texture or structure with the HR images. The warm region in Figure 5 indicates the presence of abundant similarity between the reference and HR images. It can be observed that proposed method effectively detects dissimilar regions, such as roads in the first row and a newly built airport runway in the second row. In this case, our method can preserve the individual information in LR images for better restoration.Analysis on LR Guidance. To evaluate the effectiveness of the similarity-based LR guidance loss, we perform comparative experiments using different types of downscaled image guidance, generated by a weighted average of bicubic downsampled image and Gaussian blurred image, namely Ĩ𝐿𝑅 = 𝜔 ×𝐼 𝐿𝑅 𝑏𝑖𝑐 + (1-𝜔) ×𝐼 𝐿𝑅 𝑏𝑙𝑢𝑟 , (𝜔 = 0, 0.25, 0.5, 0.75, 1). The results are shown in Figure 6 and measured by PSNR and SI. The results from our similarity-based weighting scheme are on the upper left of the corresponding curves in both datasets, showing that our scheme can achieve better PSNR and lower SI simultaneously.Influence of Different References. To validate the impact of reference differences, we employ three reference images of varying styles and two reference images of varying scenes, as illustrated in Figure 4 . Reference 1-3 are images taken at the same location but on different dates, whereas Reference 4-5 are unrelated images. The degree of similarity between each reference image and the input image was measured using the average of the similarity matrix, denoted as S. Reference 1 has the highest degree of similarity, resulting in the lowest SI, while References 4 and 5 have the highest SI since they are completely unrelated to the input image and thus the LR image needs to record all necessary information for reconstruction. Despite the diverse reference images, our method achieves stable reconstruction quality (b-d), demonstrating that it can adaptively discard information according to the given reference.Analysis on the Condition Components. We explore the effects of referencing conditions with various settings, including condition components and the shrinkage of the similarity map. The evaluations are performed in terms of reconstruction quality and SI of the similarity map, as shown in Table 2 . Case 1 shows the result the main factor, as refined correlations might be corrupted during pooling. Case 5 results in acceptable quality degradation and IC, while the result of Case 6 is similar to that of Case 2. Therefore, we take Case 5 as the final reference setting.We evaluate the combination of image rescaling and image compression methods on our RS dataset. We compare two rescaling method, bicubic interpolation and another INN-based method, IRN, and the scale of all rescaling methods is set to 2×. For a fair comparison, we adopt the JPEG algorithm as the combined lossy compression method since it requires no tuning. We adjust the quality of to control the similar compression ratio under different rescaling methods.The PSNR of the reconstructed image and bit rate are evalusted, and the R-D curves are shown in Fig. 7 . As expected, the reconstruction quality of the rescaling-based compression methods outperforms that of the native compression method at low bit rates. IRN presents a slight advantage over bicubic interpolation, while the proposed Refscale shows a significant improvement over other methods, benefiting from the introduction of reference information. 
conclusion: 
 In this paper, we propose RefScale, a reference-based image-rescaling framework for producing higher-quality reconstructed images with lower information content in downscaled images. RefScale involves a referencing network that exploits the correlations between the HR image and the reference to generate a similarity map and produce the referencing condition, which guides the generation of the LR image. We also propose a conditional INN that incorporates the mutual information into a latent variable, while retaining individual information in the LR image conditioned on the reference. Experiments demonstrate that RefScale outperforms the state-of-the-art methods in terms of both image reconstruction quality and data amount, as well as the lightweight property and high efficiency of our model. In addition, The proposed method is expected to assist high ratio compression to boost transmission efficiency, and show potential applications in scenarios where long-term observations are routinely conducted.  
 
 Appendix:  


Decision: The paper received mixed reviews but the author's rebuttal, specifically reporting additional results, helped to improve the rating of one reviewer. Since other reviews are still mixed, decision is mainly based on one review which changed from borderline accept to weak accept.

Reviews: review: paper_summary: paper_summary: This article introduces a framework called RefScale that aims to solve the problem between low information content and high visual quality in image rescaling. This framework utilizes historical reference images to recover lost information, which is achieved by building a reference network and conditional INN. The authors also designed a similarity loss to guide the training process. The main contributions of the article include introducing multi-period images as reference, proposing the RefScale framework and conducting extensive experimental verification. The results show that it performs well in terms of reconstruction quality and information content.
strength_weakness: strength_weakness: Multi-epoch images are introduced as references in repeated observation scenarios to achieve joint optimization of compression-friendly downscaling and high-quality upscaling.
A novel rescaling framework called RefScale is proposed to indicate the similarity between the reference and the current image through a design generated by reference conditions, and conditionally discard and restore during the rescaling process through a condition-based shrink and enlargement network. mutual information.
Extensive experimental results show that the method exhibits good reconstruction quality and low information content in remote sensing and driving scenarios.
limitations: limitations: The method used in the comparative experiments in this article is not the latest existing method, and it is not an excellent method.
The ablation experiments in this article also do not well prove the indispensability of the method.
The method of the article also does not elaborate on the innovation of the method.
For equation 3 and 5, please explain in detail what they do.
paper_summary: paper_summary: this paper propose a novel multi-temporal assisted reference-based image rescaling
framework (RefScale). Specifically, a referencing network is proposed to calculate the similarity map to provide the referencing
condition, which is then injected into the conditional invertible
neural network to guide the information drop at the downscaling
stage and information fusion at the upscaling stage. Additionally,
a low-resolution guidance loss is proposed to further constrain
the data amount of the downscaled LR image. Experiments conducted on both satellite imaging and autonomous driving show
the superior performance of refscale over the state-of-the-art
methods.
strength_weakness: strength_weakness: 1. the idea of reference-based scaling is novel
2. this paper has a significant performance improvement compared with existing methods
3. the writings are clear
limitations: limitations: 1. as mentioned in the article, the $I^{HR}$ is not available, so the inference and the training steps are different, a clearer explaination or algorithm detail should be provided.
paper_summary: paper_summary: The paper proposes a reference-based image-rescaling framework to minimize the amount of information conveyed in the downscaled
images while maximizing the quality of the upscaled images. A referencing network is proposed to exploit the correlations between the HR
image and the reference to generate a similarity map. Also, a conditional INN is proposed to incorporate the mutual information into a latent variable, while retaining individual information in the LR image conditioned on the reference.
strength_weakness: strength_weakness: + The idea of using multi-temporal images as references to reduce the information conveyed in the downscaled images is interesting.
+ This paper is well-structured and is presented understandably with a logical flow between sections.
+ This paper presents comprehensive experiments.
limitations: limitations: - The papers mentioned in the related work section " Image Rescaling" are too old.
- The methods compared in the experimental section are quite outdated, with the most recent ones being from 2021. Most of the methods used for comparison are from before 2021.
- How is the proposed method different from other reference-based image rescaling methods?