Prior Metadata-Driven RAW Reconstruction: Eliminating the Need for Per-Image Metadata

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-15 10:38:06

Abstract: While RAW images are efficient for image editing and perception tasks, their large size can strain camera storage and bandwidth. Techniques exist to reconstruct RAW images from sRGB data, but these methods typically require additional metadata from the RAW image, which adds to camera processing demands. To address this problem, we propose using Prior Meta as a reference to reconstruct the RAW data instead of relying on per-image metadata. Prior metadata is extracted offline from reference RAW images, which are usually part of the training dataset and have similar scenes and light conditions as the target image. With this prior metadata, the camera does not need to provide any extra processing other than the sRGB images, and our model can autonomously find the desired prior information. To achieve this, we design a three-step pipeline. First, we build a pixel searching network that can find the most similar pixels in the reference RAW images as prior information. Then, in the second step, we compress the large-scale reference images to about 0.02% of their original size to reduce the searching cost. Finally, in the last step, we develop a neural network reconstructor to reconstruct the high-fidelity RAW images. Our model achieves comparable, and even better, performance than RAW reconstruction methods based on metadata.

Main paper: 
 related works 2.1 raw image application: 
 The application of RAW data to low-level tasks has attracted more and more attention in computer vision. For example, relying on the linear relationship between RAW data and scene radiation, image denoising and image super-resolution tasks benefit from it. The work in [2] designs a RAW domain denoising network using synthetic noise data by adding shot noise and read noise, which gets significant improvements in denoising capabilities compared to state-of-the-art RGB-based denoising models. The work in [35] proposes a highly accurate noise model based on the characteristics of CMOS photosensors to address the extreme low-light RAW denoising problem. In [36] , Xing et al. present an end-to-end joint learning framework for demosaicing, denoising, and super-resolution, which enables satisfactory results to be achieved on multiple visual tasks based on RAW data. In addition, benefiting from a bigger dynamic range, RAW data also perform better in the low-light enhancement task. It is interesting to note that Chen et al. [6] build a dataset with low-light images and corresponding long-exposure high-quality images, and the promising results inspire future work. After this, the work in [10] and [16] improve the accuracy and speed of the model, respectively. Except for the above-mentioned low-level tasks, the advantages of RAW data in high-level tasks have also been discovered, such as object detection and segmentation. On the one hand, since ISP processing is only for human eye viewing, it is a practical problem to remove unnecessary steps of ISP to obtain faster processing speed, which is crucial for some real-time systems that require low time consumption and fast response, such as autonomous driving. As for related works, [23, 30, 40] are dedicated to configuring ISP as learnable procedures. Through the end-to-end optimization process of the neural network, we are able to skillfully mine ISP operations that are beneficial to downstream tasks and configure their parameters reasonably. On the other hand, a larger bit width can provide more information for perception tasks in dark scenes and highly dynamic scenes. Works [38] and [7] respectively prove that RAW data with higher bits can achieve better results in object detection and instance segmentation tasks under dark light conditions. In work [27] , the researchers propose a neural auto-exposure system that jointly trains an object detector and an image signal processing pipeline. Experiments on the vehicle detection data set prove that reasonable processing of RAW data can greatly improve the detection accuracy of the model in HDR scenes. Even though the benefits of the RAW format in these tasks have been discovered, further exploration is needed.Due to the difficulty in obtaining RAW data, the convenient use of sRGB images to reverse-generate RAW images has gradually become a hot topic. According to whether they rely on metadata information, popular sRGB2RAW methods are classified into two categories. Blind RAW reconstruction. Image signal processing enhances the captured image to make it more visually appealing to humans. This is achieved through various steps, such as white balance, demosaicing, and gamma correction, many of which can be explicitly modeled. As a result, some methods for reconstructing RAW images [2, 9] aim to reverse certain key steps of modern ISPs in order to obtain the unprocessed RAW image. For example, [9] models six key stages of the ISP, including demosaic, lens shading correction, white balance, color correction, gamma correction, and tone mapping. For camera-specific modules (such as color correction), this method represents them by constructing a learnable dictionary. While these methods offer strong interpretability, they often lack generalization ability due to the specific nature of the camera ISP. To overcome this limitation, other approaches [37, 42] propose using the concept of cycle constraints to simultaneously learn the bidirectional process of RAW-to-sRGB and sRGB-to-RAW conversion. For instance, [37] proposes using the Normalizing Flow model to learn an invertible ISP network. Due to the inherent reversibility of the Normalizing Flow model, we can conveniently perform RAWto-sRGB conversion and its inverse process. Unfortunately, due to the inevitable information loss in ISP, blind RAW reconstruction methods struggle to achieve satisfactory results. RAW reconstruction with metadata. Metadata-based methods leverage additional information to assist in the reconstruction process, such as ISP parameters [25] , sampled RAW pixels [17, 24, 28] , and latent features [33] . Early methods [25] store key parameters in the ISP forward rendering process, such as color correction matrices, auto white balance parameters, etc., but the obvious disadvantage is that some key operations cannot be quantitatively stored, such as tone mapping. The most commonly used sampling-based methods employ strategies to save a subset of pixels from the original resolution RAW image. For instance, methods in [17, 28] utilize uniformly sampled pixels as metadata and then employ interpolation algorithms or neural networks to recover the remaining pixels. The work in [28] learns the content-related pixels using a sampler network, and then performs an efficient recovery process through a U-Net network. These metadata provide reliable auxiliary information for RAW image recovery, but generating corresponding metadata for each image on the camera side can be time-consuming and labor-intensive.In contrast to the aforementioned methods, our model directly extracts representative pixels from existing RAW images as prior information, which serves as guidance and eliminates the need for an additional metadata generator. which converts sRGB pixels into a RAW-aware embedding. This embedding allows us to search the entire reference dataset for similar pixels to the target pixel. These similar pixels can then be used as prior information for RAW image reconstruction. This approach is based on the observation that archived RAW images captured by the same type of devices (same sensor) can provide valuable information for the RAW reconstruction process. At the pixel level, previous RAW pixels captured in similar scenes with similar lighting conditions can serve as high-quality references for recovering the current pixel. The design and training of the Pixel Searching Encoder will be discussed in detail in § 3.2. 
method 3.1 pipeline: 
 To achieve better generalization, the reference dataset should contain a variety of scenes with different lighting conditions and objects. However, this would require a large-scale reference dataset, causing a significant computational burden when searching for similar pixels. To address this issue, we propose the second step in our pipeline: clustering and grouping the pixels into a few of the most representative ones. Despite the large size of the entire reference dataset, many pixels exhibit similarity. Therefore, we can evaluate the similarity between the pixels in the reference dataset and select a few highly representative pixels to represent the entire dataset. To accomplish this, we design a clustering and grouping algorithm, which will be introduced in detail in § 3.3.While similar pixels in the reference dataset can provide valuable prior information, the assembled RAW image created from these pixels may not perfectly match the target RAW. Therefore, a deep neural network is required to address these differences. To fully utilize the prior information from the prior metadata and the target sRGB image, we introduce a Prior Metadata Fusion Module. This module incorporates all useful information for RAW reconstruction. We then use an encoder-decoder network to reconstruct the target RAW based on the fused inputs. This procedure will be demonstrated in Section § 3.4.The transformation from RAW to sRGB, denoted as sRGB = 𝑓 (RAW), is an information-loss procedure. As a result, the inverse transformation, RAW = 𝑓 -1 (sRGB), is not a one-to-one mapping. This means that the same sRGB pixels can represent different RAW values depending on the context. This feature makes it challenging to identify similar RAW pixels in the reference images based on the sRGB values of the target images. Instead of searching for similar pixels based on the similarity of sRGB values, we develop a deep neural network that learns to find similar RAW pixels using an sRGB image as input. This network converts the surrounding sRGB context of the pixel into a high-dimensional vector and evaluates the distance between the vectors to identify the most similar RAW pixels. When evaluating the distance between pixels, the Pixel Searching Encoder takes into account not only the similarity between their sRGB values but also the lighting conditions and scenes depicted in the context patch. This approach enhances the efficiency of finding similar RAW values.The Pixel Searching Encoder adopts a ResNet [12] -like structure as a feature extractor with two major modifications. On the one hand, the similarity among RAW pixels is considered a low-level feature, so we decrease the parameter number of the network to prevent overfitting. On the other hand, the down-sample ratio of the encoder will affect the resolution of the feature map. To maintain a higher resolution of the pixel encoding, we adjust the total down-sampling ratio of the encoder to 2. This means that the output resolution of the encoding feature map is half the size of the original sRGB image.To effectively train the Pixel Searching Encoder, we propose a contrastive training method as illustrated in Fig. 2(a) . The main idea is to bring sRGB features closer together if their RAW values are similar and push them apart if they are not. To achieve this, we first randomly divide the original training dataset into two subdatasets: a target dataset and a reference dataset. The images in the reference dataset will serve as references and will be used to search for similar RAW pixels for the target image. During the training process, we randomly select batches from the target dataset and separate batches from the reference set as references. We input the target and reference images into the Pixel Searching Encoder to extract feature maps:𝑓 𝑡 = Encoder(sRGB 𝑡 ) 𝑓 𝑟 = Encoder(sRGB 𝑟 ).(1)We then multiply the two feature maps to generate an affinity matrix between the target and reference images:EQUATIONFinally, we apply a softmax function to the affinity matrix in the reference dimension and multiply it with the reference RAW pixels to produce an assembled RAW image:EQUATIONThe assembled RAW image F RAW 𝑡 is utilized to calculate the L2 loss with the actual target RAW images. During this process, the values of the reference pixels remain fixed, and only the affinity matrix can be optimized using the gradient descent algorithm. Consequently, the Pixel Searching Encoder will learn to maximize the inner product between feature embeddings for similar RAW pixels and minimize it for dissimilar ones.The computation of the affinity matrix has a space and time complexity of O (𝑀𝐿𝑁 ), where 𝑀 and 𝑁 represent the number of pixels in each target and reference image, and 𝐿 is the size of the reference dataset. During the inference phase, this calculation becomes computationally intensive if the reference dataset size 𝐿 is very large. However, it is important to have a diverse reference dataset that includes various conditions where the target images may be captured. Only with this diverse reference dataset, the model can efficiently select the most similar pixel for the target image.In addition to using reference images to assemble the target RAW images, we can also assemble the reference dataset itself by incorporating specific pixels from the reference dataset. This approach allows us to select a subset of the most representative pixels from the dataset to represent the entire reference dataset. To achieve this, as shown in Fig. 2(b) , we propose a clustering and grouping algorithm that effectively selects and generates the representative pixels for the reference dataset.We first manually select 100 representative images from the training dataset as a reference dataset. These images are then converted into feature maps using the Pixel Searching Encoder trained with our contrastive learning method. Each image in the reference dataset is cropped and resized into a resolution of 1024 × 1024, resulting in output feature maps with a resolution of 512 × 512. The reference dataset contains a total of 26, 214, 400 pixels. We then use the K-Means algorithm to cluster the feature embeddings into several different groups. Since the reference dataset has a large number of sampling points, the clustering procedure requires significant computation resources. To address this issue, we utilize the faiss algorithm [14] , which is a library for efficient similarity search and clustering of dense vectors. With this algorithm, we can perform the clustering procedure on a single computer within a few minutes, making our methods more easily applicable. In our experiment, we use the inner product as the clustering distance measurement and randomly select the initial centroid of each cluster. We cluster a total of 5, 000 groups in our experiment and select the 200 nearest features in the reference dataset for each group. Finally, we average the 200 features and their corresponding sRGB and RAW pixel values to produce the final prior metadata. Compared to the original 26, 214, 400 pixels, the 5, 000 pixels in the prior metadata represent only about 0.02% of the total. However, in our experiment, the 5, 000 pixels in the prior metadata achieve virtually the same performance as the entire reference dataset, which will be introduced detailedly in § 4.By utilizing the grouped prior metadata and the Pixel Searching Encoder, we can assemble a target RAW image through similarity searching. However, it is important to understand that the assembled RAW image cannot be directly used as reconstructed RAW data because of slight discrepancies with the actual RAW image at each pixel. In some cases, there may be pixels in unique scenarios that cannot find exact matches. Additionally, fine details like edges may not be accurately represented by the reference pixels. To overcome this limitation, we have implemented a deep learning network to address and rectify these disparities.As illustrated in Fig. 4 , the reconstructor we propose comprises two primary modules: a Prior Metadata Fusion Module and a RAW Encoder Decoder Module. The initial stage in effectively reconstructing the desired RAW image is to efficiently merge all available prior data. To accomplish this, we have developed the module to integrate the information encoded in the prior metadata and the target sRGB image. Prior Metadata Fusion Module: Fig. 4 provides an overview of this module. To start, we transform the target sRGB image into a feature map for similarity searching using the well-trained Pixel Searching Encoder. Then, using the affinity matrix A that captures the relationship between the target features and the prior metadata features, we produce an assembled RAW image and its corresponding sRGB image:EQUATIONThe assembled sRGB image can provide valuable information for assessing the accuracy of each pixel in the assembly. When the searched sRGB pixel has little difference from the target pixels, there is a high probability that the searched RAW pixel is similar to the target RAW pixel. However, due to the non-injective nature of this theory, it may not always be correct. Nonetheless, it can help filter out most of the unreliable pixels. Using this information, we calculate the distance D between the assembled sRGB image and the target sRGB image:EQUATIONThen, we combine this distance information with the target sRGB image and the assembled RAW image. We then send them into a convolutional network for prior-aware feature extraction:EQUATIONTo eliminate the unreliable pixels in this feature map, we utilize an additional convolutional network to generate spatial self-attention:EQUATIONIn addition to providing detailed prior information to individual pixels in the generated target RAW image, the prior metadata can also serve as a global reference for reconstructing the target RAW image. To achieve this, we need the model to learn how to extract relevant information from the prior metadata. Therefore, we employ a multi-head attention mechanism that considers the feature 𝑓 𝑐 as the query, the prior feature 𝑓 𝑟 as the key, and the prior RAW pixels RAW 𝑟 as the value. During the training phase, this attention module is optimized in an end-to-end manner and effectively aggregates the desired global reference for each pixel in the feature map. Finally, the RAW reconstruction feature map 𝑓 𝑜 is obtained using a feedforward network (FFN):EQUATIONFinally, we use the Encoder-Decoder Module to reconstruct the final RAW image. RAW Encoder-Decoder Module: We utilize an Encoder-Decoder structure, as described in [24] , to accomplish the RAW reconstruction. This module takes the RAW reconstruction feature map as input and progressively reduces the features to a 1 16 resolution, allowing for the encoding of information with a large receptive field. Following the encoding procedure, the decoder gradually upsamples the feature map and combines it with the corresponding feature maps from the encoder, in order to recover the detailed information that was lost during the down-sampling procedure. In contrast to previous structures, we additionally fuse the upsampled feature map with the RAW reconstruction feature map, as it encodes important detailed information from the assembled RAW image. To achieve this, we first interpolate the RAW reconstruction feature map to the same size as the upsampled feature map, and then employ a convolution operation to fuse the two feature maps. Finally, we utilize a head network to decode the feature map into 3-channel RAW images and limit the value range to 0-1 using a sigmoid function. 
experiments 4.1 implementation detail: 
 Dataset To evaluate the performance of our method for the RAW reconstruction task, we chose the representative NUS dataset [8] as a benchmark. This dataset consists of photos captured by three different cameras: Samsung NX2000, Olympus E-PL6, and Sony SLT-A57, with 202, 208, and 268 RAW images, respectively. Following the approach of Nam et al., all the RAW images were processed using the demosaic procedure and bilinear interpolation to obtain the 3-channel RAW-RGB image with the original resolution. To obtain the corresponding sRGB images, we used a software ISP emulator [15] to convert the RAW images into the sRGB format. Loss In the training phase, the reconstruction network uses the sRGB images as input to assemble the prior RAW and prior sRGB images. This is done by taking the sRGB inputs and the prior metadata generated by the clustering and grouping algorithm. The model then produces the RAW prediction based on the sRGB images and the prior information. To determine the loss function, we utilize 𝐿 2 loss and 𝑆𝑆𝐼 𝑀 [34] loss as:EQUATIONwhere 𝑦 and 𝑌 are the prediction and ground truth, respectively. 𝛼 and 𝛽 are the balance ratio.Metrics In this paper, we adopt PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure) [34] as our main evaluation metrics, following precedents in the field. PSNR for each (𝑠𝑅𝐺𝐵 𝑡 , 𝑠𝑅𝐺𝐵 𝑟 , 𝑅𝐴𝑊 𝑡 , 𝑅𝐴𝑊 𝑟 ) in 𝑑𝑎𝑡𝑎 𝑡 do 3:𝑓 𝑡 ← 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 (𝑠𝑅𝐺𝐵 𝑡 ) 4:𝑓 𝑟 ← 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 (𝑠𝑅𝐺𝐵 𝑟 )A ← 𝑓 𝑡 × 𝑓 𝑟A 𝑠 ← 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 (A)7:F RAW 𝑡 ← A 𝑠 × 𝑅𝐴𝑊 𝑟 8:𝑙𝑜𝑠𝑠 ← 𝐿𝑜𝑠𝑠 (F RAW 𝑡 , 𝑅𝐴𝑊 𝑡 )𝐸𝑛𝑐𝑜𝑑𝑒𝑟 .𝑢𝑝𝑑𝑎𝑡𝑒_𝑤𝑒𝑖𝑔ℎ𝑡𝑠 (𝑙𝑜𝑠𝑠) 𝑓 𝑟 𝐴𝑙𝑙 ← [𝐸𝑛𝑐𝑜𝑑𝑒𝑟 (𝑠𝑅𝐺𝐵 𝑟 ) for 𝑠𝑅𝐺𝐵 𝑟 in 𝑟𝑔𝑏_𝑑𝑎𝑡𝑎 𝑟 ] 3: 𝑅𝐴𝑊 𝑟 𝐴𝑙𝑙 ← 𝑅𝐴𝑊 𝑟 for 𝑅𝐴𝑊 𝑟 in 𝑟𝑎𝑤_𝑑𝑎𝑡𝑎 𝑟 ] 4: 𝑠𝑅𝐺𝐵 𝑟 𝐴𝑙𝑙 ← 𝑠𝑅𝐺𝐵 𝑟 for 𝑠𝑅𝐺𝐵 𝑟 in 𝑟𝑔𝑏_𝑑𝑎𝑡𝑎 𝑟 ] 5: 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑠 ← 𝐾𝑀𝑒𝑎𝑛𝑠𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝑖𝑛𝑔(𝑓 𝑟 𝐿𝑖𝑠𝑡) 6:for each 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 in 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑠 do 𝑓 𝑡 ← 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 (𝑠𝑅𝐺𝐵 𝑡 ) 3:A 𝑠 ← 𝑎𝑓 𝑓 𝑖𝑛𝑖𝑡𝑦𝑀𝑎𝑡𝑟𝑖𝑥 (𝑓 𝑡 , 𝑝𝑟𝑖𝑜𝑟 ) 4: F 𝑠𝑅𝐺𝐵 𝑡 , F 𝑅𝐴𝑊 𝑡 ← 𝑎𝑠𝑠𝑒𝑚𝑏𝑙𝑒 (A 𝑠 , 𝑝𝑟𝑖𝑜𝑟 ) 5: 𝑑𝑖 𝑓 𝑓 ← 𝑎𝑏𝑠 (F 𝑠𝑅𝐺𝐵 𝑡 -𝑠𝑅𝐺𝐵 𝑡 ) 6: 𝑓 𝑝 ← 𝐶𝑜𝑛𝑣 fuse ([𝑑𝑖 𝑓 𝑓 , 𝑠𝑅𝐺𝐵 𝑡 , F 𝑅𝐴𝑊 𝑡 ]) 7: 𝑓 𝑐 ← 𝑓 𝑝 • 𝑆𝑖𝑔𝑚𝑜𝑖𝑑 (𝐶𝑜𝑛𝑣 conf (𝑓 𝑝 )) 8:𝑓 𝑜 ← 𝐹 𝐹 𝑁 (𝑀𝐻𝐴(𝑓 𝑐 , 𝑝𝑟𝑖𝑜𝑟 ))𝑂𝑢𝑡𝑝𝑢𝑡 ← 𝑈 𝑁 𝑒𝑡 (𝑓 𝑜 , 𝑠𝑅𝐺𝐵 𝑡 ) 10: end procedure measures the ratio of the maximum possible signal to the corrupting noise, indicating image reconstruction quality. SSIM, on the other hand, assesses changes in structural information, luminance, and contrast, offering a perception-based measure of image quality. These metrics together provide a balanced evaluation of both technical and perceptual aspects of RAW image reconstruction.To improve understanding and implementation of the proposed method, we have included pseudo codes for the main modules in our approach. Algorithm. 1 demonstrates the contrastive training phase of our Pixel Searching Encoder, while Algorithm. 2 showcases the reference clustering and grouping algorithm. Lastly, Algorithm. 3 presents the Prior Metadata Based RAW Reconstructing Algorithm. Clustering and Grouping To assess the impact of the clustering and grouping algorithm, we conducted a series of ablation studies on Olympus E-PL6 [8] . As shown in Table 1 , Pixel Searching directly assembles the output RAW by searching for similar RAW pixels from the entire reference dataset. Clustering and Grouping involve assembling the RAW output with search pixels based on prior metadata grouping. The results indicate that the clustering and grouping algorithm does not noticeably reduce the performance in terms of PSNR and SSIM, but it significantly reduces the computational burden of the searching procedure.There are two hyperparameters that can affect the performance and efficiency: the number of clusters and the group size. The number of clusters determines the number of pixels included in the prior metadata. A larger number of clusters results in greater pixel variety, but it also increases the computational burden when searching for similar pixels. On the other hand, the group size determines the number of pixels used for grouping. A larger group size leads to more stable feature generation, but it reduces the diversity of the final features. As shown in the table, when we choose an appropriate cluster number and group size, the clustering and grouping algorithm does not noticeably compromise performance compared to searching from the entire reference dataset. In this case, we have selected a cluster size of 5,000 and a grouping size of 1,000, resulting in a final pixel number that is only 0.02% of the original reference dataset. Network Design To evaluate the impact of different modules in the network, we conducted several ablation studies, as summarized in the table. Our baseline network is a U-Net network that takes only sRGB images as input. "Prior RAW" refers to the baseline network that takes both sRGB images and assembled RAW images as input. "Reliable Attention" represents the Prior RAW network with the reliable attention module to suppress unreliable areas. As shown in the ablation study, even the assembled RAW images achieve a high level of accuracy. When analyzing Fig. This can be attributed to the high-illumination and low-texture nature of these areas, making it difficult to find accurate similar pixels in the reference set. To understand the underlying cause of this phenomenon, we examined the affinity matrix for both the areas with high error values and those with low error values. We found that the maximum values of the affinity matrix for the low-error areas are approximately 0.9, while for the high-error areas, they fall below 0.6. Based on this finding, we attribute this discrepancy to the lack of similar pixels in the reference dataset for the high-error areas, leading to the mismatching problem. Consequently, the Pixel Searching Encoder is unable to find suitable references for these pixels, resulting in the observed discrepancies.We have named our final method PriorRAW, which incorporates the steps of prior metadata generation, metadata fusion, and the Encoder-Decoder network. As shown in Table 2 , PriorRAW achieves state-of-the-art performance overall and significantly improves the performance of assembled RAW images. From the visualization, it is evident that the high error values in the assembled RAW images are greatly reduced. This indicates that the neural network has learned to address the error caused by the mismatched pixels problem and has achieved improved overall performance by combining information from sRGB inputs and the prior metadata. Compared to metadata-based methods, we have observed that errors in metadatabased methods are concentrated in high-frequency regions. This is because the sampled metadata is unable to accurately recover the original pixels in areas with rich details. However, our method does not have this issue. Instead, the error areas of our model are concentrated in highlights areas, which suffer from the most severe information loss in the sRGB image and are therefore difficult to recover the original RAW pixels.The proposed method for RAW reconstruction based on prior metadata can also be utilized as a versatile image reconstruction pipeline. In order to assess its effectiveness in tasks beyond RAW reconstruction, we choose to evaluate our methods through a bit-depth recovery task. Bit-depth recovery involves reconstructing a high bitdepth image from a low bit-depth version, with the aim of restoring finer tonal details and reducing quantization artifacts. This process plays a crucial role in enhancing image quality by expanding its dynamic range and preserving subtle nuances that may be lost in lower bit-depth variations. To accomplish this, we have selected two widely-used datasets, namely MIT-Adobe 5K [4] and Sintel [3] , for training the 4-to-8-bit recovery model. Then, we choose the representative Kodak [1] dataset for evaluation purposes.As shown in Table 3 , we compare our methods with several representative algorithms for bit-depth recovery. By incorporating the prior metadata extracted from high-bit depth data, our model with the fusion module and encoder-decoder structure significantly outperforms state-of-the-art bit-depth recovery algorithms, achieving a PSNR of 39.79. 
conclusion: 
 In this paper, we focus on efficiently reconstructing RAW images using sRGB inputs. Unlike previous methods, we propose a new reconstruction pipeline based on prior metadata. Our approach is based on the idea that pixels captured under similar lighting conditions for similar objects can provide valuable prior information for reconstructing the current pixel. To achieve this, we design a contrastive learning method to train the Pixel Searching Encoder, enabling it to efficiently identify the most similar pixels in the reference dataset. We also develop a clustering and grouping algorithm to compress the reference features and sRGB-RAW pixel pairs into a smaller scale during the inference phase, while maintaining performance. Finally, using the assembled prior metadata and the sRGB input, we design a fusion module and an Encoder-Decoder network to reconstruct the final RAW images. Experiments show that our method achieves excellent performance on the RAW reconstruction task and can be used for similar tasks like bit-depth recovery. Our goal is to contribute to a wider range of reconstruction tasks, enhancing their applicability. 
raw srgb: 
 Meta-data Reconstructor (a) Methods based on metadata introduce additional computational burden to the mobile camera. (b) Our method does not require the camera to provide additional information. Instead, it searches for similar pixels from the reference RAW images as a reference.detection [13, 27, 38] . However, the high cost of collecting RAW data limits its application in many scenarios. On the one hand, the detailed information preserved in RAW images results in a much larger file size than their corresponding sRGB images, which can impose significant storage and bandwidth burdens on cameras when capturing large amounts of data. On the other hand, most labeled data for computer vision tasks are based on sRGB data without corresponding RAW data, which limits the applicability of RAW data in many perception tasks.To address these limitations, several methods have been proposed to reconstruct the RAW images based on their corresponding sRGB images. One approach involves using a deep neural network to de-render the RAW images, using only the sRGB data as input [9, 37, 42] . These methods are convenient as they don't require any additional inputs besides the sRGB information. However, they may suffer from relatively lower performance since the ISP processing removes detailed information from the original RAW data. It is challenging for deep neural networks to perfectly recover these dropped details without proper reference. Another approach is to use extra metadata as a reference to improve the accuracy [17, 24, 28, 33] . These methods employ a sampler in the camera to sample some RAW pixels into the metadata. When derendering the RAW data, another reconstructor is used, which takes the sRGB images and corresponding metadata as input to produce the original RAW images, as shown in Fig. 1(a) . These methods achieve good performance in terms of the quality of the predicted RAW images. However, the sampler running in the camera device adds an additional computational burden, resulting in O (𝑊 𝐻 ) computation and space complexity, where 𝑊 and 𝐻 represent the width and height of the RAW image. Also, these methods cannot be used to recover existing sRGB images without metadata. This limitation restricts the use of existing sRGB datasets for RAW-based computer vision tasks.In contrast, we propose to search for similar pixels in existing RAW images as a reference, instead of relying on online metadata sampling when capturing new photos. Therefore, our method does not introduce any additional computation and storage on the camera device. In other words, the complexity of our method on the camera is 0. While each photo is unique and distinct from all others, at the pixel level, we can identify similar pixels that were captured under similar lighting conditions and depict similar objects. Although these pixels may not be an exact match for the target pixel, they can provide valuable prior information for the deep neural network. Therefore, we utilize these pixels as references and refer to them as Prior Metadata, as shown in Fig. 1 To achieve this goal, we first train a pixel searching model using contrast learning. This model takes sRGB data as input and generates an embedding vector for each pixel in the image. We then minimize the vector distance if the corresponding RAW pixels are similar and increase the distance if the RAW pixels are different. Using this model, we can identify the most similar RAW pixels in the reference dataset and assemble prior RAW images using these pixels. Based on the prior RAW images, we build a reconstructor based on the prior RAW images and the corresponding sRGB images. However, when the reference dataset is very large, the searching procedure becomes computationally and memory intensive. On the other hand, if the reference dataset is too small, it may not contain enough pixel variety to build the prior RAW images. To overcome this challenge, we design a clustering and grouping algorithm that selects several representative pixels from the entire reference dataset to replace the actual reference dataset. Based on our analysis and experiments, we have found that sampling around 5,000 pixels can effectively represent the entire dataset and achieve similar performance. This accounts for only about 0.02% of the total pixels in the reference dataset. Experiments demonstrate that our method achieves impressive RAW construction accuracy. Even without the online metadata sampled from the target RAW images, our method can effectively construct the RAW images compared to state-of-the-art online metadata-based methods. In summary, our contribution can be summarized in four aspects:• We propose a new pipeline for extracting prior metadata from the existing RAW images and employing it as a reference for RAW reconstruction. • We propose a contrastive training approach to train the Pixel Searching Network. This network enables the search for the most similar RAW pixels in the reference dataset, using features extracted from sRGB images. • We propose a prior metadata clustering and grouping algorithm that can identify the most representative pixels for the reference dataset, thus reducing the search cost.• We propose a RAW Reconstructor with a Prior Metadata Fusion Module. This approach efficiently utilizes reliable information encoded in the prior metadata while discarding unreliable information. The model achieves excellent performance in RAW reconstruction, outperforming metadatabased methods that require per-image metadata extracted from the target RAW. 
 
 Appendix:  


Decision: The response from the authors did not change the initial ratings: Weak Accept, Borderline Accept, Borderline Reject.

After carefully reading the paper, the reviews and the response the Meta Reviewer agrees with them that the paper's topic is of interest, the solution is novel and achieves promising performance, and that the paper covers well limitations. 
At the same time, "the proposed method might be sensitive to the selection of hyperparameters and the reference dataset" (Reviewer n538) and the authors' response provides information that should be in the paper.

The strengths outweigh the weaknesses and the Meta Reviewer agrees that the paper's contributions are worthy publication and invites the authors to add "discussions regarding the experimental settings and new results"(Reviewer DuxP) to the camera ready.

Reviews: review: paper_summary: paper_summary: This paper focuses on transferring sRGB images back to their corresponding RAW images with reference to existing RAW images, which is different from previous studies that solely use sRGB images or require additional sampling procedures in the camera devices. The proposed method contains three stages: pixel search encoder, prior information compression, and prior MetaData-based reconstructor. The pixel search encoder aims to learn an effective correlation matrix between the target and reference images. To improve the computational efficiency, it uses the K-Means algorithm to extract representatives prior to serving for RAW image reconstruction. In the third stage, it reconstructs the RAW images based on the pre-trained pixel search encoder and prior metadata. The paper demonstrates the effectiveness of the proposed method on the NUS dataset which achieves promising performance.
strength_weakness: strength_weakness: 1. This paper clearly discusses the limitations of the existing methods and the structure of this paper is easy to follow.

2. The paper comprehensively evaluates the performance of the proposed method using different numbers of clusters and shows that it significantly affects the performance.

3. The visual comparison is straightforward and convincing.
limitations: limitations: 1. This method is a reference-based method, which requires that the reference images should be similar to the target images. How to choose the reference images for each target sRGB image?

2. It is hard to understand why the pixel-searching encoder uses contrastive learning. It seems that the raw version of the target sRGB is used to calculate the l2 loss. And, why does this method automatically learn the similar and dissimilar pixels?

3. The paper shows that the number of clusters significantly affects the performance. How to select these hyper-parameters for general sRGB images? Or are these parameters specifically set for each sRGB image?
paper_summary: paper_summary: The paper proposes to a method to reconstruct the raw image from sRGB image. By using Prior Meta as a reference to reconstruct the RAW image, the proposed method achieves slightly better perforance than some of the existing metadata-based models.
strength_weakness: strength_weakness: - The topic of raw image reconstruction from sRGB is interesting.
- It seems to have better performance than some metadata-based raw image reconstruction methods.
limitations: limitations: - What’s the computational cost of training/inference? How about number of parameters, FLOPS?
- The key challenge of the raw image reconstruction cannot be solved in this work, making the proposed setting less practical. A key challenge is that the process from raw image to sRGB is a lossy process and there exists inevitable serious information loss in low light and highlight areas. However, without using metadata cannot recover the lost details.
- How about the performance of the model compared with SOTA metadata method, e.g., [a]. Besides, SAM et.al. utilize 16bit sRGB image as input, which is less practical. How about the performance using 8-bit sRGB/8-bit JPEG images?
- It is a little confusing why w/ and w/o Prior Metadata can have such a huge improvement, e.g., 6db according to the ablation study. The prior metadata usually can be learned into neural networks and using a codebook usually cannot obtain such a huge improvement.

[a] https://github.com/wyf0912/R2LCM
paper_summary: paper_summary: The purpose of this work is to reconstruct raw images from sRGB data in a more efficient way. Specifically, this work utilizes Prior Meta as a reference to reconstruct the RAW images, rather than using per-image metadata directly. To achieve this goal, it designs a pixel searching network to find the most similar pixels in the reference RAW images and compresses the reference images. Afterwards, it employs reconstructor to reconstruct RAW images.
strength_weakness: strength_weakness: 1. This work uses Prior Meta as a reference to reconstruct the RAW images, rather than using per-image metadata directly, which is more efficient than previous methods.
2. This work proposes a pixel searching network to search for the most similar RAW pixels in the reference dataset and designs a clustering and grouping algorithm to identify the most representative pixels.
3. This work proposes a raw reconstructor with a prior metadata fusion module to achieve better performance.
limitations: limitations: 1. It might be better to give more experimental results compared with previous work, especially visual comparison results.
2. Since the motivation of this work is to provide a more efficient reconstruction method, it may be more convincing to provide more comparisons of the cost of model training and inference with previous work, especially the amount of computation, time, etc.
3. Does the network model need to be quantified when deployed in practice? If quantization is required, how much does the performance drop?