Self-Supervised High Dynamic Range Imaging with Multi-Exposure Images in Dynamic Scenes
Authors: Zhilu Zhang
Haoyu Wang
Shuai Liu
Xiaotao Wang
LEI LEI
Wangmeng Zuo
Venue: ICLR 2024 poster (2024)
Keywords: High Dynamic Range Imaging; Self-Supervised Learning; Multi-Exposure Images
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We take advantage of the internal characteristics of the multi-exposure images to propose a self-supervised HDR reconstruction approach SelfHDR, which achieves comparable performance to supervised ones.

Abstract: Merging multi-exposure images is a common approach for obtaining high dynamic range (HDR) images, with the primary challenge being the avoidance of ghosting artifacts in dynamic scenes. Recent methods have proposed using deep neural networks for deghosting. However, the methods typically rely on sufficient data with HDR ground-truths, which are difficult and costly to collect. In this work, to eliminate the need for labeled data, we propose SelfHDR, a self-supervised HDR reconstruction method that only requires dynamic multi-exposure images during training. Specifically, SelfHDR learns a reconstruction network under the supervision of two complementary components, which can be constructed from multi-exposure images and focus on HDR color as well as structure, respectively. The color component is estimated from aligned multi-exposure images, while the structure one is generated through a structure-focused network that is supervised by the color component and an input reference (\eg, medium-exposure) image. During testing, the learned reconstruction network is directly deployed to predict an HDR image. Experiments on real-world images demonstrate our SelfHDR achieves superior results against the state-of-the-art self-supervised methods, and comparable performance to supervised ones. Codes are available at https://github.com/cszhilu1998/SelfHDR

Published as a conference paper at ICLR 2024
SELF-SUPERVISED HIGH DYNAMIC RANGE IMAGING WITH
IMAGES IN DYNAMIC SCENES
Zhilu Zhang', Haoyu Wang', Shuai Liu, Xiaotao Wang, Lei Lei, Wangmeng Zuo
1 Harbin Institute of Technology, Harbin, China
cs eszlzhang@ outtng@ooutoo wook. com, cshy2002@gmail com, wmzuo8hit. cn
ABSTRACT
Merging multi-cxposune images is a common approach for obtaining high dynamic range (HIR) images, with the primary challenge being the avoidance of ghosting artifacts in dynamic scenes. Recent methods have proposed using deep neural networks for deghosting. However, the methods typically rely on sufficient data with HDR ground-truths, which are difficult and costly to collect. In this work, eliminate the need for labeled data, we propose SelfHDR, a sufe-supee HDR reconstruction method that only requires dynamic multi-exposure images during training. Specifically, SelfHDR learns reconstruction network under the supervision of two complementary components, which can be constructed from multi-exposure images and focus on HDR color as well as structure, respectively. The color component is estimated from aligned multi-exposure images, while the structure one is generated through a sennettrr-fooussee network that is supervised by the color component and input reference (e.g., medium-exposure) image. During testing, the learned reconstruction network is directly deployed to predict HDR image. Experiments on world images demonstrate our SelfHDR achieves superior results against the state-of-the-ar self-supervised methods, and comparable performance to supervised ones. Codes available at https: 9
//github. com/
INTRODUCTION
Scenes with wide brightness ranges often visible to human observers, but capturing them com- pletely with digital or smartphone cameras can be arduous due to the restricted dynamic range of sensors. For instance, during sunset, the sun and sky are substantially brighter than the surrounding landscape, leading the camera sensor to either over-expose the sky or under-expose the landscape. To obtain high dynamic range (HDR) photos in these conditions, exposure bracketing technology becomes a popular option. It captures multiple low dynamic range (LDR) images with varying exposures, which are then merged into an HDR result (Debevec & Malik, 2008; Mertens al., 2009).
Unfortunately, when the multi-exposune images are misaligned due to camera shake and object movement, ghosting artifacts may exist in the result. Traditional methods to remove the ghosting include rejecting misaligned areas (Zhang & Cham, 2011; Lee et al., 2014; Oh et al., 2014; Yan et al., 2017), aligning input images (TOMASZEWSKA, 2007; Hu et al., 2013; Yan et al., 2019b), and using patch-based composite (Sen al., 2012; Hu al., 2013; Ma al., 2017). With the development of deep learning (He et al., 2016; Dosovitskiy et al., 2020; Liu et al., 2021), recent advances (Kalantari al., 2017; Wu al., 2018; Yan et al., 2019a; Liu et al., 2022; Yan al., 2023a; Tel et al., 2023) proposed training deep neural networks (DNN) for deghosting in data-driven supervised manner, performing more effectively than traditional ones.
However, DNN-based HDR reconstruction methods usually require sufficient labeled data, each of which should include the input dynamic multi-exposure images and the corresponding HDR tround-truth (GT) image. In order to ensure position alignment between the input reference (e.g.. medium-exposure) frame and GT, previous works (Kalantari al., 2017; Chen et al., 2021: Liu et al., 2023) generally capture the dynamic inputs with the controllable object (generally a person)
"Corresponding author.

---

Published as a conference paper at ICLR 2024
motion in static background, and construct GT by merging static multiple-exposeee images of the reference scene. Such a collection process is cumbersome and involves high time as well as labor costs, thus limiting the number and diversity of the datasets. To alleviate the need for labeled data, FSHDR (Prabhakar et al., 2021) explores a few-shot manner, and Nazarczuk et al. (Nazarczuk et al., 2022) introduce fully self-supervised approach. The main idea is construct pseudo-inputs and pseudo-targets for HDR reconstruction. Nevertheless, their performance is unsatisfactory, as the motion and illumination in synthetic LDR images exhibit gaps with real-world ones.
In this work, we aim to reconstruct HDR images directly with real-world multi-exposure images in self-supervised manner, without synthesizing any pseudo-inpur data. This objective should be feasible, as most of the information required for HDR results can derive from input data. The property will be more intuitive when HDR color and structure are observed, respectively. Specifically, HDR color knowledge can be inferred from aligned multi-exposure images, and HDR structure can be extracted from some one of the inputs.
We further propose SelfHDR, self-supervised method for HDR image reconstruction. Inspired by the above data characteristics, SelfHDR decomposes the latent HDR GT into available color and structure components, and then takes them to supervise the learning of the reconstruction network. On the one hand, the color component is estimated from multi-exposure images aligned by optical flow. On the other hand, the structure component is generated by feeding aligned inputs into structure- focusee network, which learned under the supervision of the color component and an input reference (e-g-, medium-exposure image. Moreover, during the training phase of structure- focused and reconstruction networks, elaborate masks are embedded into loss fuucctions to circumvent harmful information in supervision. During inference, only the reconstruction network required to predict the HDR result from unseen multi-exposure images.
We evaluate the proposed sulf--uper methods using four existing HDR reconstruction networks, respectively. The models are trained on Kalantari et al. dataset (Kalantari et al., 2017), and tested on multiple datasets (Kalantari al., 2017; Sen al., 2012; Tursun al., 2016). The results show our SelfHDR obtains 1.58 dB PSNR gain compared to the state-of-the thet-httrrrtr art self-supervised method that uses the same reconstruction network, and achieves comparable performance to supervised ones, especially in terms of visual effects. Besides, we conduct extensive and comprehensive ablation studies, analyzing the effectiveness of different components and variants.
To sum up, the main contributions of this work include:
We propose self-supervised HDR reconstruction method named SelfHDR, which learns an HDR reconstruction network by decomposing latent ground-truth into constructible color and structure component supervisions.
The color component is estimated from aligned multi-exposure images, while the structure one generated using a cruuctur-ffocuset network supervised by the color component and input reference image.
Experiments show that our SelfHDR outperforms the offte-of-the-rr self self-supervised meth- ods, and achieves comparable performance to supervised ones.
2
RELATED WORK
2.1 SUPERVISED HDR IMAGING WITH MULTI-EXPOSURE IMAGES
The main challenge of HDR imaging with multi multi-exposure images is avoid ghosting artifacts. DNN-based HDR deghosting methods have exhibited more satisfactory ability than traditional ones. For the first time, Kalanrati el al. (Kalantari et al., 2017) collect a real-world dataset and propose a data-driven convolutional neural network (CNN) approach merge LDR images aligned by optical flow. Wu et al. (Wu et al., 2018) utilize the multiple encoders and one decoder architecture to handle image misalignment, discarding the optical flow. Yan al. (Yan al., 2019a) present spatial attention mechanism for deghosting. In addition, we recommend Wang et al. 's survey (Wang & Yoon, 2021) for more CNN- HDR reconstruction methods (Prabhakar al., 2019; Niu et al., 2021).
Recently, with the development of Transformer (Dosovitskiy et al., 2020; Liu et al., 2021), some works (Liu al., 2022; Song et al., 2022; Yan al., 2023a; Tel et al., 2023) bring in self- and
2

---

Published as a conference paper at ICLR 2024
cross- attention to alleviate the ghosting artifacts. For example, Liu al. (Liu et al., 2022) propose Tranffranforrrr which embeds a local context extractor into SwinIR (Liang et al., 2021)
1 basic block for jointly capturing global and local dependencies. Song et al. (Song et al., 2022) suggest selectively applying the transformer and CNN model to misaligned and aligned areas, respectively. However, both CNN- and Transformer-basee methods require sufficient labeled data for training networks, while the data collection is time-consuming and laborious.
2.2 FEW-SHOT AND SELF-SUPERVISED HDR IMAGING WITH MULTI- IMAGES
To alleviate the reliance on HDR ground- few-shot and self-supervised HDR reconstruction methods have been explored. FSHDR (Prabhakar et al., 2021) combines unlabeled dynamic samples with few labeled samples to train a neural network, then leverages the model output of unlabeled samples
Aseudo-HDR to generate pseudo-LDR images. Ultimately the HDR reconstruction network is learned with synthetic pseudo-pairs. Nazarczuk et al. (Nazarczuk et al., 2022) select well-exposure LDR patches as pseudo-HDR generate pseudo-LDR, while the static LDR patches are directly merged for HDR ground-truths. However, due to unrealistic motion and illumination in synthetic LDR images, such methods exhibit performance gaps compared to supervised ones. Recently, SAME (Yan et al., 2023b) generates saturated regions in a ceffssupervieed manner first, and then performs deghosting via a semi-supervised framework. But it still has limited performance improvement. In this work, we take full advantage of the internal characteristics of multi-exposure images to present self-supervised approach SelfHDR, which achieves comparable performance to supervised ones.
Furthermore, some works incorporate emerging techniques to investigate self-supervised HDR reconstruction. For instance, GDP (Fei et al., 2023) exploits multi-exposure images to guide the denoising process of pre-trained diffusion generative models (Ho et al., 2020; Song et al., 202 I), reconstructing HDR image. Mildenhall et al. (Mildenhall et al., 2022), Jun et al. (Jun-Seong et al., 2022), and Huang et al. (Huang et al., 2022a) employ NeRF (Mildenhall et al., 2020) to synthesize HDR images and the novel HDR views. However, these methods are less practical, since the specific models need to be re-optimized when facing new scenarios.
3 METHOD
3.1 MOTIVATION AND OVERVIEW
Revisit Supervised HDR Reconstruction. The combination of multi-exposure images enables HDR imaging in scenes with a wide range of brightness levels. In static scenes, the HDR image can be easily generated through weighted sum of multi-exposure images (Debevec & Malik, 2008). However, when applying this approach in dynamic scenes, it will lead to ghosting artifacts. As result, several recent works (Kalantari et al., 2017; Yan et al., 1019;; Liu et al., 2022; Tel et al., 2023) have suggested learning deep neural network in supervised manner for deghosting. Concretely, denote the LDR image taken with exposure time &; by I, where i = 1, 2, 3 and t < f2 < tz. They first map the LDR images to the linear domain, which can be written as,
H = 1/44
where y denotes the gamma correction parameter and is generally set to 2.2. Then they concatenate I, and H,, feeding them to the reconstruction network R with parameters OR, i.e.
Ý = R(X, X2, X3; OR),
where X, (I, H.), Ý denotes the reconstructed HDR image. The optimized network parameters can be obtained by the following formula,
+
arg min ((TÍ),TT())) oR
where L represents the loss function, denotes the HDR GT image. T is the tone- maa-mapping process, represented as,
T(Y) = log(1 + +Y)
where = 5, 000.
(4)
log(1 + u))
3

---

Published as a conference paper at ICLR 2024
Motivation of SelfHDR. The acquisition of labeled data for HDR reconstruction is usually time- consuming and laborious. To alleviate the requirement of HDR GT, some works (Prabhakar et al., 2021; Yan et al., 2023b; Nazarczuk et al., 2022) have explored few-shot and zero-shot HDR recon- struction by constructing pseudo-pairs. However, their performance is unsatisfactory due to the gaps between the simulated pairs and real-world ones, especially in a fully self-supervised manner.
In this work, we expect to get rid of the demand for synthetic data, achieving self seff-supee HDR reconstruction directly with real-world dynamic multi-exposure images. The goal should be feasible, as the multi-exposure images have provided probably sufficient information for HDR reconstruction. The property will be more intuitive when the color and structure are observed, respectively. On the one hand, the color of HDR images can be estimated from aligned inputs. On the other hand, the structure information the HDR images can be generally discovered in some one of multi-exposure images, Le., most textures exist in the medium-exposur image, dark details are obvious in the high-exposure one, and bright scenes are clearly visible in the low-exposure one.
What we need to do is to dig for the right information from the multi-exposure images for constructing the HDR image. However, it is actually difficult to explore a straightforward self-supervised method that generates HDR images directly. Considering the above properties of HDR color and structure, we treat the two components respectively for case self-supervised implementation. Note that it can be a focus emphasis on color and structure relatively, not necessarily absolute separation.
Specifically, when training a self-supervised HDR reconstruction network with given multi-cxposure images input, suitable supervision signals have be prepared. Instead of looking for complete HDR image, we construct the color and structure components of the supervision respectively (see Sec. 3.2). Then we learn the network under the guidance of both components (see See. 3.3).
3.2
CONSTRUCTING COLOR AND STRUCTURE COMPONENTS
3.2.
CONSTRUCTING COLOR COMPONENT
The color component should represent the HDR color as faithfully as possi- ble, and can be estimated by fus- ing the aligned Multiple frames in dynamic scenes are generally not aligned caused by cam- era shake or object motion. Although sometimes a few regions are aligned well, they are not enough to generate acceptable color components. In view of the effective capabilities of the optical flow estimation method (Liu et al., 2009), it a natural idea to perform pre-alignment first. Concretely, taking the medium-exposure image I2 as the reference, we calculate the optical flow from I2 to I and 13, respectively. Thus, we can back warp H, and H3 according to the calculated optical flow, obtaining H1 and H3 that are roughly aligned with H2. Then we can predict the color component Yoolor with the following formula,
(1))
A,(h)
images.
12
1
12
Figure I: The triangle function that we use as the blending weights to generate color components.
H, + A,H, + AgH.
Yostor
(5)
A + A As
where A, represents pixel-level fusion weight. We follow Kalantari et al. (Kalantari et al., 2017) and express A, as,
= = (1 = Ag(2,, 1==1-aa(b))
(6)
where (J2) is shown in Fig.
When the images are perfectly aligned, the color components Yealor can be regarded as an HDR image directly. However, such an ideal state is hard to reach due to object occlusion and sometimes non-robust optical flow model. Small errors during pre-alignment may cause blurring, while large ones cause ghosting in color components. Nevertheless, regardless of the ghosting areas, the rest can record the rough color value, and in which well-aligned ones can offer both good color and structure cues of HDR images. Morcover, for the areas with alignment errors, we further construct structure components to guide the reconstruction network in the next subsection.
4

---

Published as a conference paper at ICLR 2024
Structure-Preserving Loss (Bqn. (7)
Training Stage Preparing for Supervisions
Testing
Structure-
Structure-Expanee
Focused Network
Loss (Bqn. (8))
Optical Flow
+
Merge + (Eqn. (5)
11 X2 x
Alignment
Reconstruction Network
cotor
Training Stage Learning HDR with Constructed Supervisions
Color Mapping Loss (Eqn. (13))
Reconstruction Network
Structure Mapping Loss (Eqn. (15))
Structure- Y
Optical Flow Alignment
Focused
Trainable
X1 2 Xg
Network
Frozen
Figure 2: Overview of SelfHDR. During training, we first construct color and structure components (i.e., Yostor and stru), then take Yostor and Yarru for supervising the HDR reconstruction network. During testing, the HDR reconstruction network can be used to predict HDR images from unseen multi-exposure images. Dotted lines with different colors represent different loss terms.
3.2.2 CONSTRUCTING STRUCTURE COMPONENT
Although the medium expooure image can provide most of the texture information, it is not optimal to take it as the only structure guidance for the HDR reconstruction, as the dark areas may be unclear and over-exposed ones may exist in it. Besides, it not easy to put into practice when using the low-exposure and high-exposure images guidance, due the position and color differences between the HDR image and them. Fortunately, the previously constructed color component Yostrr can preserve the structure of dark and over-exposed areas to some extent. Therefore, we combine the medium-exposure image and color component Ycolor to help construct the structure component.
Concretely, first learn structure-focused network with the guidance of medium-exposure image and color component Yooerr During training, the network takes the multi-exposure images as input, as shown Fig. 2. On the one hand, the medium eexpouue image guides the network to preserve well-exposed textures from the input reference image. It is accomplished by mnneture-preseevingg loss Cxp. which can be written as,
Esp H2) = (I(TT(L T(H2)) * M
where Ystru denotes the network output. Map emphasizes the well-exposed areas, and mitigates the adverse effects of dark and over-exposed ones in the reference image H2. The function (I2) (see Fig. 1) can do just that, so we set Map to A2(I2). On the other hand, the color component Yeotor guides the network to learn the structure from non-reference images by calculating structure- expansion loss Lee, which can be written as,
Lee V Yatrus Yeator) = (((T() T(Ycotor) Mselii
(8)
Mae is a binary mask, distinguishing whether the pixel of Ycolor is composited from well well-aligned multi-exposure ones. We design each pixel M of Mse as,
M {
(((T(Yealor) - T(H2)) A2 < asc (((T(Ynntor) T(H2)) A2(I2))" 2 asc
(9)
where ase is threshold and set to 5/255. In short, the parameter θş of structure-focused network S jointly optimized by structure- aructure-preserving and structure-expansion loss terms, i.e.,
O arg min (Y,tru: Yestor) + Asp (ep YYtrus Hz)),
(10)
0g
where Aep denotes the weight coefficient of gructtrr-pressevvve loss and is set to 4.
5

---

Published as a conference paper at ICLR 2024
Then, feed aligned multi-exposure images rather than original ones into the pre-trate pre-trained structure- focused network S. The final structure component Yatru can be expressed as,
str =
x3:
(11)
where X1 and X3 denote aligned X1 and X3 with the reference of X₂. Such an operation can help meucture-focuser networks reduce the alignment burden, thus further enhancing the structure component. In addition, benefiting from the supervision of the color component, the structural component Y Ysuru also has some color cues, although it mainly focuses on the HDR textures.
3.3 LEARNING HDR WITH COLOR AND STRUCTURE COMPONENTS
With the color and structure components guidance, train an HDR reconstruction network R. without other ground-truths. The optimized network parameters O can be modified from Eqn. (3) the following formula,
R arg min(Ccotor (Y, Y Yeolor) + Astru (Y, Ystru)]s
(12)
where Y represents the network output. Leotor and Cutru denote color mapping and structure mapping loss terms, respectively. Astru is the weight coefficient of Ssrru and is set to
For color mapping term, we adopt & loss, which can be written as,
Leolor catoor) = II( - T(Y * Mcator
(13)
where Mostor similar as Mae. and also a binary mask. excludes areas where optical flow is estimated incorrectly when generating Yeolor Instead of using Eqn. (9), here we can utilize Ystru to design a more accurate mask, which can be expressed as,
((o(oool T(Y ) < Ccolor M Cotorr {
(14)
((T(Yestor) - ((yttr)) 2 acolor
where p denotes a pixel, a cotor is a threshold and set to 10/255. For structure mapping term, we adopt VGG-based (Simonyan & Zisserman, 2015) perceptual loss, which can be written as,
Cstru(Y Ystra) = Σ (Tx)) dx
(15)
where (b)()) denotes the output of k-th layer in VGG (Simonyan & Zisserman, 2015) network.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Framework Details. Note that this work does not focus the design of network architectures, and we employ existing ones directly. The cructure-focuseee and reconstruction networks use the same architecture. And we adopt CNN-based (i.e., AHDRNet (Yan al., 2019a) and FSHDR (Prabhakar et al., 2021)) and Transformer-basee (i.e., HDR- TooR-Transformer (Liu et al., 2022), and SCTNet (Tel al., 2023)) networks for experiments, respectively. Besides, the optical flow calculated by Liu er al. (Liu et al., 2009)'s approach, as recommended in (Kalantari et al., 2017; Prabhakar et al., 2021).
Experiments are mainly conducted on Kalantari et al. dataset (Kalantari et al., 2017). which are extensively utilized in previous works. The dataset consists of 74 samples for training and 15 for testing. Fach sample comprises three LDR images, captured at exposure values of (-2, 0, 2} or (-3, 3). alongside corresponding HDR GT image. We use these testing images for both quantitative and qualitative evaluations. Additionally, following (Kalantari et al., 2017; 7; Yan et al., 2019a; Liu et al., 2022), take the Sen et al. (Sen et al., 2012) and Tursun et al. (Tursun et al., 2016) datasets (without GT) for further qualitative comparisons.
Datasets.
and reconstruction networks trained successively, and share the same settings. The training patches of size 128 x 128 are randomly cropped from the original images. The batch size is set to 16. Adam (Kingma & Ba, 2015) with B1 0.9 and 33 0.999 is
Training Details. The structure-focuut

---

Published as a conference paper at ICLR 2024
Table I: Quantitative results on Kalantari et al. dataset (Kalantari al., 2017). "SelfHDRnetworf denotes the reconstruction network we use, i.e., AHDRNet ( (Yan et al., 2019a), FSHDR (Prabhakar al., 2021), TDRR-Trasffommr (Liu al., 2022), and SCTNet (Tel et al., 2023). The best results in each category are bolded.
Method
PSNR-u SSIM-u
PSNR-//SIMM
HDR-VDP-2
AHDRNet (CVPR 2019)
43.63 / 0.9900
41.14/0.9709
64.61
Fully-
FSHDR (CVPR 2021)
43.03 0,9902
42.27 0.9889
64.79
Supervised
HDR- Transformer (ECCV 2022)
44.21 / 0.9918
42.17 0 0,9889
64.63
SCTNet (ICCV 2023)
44.48/ 0.9916 16
42.00 0.9897
64,47
Few (K)-
FSHDR (CVPR 2021)
43.02 0.9874
41.98 0.9885
64.54
Shot
FSHDR (CVPR 2021)
42.52/ / 0.9846
41.92 0.9887
64.41
FSHDR (CVPR 2021)
42.17 / 0,9828
41,47 / 0.9884
64,21
Nazarezuk et al. (ArXiv 2022)
42.15 /
40.54 /
63.99
Self- Supervised
Our SelfHDD A DeNN
43.68 / 0.9901
41.09 0.9873
64.57
Our SelfHDR FSHD
43,80 0.9902
41.72 0.9880
64.57
Our SelfHDR# DRETrDRoDRRT Transforreer
43.94 0.9907
41.79 0.9883
64.98
Our ScTf
43.95 0.9907
41.77 0.9889
64.77
taken to optimize models for 150 epochs. The learning rate is initially set 1 10 for CNN-based networks and 2 x 10-4 for Transformer-based ones, and reduces by half every 50 epochs.
Evaluation Configurations. We PSNR and SSIM (Wang al., 2004) evaluation metrics. PSNR and SSIM are both calculated on the linear and tone-mapped domains, denoted as '-I' and '-u', respectively. Moreover, we adopt HDR-VDP-2 (Mantiuk al., 201 1) that measures the human visual difference between results and targets. The higher HDR-VDP-2, the better results.
4.2 COMPARISON WITH STATE-OF-THEE-RRY
As described in Sec. 4.1, we adopt four existing HDR reconstruction networks (i.e., AHDRNet, FSHDR, HDR- Tran-frrasf,rmer, and SCTNet) for experiments, respectively. We compare them with the corresponding supervised manners and two self-supervised methods (i.e., FSHDR and Nazarezuk et al. (Nazarczuk al., 2022)). And the results of few-shot FSHDR also provided.
Quantitative Results. Table 1 shows the quantitative comparison results. As loss functions are always calculated tone-mapped images, and HDR images typically viewed on LDR displays, we suggest taking evaluation metrics in the tone-mapped domain (i.e., PSNR-u and SSIM-u) as the primary reference. From the table, four SelfHDR versions all outperform the previous supe-supe methods. Especially, with the same reconstruction network, our SelfHDR FSHDR achieves 1.58 dB PSNR gain than FSHDR K= The results of SelfHDR can be further improved with the of more advanced reconstruction networks (i.e., HDR- Transformer and SCTNet). Moreover, in comparison with the corresponding supervised methods, SelfHDR has comparable performance overall.
Qualitative Results. The visual comparisons on Kalantari et al. dataset (Kalantari et al., 2017) as well Sen et al. (Sen al., 2012) and Tursun al. (Tursun al., 2016) datasets shown in Fig. 3 and Fig. 4, respectively. Our results have fewer artifacts than FSHDR and sometimes even outperform the corresponding supervised methods. They show the same trend as the quantitative ones. Please see Sec. E in the appendix for more results.
5 ABLATION STUDY
The ablation studies are all conducted using AHDRNet (Yan et al., 2019a) as the structure-focused and reconstruction network.
5.1 EFFECT OF COLOR AND STRUCTURE SUPERVISION
The quantitative results of color and structure components (Yootor and Y, stru) given in Table 2. From the table, the final HDR results achieve better performance than both Yostor and Yarro on PSNR-u, SSIM-u, and
It indicates that the two components complementary and
7

---

Published as a conference paper at ICLR 2024
LDRs
Our Sen
LDR patches
PSHDR
SellHD
AlIDRNet
HDR Tra. SelAIDR
SCTNet
Se
Figure 3: Visual comparison on Kalantari et al. dataset (Kalantari et al., 2017). Red and blue arrows indicate areas with ghosting artifacts from other methods. 'HDR-Tra.' denotes HDR-Transforme
LDBt
Our SeittDD
LDR patches
LDDRs
Our RecriD
LDR patches
PSUDR
ScllIDR.Gwox
SellHDReeeor
DSEDR,
ESIDR
SeHfIDR,ccum
SCTKM
(a) Vision comparison on Sen et all dataset.
(b) Vision camparison Tursun dataset.
Figure 4: Visual comparison on (a) Sen er (Sen et al., 2012) and (b) Tursun al. (Tursun et al., 2016) datasets. Red arrows indicate areas with poor quality from other methods.
Table Quantitative results of supervision components and final HDR images.
PSNR-u / SSIM-u
PSNR-I/ SSIM-I
HDR-VDP-2
Color Components Yestor
34.45 0,9652
39.01 0.9783
58.28
Structure Components Ystru.
43.38 0,9891
41.74 0.9874
64.48
Final HDR Images
43.68 0.9901
41.09 / 0.9873
64.57
laking them as supervision is appropriate and effective. Furthermore, we conduct the following two experiments further illustrate the effectiveness.
Comparision with Component Fusion. It may be a more natural idea to obtain HDR results by fusing the color and structure components. Here we implement that by calculating Meotor Y Yeolor (1 -M Ystru. We empirically re-adjust the hyperparameter Jcoter in Eqn. (14), but find it gets the best results when Moator O. In other words, is difficult achieve better results by fusing two components simply. Instead, our SelfHDR provides a more flexible and efficient way.
Refining Structure Component. Denote 5 by the reconstruction network output when inputting multi-exposure images aligned by optical flow (Liu et al., 2009). From another point of view, Y can be regarded as a refined structure component with higher quality. Thus, we further take Y Ycotor and Y as new supervisions to re-train a reconstruction model, while the performance does not improve. It demonstrates that Y generated by structure-focused network is already sufficient.
8

---

Published as a conference paper at ICLR 2024
Table Effect of loss terms se and Lep) when training structure-focused network.
Table 4: Effect of different Molor when
training reconstruction network.
Ysrr PSNR-u PSNR-
Ý
Y PSNR-u PSNR-I
Dee I Cxp
Moolor
PSNR-u PSNR-I
38.24/33.61
38,79 33.72
43,55 41.12
42.69 41.89
43.09 41.13
Eqn. (9)
43,59 41.02
43.38 / 41.74
43.68 441.99
Eqn. (14)
43.68 / 441.99
Table 5: Effect of the designed masks (Mse and Mp) when training structure-focuse network.
Table 6: Effect of pre-alignment process-
ing when constructing Cotorr and Y Ytrru.
Y
Y PSNR-u PSNR-I
Y
M. / Msp
Yestor Yaten
PSNR-u / PSNR-I
PSNR u PSNR-
/
38.26 33.68
38.82 33.71
35.50/ 34,95
38.29 33.65
38.86 33.82
X
1
V
41.66 40.90
X
/ V
43.26 / 41.73
43.60 41.0 41.07
V
X
43.41 / 40.76
43.38 / 41.74
43.68 41.09
/
43.68 / 41.09
5.2 EFFECT OF LOSS TERMS AND MASKS
Structure-Focused Network. The structure-focused network is trained with the supervision of color component and input reference, implementing by calculating structure-expansion loss C.e and snveture-preserviig loss Lup, respectively. Here we explore the effect of different supervisions by using Cap Lee only. From Table 3, it can be seen that Lap may play a weaker role, as mainly constrains the well-exposed areas whose structure may be also fine in Yotor- Nevertheless, combining two supervisions more favorable than using one, thus both are indispensable.
Moreover, we conduct ablation experiments the designed masks (Map sp and Maa) in loss terms. The results in Table show that the masks are competent avoiding harmful information from supervision. The visualizations of the masks given in Sec. A of the appendix.
Reconstruction Network. For training the reconstruction network, the adverse effect of ghosting regions from color supervision Yootor needs to be avoided as well. We utilize structure component to design a more accurate mask in Eqn. (14), and does show better results than Eqn. (9) from Table 4.
In addition, conduct the experiments with different hyperparameters ase (see Equ. (9)) and Ocolor (see Eqn. (14)) Sec. D of the appendix.
5.3 EFFECT OF OPTICAL FLOW PRE-ALIGNMENT
When constructing color and structure supervisions, the inputs need to be pre-aligned by the optical flow approach. Here we remove the pre-alignment processing separately to investigate its impact on the final HDR results, which shown in Table 6. From the table, pre-alignment during obtaining Yootor is crucial, and pre-alignment during obtaining Ystru can further improve performance. The corresponding quantitative results of Ycolor and Ystru can be in Sec. B of the appendíx.
CONCLUSION
By exploiting the internal properties of multi-exposure images, we propose self self-supervised high dynamic range (HDR) reconstruction method named SelfHDR for dynamic scenes. SelfHDR, the reconstruction network is learned under the supervision of two complementary components, which focus on the color and structure of HDR images, respectively. The color components are synthesized by merging aligned multi-exposure images. The structure components are constructed by feeding aligned inputs into the structure-foo focused network, which is trained with the supervision of color components and input reference images. Experiments show that SelfHDR outperforms the state-of of-th-----he-rt supervier methods, and achieves comparable results to supervised ones. The discussion method limitation and future work can be in Sec. F and Sec. G of the appendix.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is partially supported by the National Natural Science Foundation of China (NSFC) under Grant No. U19A2073.
REFERENCES
Guanying Chen, Chaofeng Chen, Shi Guo, Zhetong Liang, Kwan-Yee K Wong, and Lei Zhang. Hdr video reconstruction: A coarse-to-fine network and real-world benchmark dataset. In ICCV, 2021.
Paul E Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH, 2008.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. In CVPR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
Jun Hu, Orazio Gallo, Kari Pulli, and Xiaobai Sun. Hdr deghosting: How to deal with saturation? In CVPR, 2013.
Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In CVPR, 2022a.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In ECCV, 2022b.
Kim Jun-Seong, Kim Yu-Ji, Moon Ye-Bin, and Tac-Hyun Oh. Hór-plenoxels: Self-calibrating high dynamic range radiance fields. In ECCV, 2022.
Nima Khademi Kalantari, Ravi Ramamoorthi, et al. Deep high dynamic range imaging of dynamic scenes. ACM TOG, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICIR, 2015.
Chul Lee, Yuelong Li, and Vishal Monga. Ghost-free high dynamic range imaging via rank mini- mization. IEEE signal processing letters, 2014.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, 2021.
Ce Liu et al. Beyond pixels: exploring new representations and applications for motion analysis. PhD thesis, Massachusetts Institute of Technology, 2009.
Shuaizheng Liu, Xindong Zhang, Lingchen Sun, Zhetong Liang, Hui Zeng, and Lei Zhang. Joint hdr denoising and fusion: A real-world mobile hdr image dataset. In CVPR, 2023.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Zhen Liu, Yinglong Wang, Bing Zeng, and Shuaicheng Liu. Ghost-free high dynamic range imaging with context-aware transformer. In ECCV, 2022.
Kede Ma. Hui Li, Hongwei Yong, Zhou Wang. Deyu Meng, and Lei Zhang. Robust multi-exposure image fusion: a structural patch decomposition approach. IEEE TIP, 2017.
10

---

Published as a conference paper at ICLR 2024
Rafal Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM TOG, 2011.
Tom Mertens, Jan Kautz, and Frank Van Reeth. Exposure fusion: A simple and practical alternative to high dynamic range photography. In Computer Graphics Forum, 2009.
Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron, Nerf the dark: High dynamic range view synthesis from noisy raw images. CVPR, 2022.
Michal Nazarczuk, Sibi Catley-Chandan Ales Leonardis, and Eduardo Pérez Pellitero. Self-supervised hdr imaging from motion and exposure cues. arXiv preprini arXiv:2203. 2022.
Yuzhen Niu, Jianbin Wu, Wenxi Liu, Wenzhong Guo, and Rynson WH Lau. Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions. IEEE 2021.
Tae-Hyun Oh, Young Lee, Yu- Tu-Wing Tai, and In So Kweon. Robust high dynamic range imaging by rank minimization. IEEE TPAMI, 2014.
K Ram Prabhakar, Rajat Arora, Adhitya Swaminathan, Kunal Pratap Singh, and Venkatesh Babu. A fast, scalable, and reliable deghosting method for extreme exposure fusion. In ICCP, 2019.
K Ram Prabhakar, Gowtham Senthil, Susmit Agrawal, R Venkatesh Babu, and Rama Krishna Sai S Gorthi. Labeled from unlabeled: Exploiting unlabeled data for few-shot deep hdr deghosting, In CVPR, 2021.
Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B Goldman, and Bli Shechtman. Robust patch-based hdr reconstruction of dynamic scenes. ACM TOG, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Jou Won Song, Ye-In Park, Kyeongbo Kong, Jaeho Kwak, and Suk-Ju Kang. Selective transhdr: Transformer-basee selective hdr imaging using ghost region mask. In ECCV, 2022.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using
pyramid, warping, and cost volume. In CVPR, 2018.
Steven Tel, Zongwei Wu, Yulun Zhang, Barthélémy Heyrman, Cédric Demonceaux, Radu Timofte, and Dominique Ginhac, Alignment-frre hdr deghosting with semantics consistent transformer. In ICCV, 2023.
A TOMASZEWSKA. Image registration for multi-exposure high dynamic range image acquisition. In Proceedings of International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG), 2007.
Okan Tarhan Tursun, Ahmet Oguz Akyüz, Aykut Erdem, and Erkut Erdem. An objective deghosting quality metric for hdr images. In Computer Graphics Forum. Wiley Online Library, 2016.
Lin Wang and Kuk-Jin Yoon. Deep learning for hdr imaging: State-of-the-ae and future trends. JEEE TPAMI, 2021.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
Shangzhe Wu, Jiarui Xu, Yu- Winging Tai, and Chi-Keung Tang. Deep high dynamic range imaging with large foreground motions. In ECCV, 2018
Qingsen Yan, Jinqiu Sun, Haisen Li, Yu Zhu, and Yanning Zhang. High dynamic range imaging by sparse representation. Nearocomputing, 2017.
11

---

Published as a conference paper at ICLR 2024
Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, lan Reid, and Yanning Zhang. Attention sunddd network for ghost-free high dynamic range imaging. CVPR, 2019a.
Qingsen Yan, Yu Zhu, and Yanning Zhang. Robust artifact-free high dynamic range imaging of dynamic scenes. Multimedia Tools and Applications, 2019b.
Qingsen Yan, Weiye Chen, Song Zhang. Yu Zhu, Jinqiu Sun, and Yanning Zhang. A unified hdr imaging method with pixel and patch level. In CVPR, 2023a.
Qingsen Yan, Song Zhang, Weiye Chen, Hao Tang, Yu Zhu, Jinqiu Sun, Luc Van Gool, and Yanning Zhang. Smae: Few-shot learning for hdr deghosting with saturation-awars aware masked autoencoders. In CVPR, 2023b.
Wei Zhang and Wai-Kuen Cham. Gradient-directed multiexposure composition. IEEE TIP, 201 1.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
The content of the appendix material involves:
Analysis and visualization of masks in Sec. A.
Effect of optical flow pre-alignment in Sec. B.
Effect of different alignment methods in Sec. C.
Ablation study on adjusting Use and color in Sec. D.
Additional qualitative comparisons in Sec. E.
Limitation in Sec. F.
Future work in Sec. G.
A ANALYSIS AND VISUALIZATION OF MASKS
In order to avoid harmful information in supervision during training stracture-focused network, we carefully design the mask Map and M.c. for calculating structure-preserving loss Lsp and structure- expansion loss Lee, respectively. The quantitative results of related ablation experiments shown Table 5. Here we give more analysis about the elaborate masks and visualize an example in Fig. A. Therein, the corresponding color component is shown in Fig. A (g).
Mask Map in Structure- Preettrrin Loss. The structure-preserviin to preserve textures of the input reference image, and it is calculated between model output and linear medium-exposiiee distance between the directly, the structural information of dark and over-exposed regions is incomplete in medium-exposure image H₂.
loss aims at guiding the network
image H,. From Table 5, it leads to poor performance when measuring the
Thus, suggest embedding mask Msp into the loss, and it should emphasize the well-exposed areas and mitigate the adverse effects of dark as well as over-exposed areas. A2( in Fig. can do just that, and we adopt it as M.p. The visualization of a (2 example is shown in Fig. (i). can be seen that the overexposed area surrounded by the blue box is successfully suppressed.
Mask M,e in Structure-Expansionn Loss. The structure-ewt expannsionnn loss aims at guiding the network to Iearn textures from non-reference inputs, and it is calculated between model output and color component Yeotor- As Yeolor obtained by fusing aligned multi-exposure images (see Fig. A (b), (e), and (f)), it is inevitable that ghosting artifacts exist in Yeoter (see the area surrounded by the red box in Fig. (g)) when the alignment fails.
Thus, a mask M.e should be designed to circumvent the adverse effects of these ghosting areas for better guiding the network. It is not appropriate to calculate the mask based on the simple difference between Ycolor and reference image H.. Because even if the dark and over-exposed areas are aligned well, the difference between Ycotor and H₂ is still large (see the area surrounded by the blue box in Fig. A (i)). As result, utilize (2 again to mitigate the adverse effects of these areas. Specifically, we multiply A2 (T2) (2)())))))) to Yeator and H2 for calculating the difference, as shown in Eqn. (9). A mask example is shown in Fig. A (k). It can be seen that the well-aligned over-exposed areas surrounded by the blue box are successfully retained, and only the misaligned area is masked.
With the designed masks, the generated structure component Y in Fig. A (I) combines the strengths of the supervisions Yooter and H2, while discarding their weaknesses.
B EFFECT OF OPTICAL FLOW PRE-ALIGNMENT
When constructing color and structure supervisions, the inputs need to be pre-aligned by the optical flow approach (Liu et al., 2009). We remove the pre-alignment processing separately to investigate its impact the final HDR results, which are shown in Table A. From the table, the pre-alignment during obtaining Yeator is crucial, as Yeotor affects the quality of Y Ystrus while Ycotor and Yutr decide the final HDR result. On this basis, are-alignment during generating Ystru further improve performance, achieving 0.27 dB PSNR gain the HDR result Y.
13

---

Published as a conference paper at ICLR 2024
(+) Exposure Image 4
Expesure Image 4
(c) High Exposure Image f
(1) Ground-
(c) Aligned Low Man-EExpuuue Image
( Aligned High Exposureee Image 7,
(c) Color Component Yesoor
(h) Lincar Medium Eaillum---ppoureeee Image Hj
(0)Ag(12) Fig
1 Mae (M/o Ag(F2)
(4) M,, (w/
() Structure Component Yytrs
Figure A: Visualization of masks and related images. (a)~(() show the multi exposure images, while (d) is the corresponding ground-trufe -tutt from Kalantari al. (Kalantari et al., 2017) dataset. (c) and (f) show the aligned low-exposure and aligned high-exposure images, respectively, which are obtained by optical flow (Liu et al., 2009) alignment with the reference medium-exposure image. (g) the constructed color component by fusing aligned multi-exposure images. (h) is the medium-exposure image the linear domain. (i) the mask as a blending weight in Fig. 1. (i) and (k) denote the masks 1 M. (see Eqn. (9)) constructed without and with A2 respectively. (l) is the generated structure component. The red box indicates the area where optical flow alignment fails, and the blue box indicates the area with high brightness.
Table A: Effect of pre-alignment processing when constructing supervision information (Yentor and Sttr)). The final HDR results (Y) are obtained by learning the model with corresponding (Yootor and Ysrr)) supervisions.
Yestor
Ystro
Y Yettor
Y
PSNR-u PSNR-I
PSNR-u PSNR-I
PSNR-u PSNR-I
X
x
25.69 / 31.31
34.58 34.35
35.50 34.95
Pre-Alignment Processing
X
V
25.69 / 31.31
39.04 / 40.38
41.66 40.90
V
34.45 39.01
43.07 40.45
43.41 / 40.76
V
34.45 / 39.01
43.38 41.74
43.68 41.09
In addition, further evaluate the effect of pre-alignment generating Y Specifically, test the structurefocusee network on 74 training scenes with and without optical flow pre-alignment, respectively. The results of sttru show the pre-alignment manner has 0.44dB PSNR-u and 1.46dB PSNR-l / gains on average. Morcover, we compare the results between the two manners one by one. We find that only in 6 scenes, the results without pre-alignment are more than d better than those with pre-alignment on PSNR-u. In the other 68 scenes, the pre-alignment manner always gives better or comparable results.
C EFFECT OF DIFFERENT ALIGNMENT METHODS
The quality color components mainly relies on the alignment method. In this work, for the sake of faimess, follow Kalantari et al. (Kalantari et al., 2017) and FSHDR (Prabhakar al., 2021), adopting Liu et al. (Liu et al., 2009)'s approach for optical flow alignment. Besides, we additionally conduct experiments with other commonly used optical flow networks, ie.. PWC-Net (Sun et al.,
14

---

Published as a conference paper at ICLR 2024
Table B: Effect optical flow alignment methods.
Alignment Method
PSNR-u / SSIM-u
PSNR-I/ SSIM-I
HDR-VDP-2
PWC-Net (Sun al., 2018)
43,45 / 00.998
40.67 0.9864
64.07
FlowFormer (Huang et al., 2022b)
43.50 0.9900
40.60 0.9862
64.43
Liu al. (Liu et al., 2009)
43.68 / 0.9901
41.09 0.9873
64.57
Table C: Effect of ase in Eqn. (9).
Table D; Effect of Centor in Eqn. (14).
Yatru
Y
Y
Jun
PSNR-u / PSNR-I
PSNR-u PSNR-
Tenlor
PSNR-u PSNR-I
2.5/255
43.27 / 41.49
43.64 / 40.88
5/255
43.60 / 41.08
5/255
43.38 41.74
43.68 41.09
10/255
43.68 41.09
7.5/255
43.18 / 41.67
43.57 41.04
15/255
43.61 41.10
2018) and FlowFormer (Huang al., 2022b). As shown in Tab. B, although Liu et al. ' approach relatively early, it is more robust for multi-exposure image alignment than recent learning-base baasd PWC-Net and FlowFormer.
D ABLATION STUDY ON ADJUSTING Ose AND O Celor
The hyperparameters ae (see Eqn. (9)) and Ocotor (see Eqn. (14)) are set to 5/255 and 10/255 by default for experiments, respectively. Here, we vary dse or Tcolor conduct experiments. Table C and D show the experimental results, respectively. The results show that the sensitivity ase and Ccolor of our SelfHDR acceptable.
E ADDITIONAL QUALITATIVE COMPARISONS
Additional visual comparisons on Kalantari et al. (Kalantari et al., 2017) dataset are shown in Fig. B and Fig. C, respectively. Our SclfHDR has fewer ghosting artifacts than zero-shot FSHDR (i.e.. oo) (Prabhakar et al., 2021). Sometimes, SelfHDR even outperforms the corresponding supervised methods. Red arrows in the results indicate areas with ghosting artifacts in other methods.
F LIMITATION
First, the main limitation is the requirement for clear input images, i.e., they should be noise-free and blur-free. When noise exists short-exposure images or blur exists in long-exposure images, SelfHDR can not remove noise and blur, as shown in Fig. D. Second, when the scene irradiance changes drastically in shooting multi-exposure images, SelfHDR may fail, as the constructed color components may be inaccurate.
Actually, most existing multi- extosure HDR reconstruction methods (including supervised and self- supervised ones) only focus removing ghosting artifacts caused by misalignment between inputs, having these limitations as well. Our SeifHDR has taken a step toward more realistic self-supervised HDR imaging by deghosting, while ongoing work is to further address these limitations.
G FUTURE WORK
In future work, we will explore self-supervised HDR reconstruction when considering more realistic shooting conditions. In low-light environments, there may exist noise in short-exposure images and blur long-exposure images. In order to achieve a self-supervised algorithm, we can combine HDR self-supervised denoising and debluring works process input images
reconstruction with for removing undesirable degradations. Moreover, an adaptive method may need to be explored to select more appropriate image base frame. For example, when medium-exposure image
15

---

Published as a conference paper at ICLR 2024
LDRs
Our SelfHDRscTNet
LDR patches
FSHDR
FSHDR
S
AHDRNet
SelfTDDR ANDRNet
HDR-Tra.
SelfHDR
SCTNet
SCTNet
GT
Visual comparison on Kalantari el al. dataset (Kalantari et al., 2017). Red arrows indicate areas with ghosting artifacts from other methods. "HDR-Tra." denotes HDR- TranTrrrnforrr..
Figure B:
suffers more severe degradations than others, the method should adaptively take short-exposure or long-exposure images new base frame for HDR reconstruction.
In addition, as a self-supervised method, has the potential to produce better results and bring better generalization by exploiting more multi-exposure images without the target HDR images. We will explore scaling training data in the future.
16

---

Published as a conference paper at ICLR 2024
LDRs
Our RelfHDRScTNNet
LDR patches
FSHDR
FSHDR
SelfHDR PSHDR
AHDRNet
SelfHDD SITHIDAHBE AHDRNet
Tr..
SelfHDR HDR-Tra.
SCTNet
SelfHDRsSTNNe
GT
Figure C: Visual comparison on Kalantari et al. dataset (Kalantari et al., 2017). Red arrows indicate areas with ghosting artifacts from other methods. 'HDR-Tra." denotes HDR-Transformer.
17

---

Published as a conference paper at ICLR 2024
LDRs
LDR patches
SelfHDR ANDRNet
(a) Noise in the result.
LDRs
LDR patches
SelfHDR A AHDRNee
(b) Blur in the result.
Figure D: Failure cases. Noise or blur may exist in the results.
18


Reviews:
[{"review_id": "G3oQdRWhjb", "reviewer": "Reviewer_Vk35", "summary": "This paper proposes a self-supervised method to fuse multi-exposed images in dynamic scenes. It learns a reconstruction network under the supervision of two complementary components, including the color component and the structure component. The color component is estimated from aligned multi-exposure images. The structure one is generated through a structure-focused network that is supervised by the color component and an input image. These components construct a pseudo reference for training.", "questions": "1. Does this decomposition way (decomposing an image into color and structure components) have a corresponding theoretical basis, and does it correspond to some existing image decomposition theory?\n2. “Regardless of the ghosting areas, the rest can record the rough color value, and in which well-aligned ones can offer both good color and structure cues of HDR images”. The experiment lacks the HDR results in the presence of alignment errors.\n3. The mask-based weighting does not consider pixel neighborhood relationships. Will it lead to artifacts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is a self-supervised method, which overcomes the need for supervised labeled data in supervised methods.\n2. Experiments show that our SelfHDR outperforms the state-of-the-art self-supervised methods, and achieves comparable performance to supervised ones.\n3. The way of decomposing an image into color and structure components is new.", "weaknesses": "1. The drawback of existing self-supervised methods is that they construct pseudo-pairs for HDR reconstruction. The performance of these methods is unsatisfactory, as their simulated pairs still have gaps with real-world ones. The proposed self-supervised way is also realized by constructing a pseudo reference. Why does this method show advantages? The theoretical discussion and differentiation with existing self-supervised methods are not very sufficient.\n2. For dynamic scenes, the method relies on existing registration methods to obtain more accurate color components for registration.\n3. This method constructs a pseudo ground truth for self-supervision. The keys lie in the combination optimization (for color components), and mask-based weighting (for structure components). The construction process is simple and mainly based on the functions in Figure 1 which is set by prior.\n4. One of the contributions is the construction of a color component. However, it lacks the comparison of the overall results which can directly reflect this advantage in terms of color.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGIZaFbYbK", "reviewer": "Reviewer_nVjt", "summary": "This work proposes a self-supervised HDR reconstruction method SelfHDR for dynamic scenes, which only requires input multi-exposure images during training. Specifically, SelfHDR decomposes latent ground-truth into constructible color and structure component supervisions. Experiments show that SelfHDR achieves comparable performance to supervised ones.", "questions": "Please see the above weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.This work proposes a self-supervised HDR reconstruction method, addressing the problem of the difficult collection of paired HDR data.\n\n2.The idea of constructing color and structure supervision respectively is somewhat novel.\n\n3.This work achieves competitive results compared to supervised methods.", "weaknesses": "1.Is the performance upper limit of this work limited by the upper limit of structure or color supervision? What designs correspond to solving this problem?\n\n2.This method may also be limited by the alignment method.\n\n3.Some visualizations of structure and color supervision can be given. \n\n4. Discussion with some traditional multiple exposure fusion methods, such as HDR plus?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v0Qf4I62kC", "reviewer": "Reviewer_TwnM", "summary": "This paper presents \"SelfHDR,\" a self-supervised High Dynamic Range (HDR) image reconstruction method. Traditional HDR reconstruction methods require ample ground-truth data, which can be difficult and expensive to collect. The novelty in SelfHDR comes from learning a reconstruction network under two complementary components, HDR color and structure, constructed from multi-exposure static images. They designed a detailed process to generate these components and used them to train the reconstruction network. Experimental results showed that SelfHDR surpassed state-of-the-art self-supervised methods and achieved comparable performance with supervised methods. Contributions include a novel solution to alleviate the need for extensive labeled data for HDR image reconstruction.", "questions": "- How sensitive is the SelfHDR method to the quality of the input images (for example, noise)? Would substantial noise or minor shifts in alignment between images drastically affect the performance of the method? This might be a bit hard to comment on, as the misalignments in the Kalantari dataset are already fixed.\n- Have the authors considered combining the SelfHDR method with other techniques (perhaps pre-processing or post-processing techniques) to enhance its performance? For example, use a pre-trained denoising network to process the low-exposure image and use a hallucination network such as Stable Diffusion to generate the content in the over-exposed regions.\n- Can the authors provide more insights into the cases where the SelfHDR method might fail or provide subpar results? For instance, are there certain types of scenes, color distributions, or specific types of exposure variations that may destabilize the method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The originality of the paper is substantial as it proposes an innovative self-supervised method for HDR image reconstruction, eliminating the need for extensive labeled data, a factor that typically hinders HDR reconstruction methods. The quality of the research is commendable. To build this method, they analyzed HDR images thoroughly and isolated the primary components, color, and structure that could be derived from multi-exposure static images. The paper is coherent and clear in its exposition, and the authors elegantly illustrate the problem, their approach, and results. The significance of this work is considerable as it presents a promising alternative for HDR image reconstruction, which demonstrates competitiveness with supervised methods while minimizing data requirements.", "weaknesses": "A potential weakness of the paper is the lack of a broader experimental validation. While they have tested the model on several datasets, the application in real-world scenarios, especially dealing with complex situations and variable lighting conditions, is not clearly examined. The paper could have also delved more into the limitations of the proposed method, such as the cases where the SelfHDR method may not provide optimal results. An exploration of the generalizability limitations of the method would have been appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m642qI3iQQ", "reviewer": "Reviewer_DQCr", "summary": "The paper proposed a self-supervised HDR reconstruction method from a triplet of LDR images with relative motion. The proposed method first prepares two intermediate components focusing on HDR structure and color, and then learns a neural network to estimate the final HDR under the supervision of both. The approach is superior to other self-supervised learning approaches and is comparable to supervised learning methods with regard to objective metrics and visual qualities.", "questions": "The structure-focused network was proposed to avoid errors caused by optical flow alignment, but it is integrated with optical flow alignment in training stage 2, which doesnt make sense to me. Although Tab. 6 shows improvement in general, I think it will hurt the performance in the cases when the alignment has more errors, whereas those cases can demonstrate the core value of the paper. The authors should elaborate on this and experiment on those cases to justify their statement when necessary.\n\nMisc:\n* HDR-VDP-2 metric is not as commonly used as PSNR/SSIM. The authors should briefly introduce what the value means, and if a bigger/smaller value indicates better image quality.\n* In Tab 3-6 the highest scores can be marked in bold.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of using two-stage training to learn structure and color information first and then learn the final output is novel to me. The overall design of the method is reasonable.\n\n* experiments are extensive and demonstrate the compelling performance of the proposed method.", "weaknesses": "* The proposed method is specific to 3 inputs. The 3 inputs should be informatively captured, especially the mid-exposed image, which should have texture and color of high fidelity as the reference. The requirement for the input is strict and can limit the application of the proposed method.\n\n*The authors should point readers to the appendix when appropriate, e.g., visualization of masks and selection of sigmas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)