ECMamba: Consolidating Selective State Space Model with Retinex Guidance for Efficient Multiple Exposure Correction

Authors: wei  dong
han  zhou
yulun  zhang
xiaohong  liu
jun  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:39:23

Abstract: Exposure Correction (EC) aims to recover proper exposure conditions for images captured under over-exposure or under-exposure scenarios. While existing deep learning models have shown promising results, few have fully embedded Retinex theory into their architecture, highlighting a gap in current methodologies. Additionally, the balance between high performance and efficiency remains an under-explored problem for exposure correction task. Inspired by Mamba which demonstrates powerful and highly efficient sequence modeling, we introduce a novel framework based on \textbf{Mamba} for \textbf{E}xposure \textbf{C}orrection (\textbf{ECMamba}) with dual pathways, each dedicated to the restoration of reflectance and illumination map, respectively. Specifically, we firstly derive the Retinex theory and we train a Retinex estimator capable of mapping inputs into two intermediary spaces, each approximating the target reflectance and illumination map, respectively. This setup facilitates the refined restoration process of the subsequent \textbf{E}xposure \textbf{C}orrection \textbf{M}amba \textbf{M}odule (\textbf{ECMM}). Moreover, we develop a novel \textbf{2D S}elective \textbf{S}tate-space layer guided by \textbf{Retinex} information (\textbf{Retinex-SS2D}) as the core operator of \textbf{ECMM}. This architecture incorporates an innovative 2D scanning strategy based on deformable feature aggregation, thereby enhancing both efficiency and effectiveness. Extensive experiment results and comprehensive ablation studies demonstrate the outstanding performance and the importance of each component of our proposed ECMamba. Code is available at \url{https://github.com/LowlevelAI/ECMamba}.

Main paper: 
 introduction: 
 Images captured under over-exposure and under-exposure conditions suffer from various degradations, including reduced contrast, color distortion, and information loss in extremely dark or bright regions. The objective of exposure correction is to enhance the visibility, contrast, and structural details for images with various illumination conditions, which is pivotal for improving the performance of a plethora of downstream applications such as object detection, tracking, and segmentation systems [10, 39, 31] in scenarios with improper exposure. Similar to other image restoration tasks [3, 2, 32, 25, 26, 23, 42, 12] , many deep learning models [38, 6, 34, 44] have been proposed for under-exposed image enhancement and demonstrate commendable results. However, our preliminary experiments indicate that these methods generally perform poorly in multi-exposure correction. This inadequacy stems from the distinct mapping flow between Over-Exposed (OE) and Normal-Exposed (NE) images compared to that between Under-Exposed (UE) and NE images. Recently, some promising works [20, 21, 1, 19] have introduced several interesting deep learning networks to learn consistent exposure representations for multi-exposure correction. However, despite its widespread adoption and outstanding performance in under-exposure correction, From column 2-4, we observe a noticeable improvement on color preservation and structure recovery, which demonstrates the importance of our introduced two-branch Retinex-based pipeline and the effectiveness of our proposed ECMamba network. models for multi-exposure correction. Moreover, current state-of-the-art (SOTA) performance is achieved through introducing speci®c designs (exposure normalization [16] or exposure regularization term [15] ) to existing networks. However, these methods present limited generalization, it is essential to develop stronger foundational model with good generalizable ability. Additionally, many methods face a trade-off between performance and ef®ciency, particularly those based on transformers.To address these issues, we introduce a novel two-branch Exposure Correction network (ECMamba) based on standard Mamba architecture, which has demonstrated impressive sequence modeling ability with high ef®ciency [9] . Speci®cally, we derive the Retinex theory and develop a Retinex estimator to transform the input into two intermediary spaces, each approximating the target re¯ectance and illumination map, respectively. As shown in Fig. Aa , compared to the input distribution, the generated intermediary space (R ′ ) shows closer approximation of target distribution, thus enabling subsequent network to execute the ®ne-grained restoration. Moreover, visual compelling results in Fig. Ab column 4 demonstrate that our proposed two-branch framework offers more precise estimation and improved performance than simply optimizing the re¯ectance. Furthermore, we develop a novel 2DSelective State-space layer guided by Retinex information (Retinex-SS2D) as the core operator of our ECMamba. Compared to other scan strategies (i.e., cross-scan mechanism [18] ) which consider the scanning of 2D data to 1D sequence a ª direction-sensitiveº problem, we regard this issue as a ª feature-sensitiveº problem. Therefore, we ®rst perform feature fusion and then introduce a Deformable Feature Aggregation (DFA) guided by Retinex information. Then based on the activation response map derived from DFA, we develop a Feature-Aware 2D Selective Scanning (FA-SS2D) mechanism to ¯atten the aggregated feature into 1D sequence, which is subsequently fed into the standard Selective State Space process (S6) to capture long-range dependencies.The contributions of this work are summarized as follows:⋄ We present a novel dual-branch framework that fully embeds Retinex theory for exposure correction and we provide detailed explanation for its signi®cance.⋄ By analyzing the operating mechanism of Selective State Space Model, we regard the scanning of vision data is a ª feature-sensitiveº issue and we propose an ef®cient Retinex-SS2D layer with Retinexguided deformable feature aggregation and Feature-Aware 2D Selective Scanning Mechanism.⋄ Extensive experiments and ablations demonstrate the impressive performance and high ef®ciency of our propose method. From column 2-4, we observe a noticeable improvement on color preservation and structure recovery, which demonstrates the importance of our introduced two-branch Retinex-based pipeline and the effectiveness of our proposed ECMamba network.Retinex theory [21] has not yet been deeply integrated into deep learning models for multi-exposure correction. As deep learning models often struggle to distinguish between illumination information and the intrinsic reflectance properties of objects in images, simply adopting deep learning models to address such a difficult problem usually obtains sub-optimal results and the incorporation of Retinex theory offers a physically justified way to decompose the illumination and reflectance within deep learning models. Moreover, current state-of-the-art (SOTA) performance is achieved through introducing specific designs (exposure normalization [20] or exposure regularization term [19] ) to existing networks. However, these methods present limited generalization, it is essential to develop stronger foundational model with good generalizable ability. Additionally, many methods face a trade-off between performance and efficiency, particularly those based on transformers.To address these issues, we introduce a novel two-branch Exposure Correction network (ECMamba) based on standard Mamba architecture, which has demonstrated impressive sequence modeling ability with high efficiency [13] . Specifically, we derive the Retinex theory and develop a Retinex estimator to transform the input into two intermediary spaces, each approximating the target reflectance and illumination map, respectively. As shown in Fig. 1a , compared to the input distribution, the generated intermediary space (R ) shows closer approximation of target distribution, thus enabling subsequent network to execute the fine-grained restoration. Moreover, the visually compelling results in Fig. 1b column 4 demonstrate that our proposed two-branch framework offers more precise estimation and improved performance than simply optimizing the reflectance. Furthermore, we develop a novel 2D Selective State-space layer guided by Retinex information (Retinex-SS2D) as the core operator of our ECMamba. Different from other scan strategies (i.e., cross-scan mechanism [23] ) which considers the scanning of 2D data to 1D sequence a "direction-sensitive" problem, we regard this issue as a "feature-sensitive" problem. Therefore, we first perform feature fusion and then introduce a Deformable Feature Aggregation (DFA) guided by Retinex information. Then based on the activation response map derived from DFA, we develop a Feature-Aware 2D Selective Scanning (FA-SS2D) mechanism to flatten the aggregated feature into 1D sequence, which is subsequently fed into the standard Selective State Space process (S6) to capture long-range dependencies.The contributions of this work are summarized as follows:We present a novel dual-branch framework that fully embeds Retinex theory for exposure correction and we provide detailed explanation for its significance. From column 2-4, we observe a noticeable improvement on color preservation and structure recovery, which demonstrates the importance of our introduced two-branch Retinex-based pipeline and the effectiveness of our proposed ECMamba network.Retinex theory [22] has not yet been deeply integrated into deep learning models for multi-exposure correction. As deep learning models often struggle to distinguish between illumination information and the intrinsic reflectance properties of objects in images, simply adopting deep learning models to address such a difficult problem usually obtains sub-optimal results and the incorporation of Retinex theory offers a physically justified way to decompose the illumination and reflectance within deep learning models. Moreover, current state-of-the-art (SOTA) performance is achieved through introducing specific designs (exposure normalization [19] or exposure regularization term [21] ) to existing networks. However, these methods present limited generalization, it is essential to develop stronger foundational model with good generalizable ability. Additionally, many methods face a trade-off between performance and efficiency, particularly those based on transformers.To address these issues, we introduce a novel two-branch Exposure Correction network (ECMamba) based on standard Mamba architecture, which has demonstrated impressive sequence modeling ability with high efficiency [14] . Specifically, we derive the Retinex theory and develop a Retinex estimator to transform the input into two intermediary spaces, each approximating the target reflectance and illumination map, respectively. As shown in Fig. 1a , compared to the input distribution, the generated intermediary space (R ′ ) shows closer approximation of target distribution, thus enabling subsequent network to execute the fine-grained restoration. Moreover, the visually compelling results in Fig. 1b column 4 demonstrate that our proposed two-branch framework offers more precise estimation and improved performance than simply optimizing the reflectance. Furthermore, we develop a novel 2D Selective State-space layer guided by Retinex information (Retinex-SS2D) as the core operator of our ECMamba. Different from other scan strategies (i.e., cross-scan mechanism [27] ) which considers the scanning of 2D data to 1D sequence a "direction-sensitive" problem, we regard this issue as a "feature-sensitive" problem. Therefore, we first perform feature fusion and then introduce a Deformable Feature Aggregation (DFA) guided by Retinex information. Then based on the activation response map derived from DFA, we develop a Feature-Aware 2D Selective Scanning (FA-SS2D) mechanism to flatten the aggregated feature into 1D sequence, which is subsequently fed into the standard Selective State Space process (S6) to capture long-range dependencies.The contributions of this work are summarized as follows:⋄ We present a novel dual-branch framework that fully embeds Retinex theory for exposure correction and we provide detailed explanation for its significance.⋄ By analyzing the operating mechanism of Selective State Space Model, we regard the scanning of vision data is a "feature-sensitive" issue and we propose an efficient Retinex-SS2D layer with Retinex-guided Feature-Aware 2D Selective Scanning Mechanism.⋄ Extensive experiments and ablations demonstrate the impressive performance of our proposed method. 
related works: 
 Learning based Multi-Exposure Correction Multi-exposure correction is a challenging task due to the opposite optimization flows of under-exposure and over-exposure correction. MSEC [1] introduces a Laplacian pyramid architecture to restore lightness and structures. Later, several normalization and regularization methods [4, 21, 19, 20] are proposed for exposure correction. For example, the exposure normalization [19] is proposed for exposure compensation, ECLNet [21] introduces exposure-consistency representations with bilateral activation mechanism, and FECNet [20] opts to correct illumination in the frequency domain. Different from previous methods, we aim to develop a Retinex-based network, where Retinex guidance is utilized to modulating the optimization flows of under-exposure and over-exposure correction.State Space Model (SSMs) Due to its impressive modeling capability for long-range dependencies and its promising efficiency, State Space Models (SSMs) and recent proposed Structured State-Space Sequence model (S4) [15] has attracted great interests among researchers. Based on S4, several models and strategies are introduced to improve the efficiency and boost the capability, among which Mamba [14] introduces an input-dependent SSM with selective mechanism and achieves superior performance than Transformers for natural language processing. Moreover, some pioneering works have applied Mamba on vision task such as image segmentation [27] , classification [47] and even restoration [17, 29] . We are the first to address exposure correction problem based on Mamba, and we innovatively introduce an efficient feature-aware scanning strategy in this work. 
preliminaries: 
 State Space Model (S4) State Space Model (S4) is introduced by combining recurrent neural networks (RNNs), convolutional neural networks (CNNs), and classical state space models. Specifically, for a sequence of L length x ∈ R L , the input at any time step x(t) ∈ R can be mapped to an output y(t) ∈ R through the following state space modeling:EQUATIONwhere h(t) ∈ R N ×1 represents latent state and N denotes the dimension scaling ratio in latent state. A ∈ R N ×N , B ∈ R N ×1 , and C ∈ R 1×N are state transition matrix, control matrix, and output matrix, respectively. Mathematically, the differential equation in Eq. 1 has an equivalent integral equation and it can be solved using numerical computation. In order to integrating state space modeling into deep learning models, the discretization is required to convert continues-time model to discrete-time system by introducing the timescale ∆ ∈ R. Specifically, the zero-order hold (ZOH) rule, which is commonly used in SSM-based deep learning algorithms, is applied to transform continuous parameters A, B in Eq. 1 to discrete matrix Ā, B as follows:EQUATIONwhere out irrelevant noise and focusing on important tokens, thereby achieving outstanding performances in multiple language and vision tasks.Ā ∈ R N ×N , B ∈ R N ×1 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper proposes a new algorithm for image exposure correction, there is no negative impact of our method, and the positive impact is discussed in Sec. 6. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper The paper poses no such risks.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
methodology: 
 The overall framework of our method is shown as Fig. 2 , which demonstrate that our proposed exposure correction network is designed based on Mamba and Retinex theory. In this section, we first introduce our formulated Retinex-based Exposure Correction Framework (Sec. 4.1), then we propose to utilize Exposure Correction Mamba Module (ECMM) to achieve precise restoration for the reflectance and illumination map (Sec. 4.2). More importantly, to enhance the efficiency and effectiveness, we introduce a new 2D Selective State-space layer with an innovative scanning mechanism guided by Retinex information (Retinex-SS2D) in Sec. 4.3 and Sec. 4.4.The Retinex theory can be expressed as I GT = R GT ⊙ L GT , where ⊙ denotes Hadamard product, I GT is an ideal image without degradation, R GT and L GT represent the reflectance image and illumination map, respectively. However, a low-quality image I LQ captured under non-ideal illumination conditions (under-exposure or over-exposure scenes) inevitably suffer from severe noise, color distortion, and constrained contrast. Therefore, we introduce a perturbation to R GT and L GT respectively ( R and L) to model these degraded images as:EQUATIONSome existing Retinex-based methods [6, 13, 18, 33] regard the reflectance component R GT as the final enhanced result, thus they ignore the last three terms in Eq. 3 and focus on modeling the mapping: R GT = F(I LQ ) ⊙ (I LQ ) using network F. However, these models can only achieve sub-optimal performance due to the difficulty of acquiring accurate mapping, especially for multiple exposure correction task, where multiple Under-Exposed (UE) and Over-Exposed (OE) inputs correspond to one Normal-Exposed (NE) image. Therefore, we choose to restore the reflectance and illumination component simultaneously in order to obtain satisfactory outputs. Specifically, we element-wisely multiply the both sides of Eq. 3 by L and R respectively as: Compared to other 2D scanning methods, our approach generates a sequence ordered by feature importance, thereby maximizing the robust sequence modeling capabilities of Mamba.EQUATIONwhere L and R are the matrix such that L ⊙ L GT = 1 and R ⊙ R GT = 1, and we assume we can approximate L and R via Retinex estimatorE. R GT ⊙ L ⊙ L + R + R ⊙ L ⊙ L and R ⊙ L GT ⊙ R + L + R ⊙ L ⊙ R indicate the remaining degradation in R ′ and L ′ .Therefore, the well-exposed result can be retrieved using deep-learning networks byEQUATIONwhere M R and M L are networks utilized to predict the minus degradation in R ′ and L ′ , and F c serves as a Retinex guidance information derived from the I LQ .As shown in Fig. 2 , the Retinex estimator E takes I LQ and its mean matrix along the channel dimension (which is omitted for clarity in Fig. 2 ) as inputs. E firstly utilizes a 1 × 1 convolution and a depth-wise convolution with 5 × 5 kernel to extract features. Then, L, R and F c are generated by one 1 × 1 convolution, respectively. More importantly, R ′ and L ′ are fed into M R and M L for further restoration. In addition to optimizing I out to approximate I GT , our training objective incorporates a constraint on L and R, as discussed in Sec 4.5.Discussion (i) Many Retinex-based methods [37] aims to learn the mapping between the input and the reflectance image and illumination map, then obtain the final result using Hadamard product operation. However, this strategy is not suitable for multi-exposure correction task. Fig. 1a illustrates the complicated and distant distribution patterns of OE and UE images relative to their normally exposed equivalents. Such complex distributions challenge the establishment of accurate mappings from inputs. However, by carefully analyzing Retinex theory, we construct an intermediary space that significantly reduces the distance to our optimization objectives and facilitates the subsequent fine-tuning restoration process, as shown in Fig. 1b . (ii) Some methods [6, 13, 33, 18] treat R GT as the final enhanced result, which deviates form the original explanation of Retinex theory and leads to limited performance. Therefore, we adopt a two-branch framework to reconstruct the reflectance and illumination map using distinct deep learning networks. The significance of our framework is discussed in the ablation study (Sec. 5.3).Together with our proposed Retinex-guided exposure correction framework, we also develop innovative networks that serves as M R and M L in Eq. 5 to estimate the remaining corruption in R ′ and L ′ . In order to develop an effective and efficient module that is capable to achieve high performance and is friendly to resource-limited devices, we propose an novel Retinex-guided Exposure Correction Mamba Module (ECMM) which succeeds the powerful modeling capability of Mamba. Notably, M R and M L shares similar structure and we discuss the details of M R in this section.As illustrated in Fig. 2 , our ECMM adopts a two-scale U-Net architecture. For the encoding process, the input R ′ is firstly processed by a conv 3 × 3 and one RetinexMamba Block (RMB) to generate the initial feature F 0 . Then the downsampling operation is achieved by one 4 × 4 convolution with stride 2, and the down-sampled feature is fed into another RMB to obtain the middle feature F 1 . For the decoding stage, F 1 is firstly up-scaled to F ′ 0 by a 2 × 2 deconv with stride 2. To alleviate the information loss caused by the down-sampling process, we introduce an adaptive mix-up feature fusion [45] to transfer the encoding information to the decoder stage as:EQUATIONwhere θ represents a learnable coefficient, and σ denotes the sigmoid function. Then the fused feature F a is fed into the RMB and the convolution layer sequentially and the restored reflectance R out is obtained by a residual addition accorrding to Eq. 5.As the core operator to extract and aggregate features in ECMM, our RMB block adopts a similar architecture with Transformer block. However, the significant computational demands of selfattention and cross-attention mechanisms obviously compromise the efficiency of Transformer-based methods, precluding their application in real-time or resource-constrained environments. To this end, we remove the attention process and introduce an innovative Retinex-guided 2D Selective State-space (Retinex-SS2D) layer to capture long-range dependencies and facilitate dynamic feature aggregation. Therefore, the feature flow of our RMB can be described as:EQUATIONwhere LN denotes the LayerNorm, F in and F in represent the input and output feature of RMB. F c is the Retinex guidance information extract by the Retinex estimator E. Moreover, inspired by ConvNext [28, 8] , we remove the gating mechanism and the depth-wise convolution to develop an Efficient Feed Forward (EFF) layer that follow the conv 1 × 1 → GELU → conv 1 × 1 flow, which operates similarly to MLPs in Transformers, while requiring fewer parameters.The detailed illustration of Retinex-SS2D layer is shown as Fig. 3 . We first conduct feature fusion for input feature F in and Retinex guidance feature F c by linear operation, depth-wise convolution, element-wisely multiplication, and SiLU operation. Subsequently, the fused feature F f is fed into our proposed Feature-Aware 2D Selective State-space (FA-SS2D) mechanism to capture dynamic long-range dependencies and achieve adaptive spatial aggregation. Besides, a gating signal G s and a linear operation is utilized to obtain the aggregated feature F g . Feature-Aware 2D Selective State-space Mechanism The standard Selective State-space Model (S6) achieves outstanding performance on sequence modeling, especially for NLP task that involves temporal sequence. However, significant challenges arise when applying S6 to 2D image. To better modeling the spatial information in 2D images, several interesting works propose multiple scan strategies to unfold image patches into 1D sequences. For example, [27] introduces cross-scan strategy that generate four sequences along four distinct traversal paths, and each sequence is processed by a separate S6 operation. However, such strategy incredibly increase the computational demands, which contradicts the inherently high efficiency and low computational requirements of S6. Furthermore, these techniques only involve simple scanning of images across different directions, which results in a substantial separation of local textures and global structures in some sequences. This separation, to some extent, impairs the S6 framework's modeling ability for images.The deficiencies in the existing scanning approach drive us to reassess how S6 can be more effectively utilized for 2D images. As described in Eq. 2, for each token in a sequence, the output y(t) depends on its input x(t) and previous inputs {x(1), x(2), • • • , x(t -1)}. This mechanism requires the 1D sequences transformed from 2D image to meet the following two criteria to ensure excellent performance:(1) The sequence should prioritize the most critical feature regions at the beginning, while relegating less significant information to the end. (2) Spatially adjacent features should be 1 : Quantitative comparisons of different methods on multi-exposure correction datasets. The best and second-best results are highlighted in bold and underlined, respectively."↑" means the larger, the better. Note that we obtain these results either from the original papers, or by running the officially released pre-trained models. " * " means that original papers don't report corresponding performance and we train their models using their officially released code.sequenced closely to avoid significant gaps in the sequence. However, existing 2D scanning strategies fail to meet these two requirements, motivating us to propose new solutions to address this gap.Based on these observations, we introduce an efficient Feature-Aware 2D Selective State-space (FA-SS2D) mechanism, as shown in Fig. 3 . Firstly, we develop a deformable feature aggregation operation modulated by Retinex information. Specifically, Deformable Convolution (DCN) [48, 9] is adopted to capture dynamic long range dependencies of the fused feature F f . For example, when DCN is applied to the token delineated by the red frame in F f of Fig. 3 , its receptive field is an irregular kernel and the activated tokens are outlined in blue. More importantly, when the red frame is sliding across the feature map, the irregular kernel varies and we record which tokens are activated.After this process, we obtain the total activation number and calculate the average activation frequency for each token. Therefore, we can obtain an activation response map shown in F d of Fig. 3 , where tokens with higher activation frequency represent important features. Specifically, relatively brighter areas in under-exposed images or relatively normally exposed areas in over-exposed images contain important features and exhibit a large activation response. Based on the obtained activation response map, we propose a feature-aware scanning strategy. Different from "direction-sensitive" scanning method [27] , our feature-aware strategy ranks tokens by their activation frequency and place tokens with higher frequencies at the start of the sequence. Therefore, our generated sequence effectively meets Mamba's requirements, thereby maximizing its modeling capabilities for vision data.In this work, we adopt the one-stage strategy to train our proposed ECMamba network, which means that the E, M R , and M L are optimized simultaneously. Our ultimate training objective is to approximate L out to I GT , and we also integrate several constraints on L, R, and R out to achieve stable training. Therefore, our complete optimize strategy is shown as below:EQUATIONwhere L 1 ( L ⊙ L out , 1) and L 1 ( R ⊙ R out , 1) are constraints applied on L and R, and they essentially employ a self-supervised strategy to learn L and R. In addition, considering this optimization is inherently an ill-posed problem, we adopt L 1 (R out , I GT ) to guide the optimization towards the appropriate direction. Moreover, M L L(I out , I GT ) is the primary loss function in our training process and it can be calculated by: where L ssim [46] denotes the structure similarity loss and L per [43] represents the difference between features extracted by VGG19 [30] . The coefficients for corresponding loss functions are set as: ϕ ssim = 0.2, ϕ per = 0.01, λ = 0.1, λ R = 0.1, and λ L = 0.1 in this work.EQUATION 
experiment settings: 
 Datasets To evaluate the performance of our method, we conduct experiments on five prevailing datasets for multi exposure correction and under-exposure correction: ME [1] , SICE [5] , LOLv1 [36] , LOLv2-real [41] , and LOLv2-synthetic [41] datasets. Specifically, each scene in ME dataset has five exposure levels, and we regard the images with the first two exposure level as under-exposed images and the test as over-exposed images. For SICE, following FECNet [20] , we select the middle exposure subset as the ground truth, and define the second and the last second exposure subset as the under-exposed and over-exposed images, respectively. For LOLv1, LOLv2-real, LOLv2-synthetic datasets, we leverage their official training and testing data for model training and evaluation.Implementation Details We use the Adam optimizer with default parameters (β 1 = 0.9, β 2 = 0.99) to implement our model by PyTorch. The initial learning rate is set to 1 × 10 -4 and then it is steadily decreased to 1 × 10 -6 by the cosine annealing scheme, respectively. We utilize random flipping and rotation for data augmentation. Image pairs are cropped as 256 × 256 and the batch size is set to 4. The total training iterations is set to 300K for ME dataset and 150K for other benchmarks, respectively. During the evaluation, we utilize Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM [35] ) for numeric evaluation.Quantitative Results We compare the performance of our ECMamba with current SOTA methods on multi-exposure correction datasets, and we report the quantitative results as Tab. Table B : Quantitative comparisons of different methods on LOLv1 [24] and LOLv2 [29] .is steadily decreased to 1 × 10 -6 by the cosine annealing scheme, respectively. We utilize random flipping and rotation for data augmentation. Image pairs are cropped as 256 × 256 and the batch size is set to 4. The total training iterations is set to 300K for ME dataset and 150K for other benchmarks, respectively. During the evaluation, we utilize Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM [23] ) for numeric evaluation. Table 2 : Quantitative comparisons of different methods for under-exposed correction. Notably, compared to SOTA methods, our ECMamba achieves enhanced performance on LOLv1 [36] , LOLv2real [41] , and LOLv2-synthetic [41] datasets, demonstrating the effective of our proposed dual-branch Retinex-based framework and feature-aware SS2D layer.compared to the second best performance, our improvement has increased to 0.83 dB and 0.051 on SICE dataset. Tab. 2 summarizes the quantitative comparisons between our method with current SOTA methods on on under-exposure correction. Specifically, our ECMamba outperforms the second best performance (LLFlow-SKF) by an average 1.10 dB increase on PNSR with only 4.4% parameters, revealing the impressive effectiveness and high efficiency of our proposed ECMamba. These numbers demonstrate the superior quality of our enhancement and prove the effectiveness of our proposed ECMamba and two-branch Retinex-based pipeline.Qualitative Comparisons We present the enhanced images of different methods in Fig. 4 (ME) and Fig. 5 (SICE). Our appealing and realistic enhancement results demonstrate our model can generate images with pleasant illumination, correct color retrieval, and enhanced texture details. For example, the rich structural details of cloud patterns (row 2) mountain surface (row 4) and in Fig. 4 , the well preserved bridge and its edge contours (row 1) and the vivid presentation of words in the Table 3 : Ablation studies on SICE dataset and the average PSNR and SSIM are reported. Left is used to verify the effectiveness of the two-branch Retinex-based framework, right is to present the importance of our proposed ECMamba module and FA-SS2D strategy. [Key: * : When M L is removed, the hidden dimension of M R is increased to ensure its parameter number is comparable to ECMamba; ViT/Retiformer: utilized to replace our ECMamba module; Cross-Scan Mechanism: adopted to replace our FA-SS2D strategy.] display board (row 2) in Fig. 5 . In contrast, previous methods struggle to preserve color fidelity and illumination harmonization.To verify the effectiveness of our proposed ECMamba, we conduct extensive ablation experiments and report the average performance on SICE dataset.The Contribution of Two-branch Retinex-based Framework We first remove the branch (M L ), which is utilized for accurate restoration of the illumination map. Therefore, the remaining network aims to optimize R out to the ground truth and the performance is reported in Tab. 3, which still presents competitive performance compared to the current SOTA in Tab. 1. However, compare to our complete ECMamba, only optimizing the relectance inevitably leads to sub-optimal performance. Furthermore, we also adopt a more complicated M R , whose parameters is comparable to the original two-branch framework. However, compared to our two-branch ECMamba, this network still demonstrate poor performance. Finally, similar to other Retinex-based methods [37] , we then remove the Retinex estimator E and directly adopt the remaining network for exposure correction. However, as shown in Tab. 3, this adaptation largely decrease the performance of our ECMamba, indicating the importance of our analysis regarding the intermediary space (R ′ and L ′ ) in Sec. 4.1.The Importance of Our Proposed ECMamba Module First, we replace our ECMamba module with Vision Transformer (ViT) [11] and Retiformer [6] architecture in our two-branch framework.As presented in Tab. 3, our complete ECMamba module offers impressive performance better than ViT. More importantly, ECMamba's efficiency is comparable to Retiformer, which is a famous efficient under-exposure correction approach. Furthermore, to study the significance of our proposed Retinex-SS2D layer and FA-SS2D, we replace the Retinex-SS2D layer with a cross-scan mechanism proposed in VMamba [27] . The increased parameters and decreased performance demonstrate the superiority of our proposed Retinex-SS2D layer and FA-SS2D strategy. 
conclusions: 
 We propose a new two-branch Retinex-based Mamba architecture for exposure correction. By carefully deriving Retinex theory, we propose an two-branch framework guided by Retinex information.To better balance the performance and efficiency, we introduce ECMamba as the primary restoration module with efficient Retinex-guided SS2D layer and Feature-aware scanning strategy. Extensive experiments demonstrate that our ECMamba significantly outperforms the current SOTA methods on both multi-exposure correction datasets and under-exposure correction datasets. We recognize that our work, while pioneering in certain aspects, also highlights avenues for future investigation. For example, similar to other methods, ECMamba struggles to deliver satisfactory results in scenarios involving extreme exposure cases (extremely dark or over-exposed environments) due to the extensive information loss inherent in degraded images. Essentially, directing recovery from severely degraded images is challenging, but recent advances in image restoration have utilized generative priors to infer the degraded details, and achieve favorable results. In the future, we plan to integrate Mamba with generative priors to effectively alleviate the performance drop on extreme exposure cases.NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited, and the license and terms of use are explicitly mentioned and are properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new assets introduced in the paper are well documented with the documentation provided alongside the assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: Following extensive interactions between the authors and reviewers, all reviewers have expressed a positive opinion on the paper. The AC believes the paper merits a clear acceptance. Although the authors have attempted to justify their use of Mamba and Retinex for this task, the rationale still lacks a solid foundation. The authors are encouraged to improve the paper by adhering to the commitments and clarifications made during the rebuttal and discussion

Reviews: review: paper_summary: paper_summary: This paper introduces ECMamba, a novel framework that integrates Retinex theory and the Mamba framework to address the complex issue of exposure correction. ECMamba adapts the Retinex theory to suit the needs of exposure correction and develops a Retinex estimator to assess both reflectance and illumination maps. Subsequently, the framework employs Mamba to orchestrate the restoration process, termed ECMM. The ECMM module features a core operator, Retinex-SS2D, which employs a two-dimensional scanning strategy and deformable feature aggregation. The final output is generated by multiplying the enhanced reflectance map with the enhanced illumination map, yielding a superior result.
strength_weakness: strength_weakness: 1. The rationale for employing Retinex theory and the Mamba framework in multi-exposure correction is not adequately articulated. The author mentions that Retinex theory has not been deeply integrated into this field, yet this alone does not establish its superiority over alternative methods. A more robust justification for using Retinex theory, as well as the Mamba framework, is necessary to substantiate their relevance to solving this specific problem.

2. The initial two rows of Figure 4 are presumably intended to illustrate over-exposed scenarios. However, these images do not appear over-exposed as they lack saturated regions and display clear details and contrast. It would be advisable to replace them with images that more accurately represent over-exposure.

3. The inference speed of the proposed method is a critical aspect that is currently unaddressed in the paper.
questions: questions: Shown in the "Weakness" part.
limitations: limitations: The paper does not discuss the limitations. No negative social impact is present in this work.
paper_summary: paper_summary: This paper introduces a new pipeline called ECMamba for multiple exposure correction. Based on the analysis of Retinex theory, the authors develop a dual-branch framework, and each pathway is designed to restore the reflectance image and the illumination map, respectively. Besides, considering the powerful and efficient sequence modeling of the recently proposed Mamba, this paper also incorporates the Mamba architecture within each pathway and attempts to achieve effective and efficient exposure correction. To exploit Mamba to process image data, this paper develops an innovative "feature-aware" 2D scanning strategy based on deformable feature aggregation, which are quite different from other "direction-sensitive" scanning approaches. Finally, extensive experiments on multi-exposure and under-exposure datasets are conducted, and the reported results demonstrate the proposed method in this paper outperforms current SOTA approaches.
strength_weakness: strength_weakness: (1) This paper introduces an interesting two-branch framework based on retinex theory. Compared to a single-branch network, it is more difficult to train a two-branch network especially when the final result is obtained via Hadamard product. However, the constraint used in the training phase is less discussed in this paper.

(2) The proposed "feature-sensitive" scanning mechanism in Mamba is quite novelity. However, the difference in the activation frequency map between under-exposed images and over-exposed images is less explored.

(3) This paper adopts deformable convolution for feature aggregation and the generation of the the activation frequency map. However, how to get the activation frequency map is not very clear. For example in Fig.3, my understanding is that the 9 blue tokens are activated when a 3x3 deformable convolution is applied to the red token. Also, I think the numbers in F_d denote the average activtion frequency for each token when sliding the red token across the entire F_f. Am I correct?

Minor: 
The reference [1] in line 194 is a typo, the reference should be Vmamba, which is cited as [18] in this paper. 
"Deformable Feature Aggegration" in Fig.3 should be "Deformable Feature Aggregation".
questions: questions: 1) What constraint is adopted to train the proposed two-branch network? Is that possible that R_out and I_out are quite different form their corresponding ground truth, but their product I_out is very close to the well-exposed image I_GT?
2) What is the difference in the activation frequency map between under-exposed images and over-exposed images?
3) How to get the activation frequency map in Fig.3? Please explain in detail.
limitations: limitations: Authors have briefly discussed the limitations in Sec.6. There are no potential negative ethical and societal implications of this work.
paper_summary: paper_summary: The authors propose a novel Mamba architecture for the exposure correction task based on Retinex theory. They design a separate Retinex estimator and two exposure correction modules to restore reflectance and illumination. In these modules, the authors improved efficiency in terms of time and resources by introducing the Mamba module. This model demonstrates excellent performance on the representative ME and SICE datasets.
strength_weakness: strength_weakness: * It is necessary to explain the design motivation of the Retina estimator. Unlike general networks, the authors start with a 1 by 1 convolution. It is necessary to explain the motivation behind designing such a network.
* A detailed explanation is needed on how \bar{L} and \bar{R} are constrained. R_gt and I_gt do not seem to be values that can be obtained directly.
* There is a lack of analysis and ablation studies on the network.
  * I am curious about the performance of the synthesized result when removing M_R and M_L, which are related to the terms overlooked in other papers. If I understand correctly, the performance should be similar to other works even without these two modules, especially since they are used in a residual form.
  * The comparison experiment between deformable convolution and standard convolution is missing.
questions: questions: * The authors mention that the modulated reflectance (R′) demonstrates a closer approximation of Normal-Exposed (NE) images, but this is not well illustrated in Figure 1(a). If they could investigate and show the distribution of distances reduced via t-SNE, their claim would be more trustworthy.
* I am willing to increase the score if I have misunderstood something.
limitations: limitations: * This study is far from having a social impact. However, It is regrettable that the limitations and failure cases of the network's performance are not adequately addressed.
paper_summary: paper_summary: The manuscript proposes a novel model that deeply embeds Retinex theory into the Mamba model. The proposed model consists of two modules: Retinex estimator and exposure correction. Comparative experiments on five datasets demonstrate the superiority of the proposed method, and subsequent ablation experiments demonstrated the rationality of the model construction.
strength_weakness: strength_weakness: The experimental validation leaves much to be desired as I mention in the questions section.
questions: questions: 	In page 1, line 35, the authors stated that “Retinex theory has not been deeply integrated into…”, which could cause some negative effects? Please provide some explanations.
	The RMB in Figure 2 is mentioned in page 5, line 162, but the corresponding model diagram is missing. Please provide more explanations.
	In Figure 3, which arrows are marked with ∆p and m?
	In the ablation experiments, the author removed ML twice, what are the differences between those operations?
	In qualitative comparisons, the author do not list all the comparison experiment results, only list some results. For example, as can be seen in Table 1, the results of FECNet+ERL are better than FECNet, why not choose FECNet+ERL for displaying?
limitations: limitations: The authors do not point out the limitations of the work and do not offer further perspectives. I hope it will be improved.

Comments: title: Additional Comments
Would you please have a look at the rebuttal?
Would you please have a look at the rebuttal?
comment: Dear Reviewer EgGV,
 
We are glad that our rebuttal has addressed your concerns. Thanks again for your valuable comments. 

Sincerely,

Authors of Submission 3490
Dear Reviewer 5Zdz,

Thanks again for your constructive comments. We are glad that our response resolved the majority of your concerns. We will add the explanations in the revised manuscript for more clarity.

Sincerely,

Authors of Submission 3490
Dear Reviewer fpJd,

We appreciate your positive review and we are happy that our response well addressed your concerns.

Thank you again for your valuable comments.

Sincerely,

Authors of Submission 3490
Thank you for your rebuttal. You have solved the majority of my concerns. Though I still have reservations about the motivation of utilizing Mamba and Retinex theory, I find most of the other explanations persuasive. Consequently, I am considering adjusting my rating to ``borderline accept``.
Thanks for your responses, which have well  addressed my concerns. Thus, I tend to keep my original score.
The authors have addressed my concerns through additional experiments. I will raise my score from 4 to 5. Thank you for your hard work during the rebuttal period.
Dear Reviewer,

Thanks a lot for contributing to NeurIPS2024.

The authors have provided detailed responses to your review. Would you please have a look at them at your earliest convenience?

Thanks again. AC
Dear Reviewer,

Thanks a lot for contributing to NeurIPS2024.

The authors have provided detailed responses to your review. Would you please have a look at them at your earliest convenience?

Thanks again.
AC