MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space

Authors: jiangwei  weng
zhiqiang  yan
ying  tai
jianjun  qian
jian  yang
jun  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:38:56

Abstract: Recent advances in low light image enhancement have been dominated by Retinex-based learning framework, leveraging convolutional neural networks (CNNs) and Transformers. However, the vanilla Retinex theory primarily addresses global illumination degradation and neglects local issues such as noise and blur in dark conditions. Moreover, CNNs and Transformers struggle to capture global degradation due to their limited receptive fields. While state space models (SSMs) have shown promise in the long-sequence modeling, they face challenges in combining local invariants and global context in visual data. In this paper, we introduce MambaLLIE, an implicit Retinex-aware low light enhancer featuring a global-then-local state space design. We first propose a Local-Enhanced State Space Module (LESSM) that incorporates an augmented local bias within a 2D selective scan mechanism, enhancing the original SSMs by preserving local 2D dependency. Additionally, an Implicit Retinex-aware Selective Kernel module (IRSK) dynamically selects features using spatially-varying operations, adapting to varying inputs through an adaptive kernel selection process. Our Global-then-Local State Space Block (GLSSB) integrates LESSM and IRSK with layer normalization (LN) as its core. This design enables MambaLLIE to achieve comprehensive global long-range modeling and flexible local feature aggregation. Extensive experiments demonstrate that MambaLLIE significantly outperforms state-of-the-art CNN and Transformer-based methods. Our code is available at https://github.com/wengjiangwei/MambaLLIE.

Main paper: 
 introduction: 
 Low light image enhancement is a challenging task in computer vision due to insufficient lighting and sensor degradation. Consequently, images often suffer from poor global visibility and local issues such as color distortion and noise. These degraded images can adversely affect human perception and related vision tasks, such as object detection [5] and depth estimation [57] .Traditional techniques, such as histogram equalization [1] and gamma correction [7] , enhance images through global mapping operations. However, these global operations often struggle to address local degradation effectively. In recent years, many methods based on CNNs and Transformers have gradually come to dominate this field [53, 64, 15, 38, 56, 3] . CNN-based methods [53, 64, 15, 38, 55] have achieved significant advancements by effectively aggregating local information, thus substantially improving performance in low light enhancement. Nevertheless, the limited receptive field and weight-sharing strategy of CNNs result in a local reductive bias, making the models less adaptive to varying inputs. On the other hand, Transformer-based methods [56, 3, 62] achieve a Figure 1 : The Effective Receptive Field (ERF) visualization [34] for SNR-Net [56] , Retinexformer [3] , MambaIR [16] and our MambaLLIE. A broader distribution of bright areas signifies a larger ERF. The receptive field of SNR-Net is large but messy, due to the SNR-aware mechanism, Retinexformer achieves a larger receptive field of the central point, and MambaIR has the global receptive field, but presents the limited local perception. Only our proposed MambaLLIE achieves a global perception ability outwards from central point and preserves the large local receptive field. larger and adaptive receptive field by emphasizing long-term dependencies through the self-attention mechanism. However, the vanilla attention mechanism scales quadratically with input size, resulting in significant computational overhead.Recently, State Space Models (SSMs) [10, 31, 28] have garnered significant attention in the field of computer vision. These internal state space models demonstrate great potential for global information modeling with linear complexity. However, a straightforward implementation of vision state space models for low light image enhancement is inadequate. This is because SSMs are primarily designed for long-range modeling and lack the flexibility needed to capture local information effectively [66] . For instance, as illustrated in Fig. 1 , the receptive field of MambaIR [16] , a simple yet effective vision state space model, achieves longer-range dependencies compared to CNN and Transformer-based methods. Nevertheless, it falls short in refining local interactions.In this work, we introduce MambaLLIE, a novel framework for enhancing low light images that integrates an implicit Retinex-aware approach within a global-then-local state space model. Mam-baLLIE not only explores the capabilities of state space models in low light image enhancement but also incorporates a Retinex-aware structure providing both explicit and implicit guidance. Our framework introduces a unique global-then-local state space block, enhancing global long-range degradation modeling and local feature aggregation through an augmented state space. Additionally, we incorporate a Retinex-aware selective kernel mechanism in the enhancement process, enabling adaptive modulation of illumination strength through specific spatial operations.Our contributions and main findings can be summarized as follows: 1) We introduce a novel globalthen-local state space block that integrates a local-enhanced state space module and an implicit Retinex-aware selective kernel module. This design effectively captures intricate global and local dependencies. 2) We devise an implicit Retinex-aware selective kernel mechanism to guide deeper neural representations, eliminating the need for complex structural design and constraints to estimate physical priors, the prior feature tends to segregate into independent positive and negative illumination components before integrating them, a capability lacking in explicit methods. 3) Experimental results on benchmark datasets and real-world evaluations consistently demonstrate the superior performance of our proposed method compared to state-of-the-art approaches. 
related work: 
 Low Light Image Enhancement. Nowadays, the existing deep learning-based methods have mainly been categorized into end-to-end and Retinex-based methods [27] . To the best our knowledge, LLNet [33] firstly introduced a deep neural network for low light image enhancement by supervised learning. LightenNet [2] adopted the CNN for single image contrast enhancement. MBLLEN [35] proposed the multi-branch fusion within CNNs to extract rich features. Besides, SNR-Net [56] , Restormer [62] , LLFormer [22] and SCENet [36] adopted the self-attention mechanism to achieve excellent performance. However, all these end-to-end models mainly depend on the distribution of training dataset and ignore the inherent illumination prior. As contrast, ZeroDCE [15] , RUAS [30] and subsequent works [38, 9, 51] represent impressive solutions for image enhancement, as ones precisely using physical priors to enhance the images. However, due to the absence of an ideal reference for guidance, these methods usually exhibit a certain gap compared to supervised learning models.As for supervised Retinex-based models, these methods aim to decompose the image into illumination and reflectance maps, and then enhance the image by optimizing these maps. For instance, Retinex-Net [53] divided image enhancement into decomposition, adjustment and reconstruction stages, which providing a good representation of image enhancement process. KinD [64] and URetinex-Net [55] further introduced the novel multi-branch and multi-stage frameworks, respectively. However, striking a balance between complexity and efficiency remains challenging for these methods. Recently, Retinexformer [3] simplified a one-stage Retinex-based low light enhancer with a efficient Transformer. Diff-Retinex [61] designed a transformer-based decomposition network and adopted generative diffusion networks to reconstruct the results. Overall, they typically applied the Retinex theory in a direct way, which may be limited for low light enhancement problem.Vision State Space Model. State Space Model (SSMs) [11, 12, 13] are emerging new sequence models for deep learning, which first swept the natural language processing (NLP) community such as language understanding [42] , content-based reasoning [66] . Recently, SSMs have also garnered considerable attention in computer vision (CV) tasks. To our knowledge, S4ND [39] first explored state space mechanism into CV tasks by swapping Conv2D and self-attention layers with S4ND layers in existing models. VMamba [31] bridged the gap between ordered sequences and non-causal visual images, enabling the extension of vision selective state space model with global receptive fields. Vim [65] proposed the bidirectional state space modeling with positional awareness, achieved the global visual perception. Furthermore, LocalMamba [18] was focused on the local scanning strategy, preservation of local context dependencies. EfficientVMamba [41] designed a light-weight SSMs with an additional convolution branch to learn both global and local representational features. MambaIR [16] employed convolution and channel attention to enhance the capabilities of the Mamba. But existing vision state space model do not pay enough attention on capturing local information, as vanilla SSMs are designed for long sequence and the invariant of local vision data is not taken into account in the existing vision state space models. 
methodology: 
 This work aims to introduce a novel implicit Retinex-aware low light enhancer with global-then-local state space. In this section, we revisit the Retinex theory and the state space model, offering a concise overview. Following that, the details of our proposed MambaLLIE are introduced.Retinex Theory. The ideal Retinex theory [25] for low light enhancement assumes that the captured images can be decomposed into reflectance and illumination maps. Following [38, 45] , explicit Retinex-based methods emphasize estimating either an illumination map while regarding the reflectance map as the enhanced result, or estimating concrete reflectance and illumination maps and then restoring the well-exposed images. Specifically, given a low light image L ∈ R H×W ×3 , where H and W represent height and width respectively, the derived maps can be denoted as:EQUATIONwhere • denotes the element-wise multiplication, R ∈ R H×W ×3 denotes reflectance map, a static property of captured objects; I ∈ R H×W denotes illumination map; N ∈ R H×W ×3 denotes enhanced images; R, Ĩ ∈ R H×W ×3 denotes the estimated reflectance and illumination maps, respectively.Consequently, the former assumption N = L/ Ĩ disregards the noise and artifacts resulting from sensor degradation in the captured images, rendering pixel-wise illumination adjustments inadequate. The latter N = R • Ĩ aims to restore the reflectance and illumination maps to enhance the images. However, this requires the design of multiple branches and constraints to guide the training [64] .State Space Model. The SSMs, such as structured state space sequence models (S4) [12] and Mamba [10] , can be regarded as the continuous linear time-invariant (LTI) systems [54] . Given an one-dimension sequence x (t) ∈ R, it projects into a new one-dimension sequence y (t) ∈ R through the hidden state h (t) ∈ R m , the system can be defined as a linear ordinary differential equation:EQUATIONwhere m denotes the state size, A ∈ R m×m , B ∈ R m×1 , C ∈ R 1×m and D ∈ R denotes state, input projection, output projection parameters, and kernel selective parameters.As the raw state-space models are continuous, the systems adopt the discrete versions before feeding the computer, in which the zero-order hold (ZOH) is used to transform the continuous parameters A and B to discrete parameters Ā and B as followsEQUATIONwhere ∆ denotes the step size. Overall, the discretized version can be rewritten as:EQUATIONHowever, the current system remains static for varying inputs. To address this limitation, Mamba [10] introduces selective state space models, allowing parameters to adapt with the input, thereby enhancing selective information processing across sequences. This parameter selection mechanism can be expressed as:EQUATIONwhere f B (x t ), f C (x t ) and f A (x t ) are linear functions that broadens feature to the hidden state dimensions. As SSMs are tailored for long sequences, it is limited in capturing complicated local information. As for visual data, VMamba [31] , Vim [65] , etc., proposed the specific location-aware scan strategies to maintains the integrity of 2D image structures. However, the specific directed sequences overlook the vision information of pixels neighborhood structure. Inspired by [66] , we explore a global-then-local state space, which receives the global perception before the details, supplementing the lack of local information.We first present the overall pipeline of our MambaLLIE, an U-shaped architecture as shown in Fig. 2 (a), which includes encoding and decoding parts with the convolutional downsampling and upsampling layers. The encoder features are concatenated with the decoder features via skip connections.Next, We propose a global-then-local state space block (GLSSB) as the basic core of MambaLLIE, the max and mean priors concatenated with low light image are projected into GLSSB by convolutional layers. Therein, GLSSB is composed of the local-enhanced state space module (LESSM) and the implicit Retinex-aware selective kernel module (IRSK), interleaved with layer normalization.Illumination Prior. Given a low light image L ∈ R H×W ×3 , we employ a 3 × 3 convolution layer to project the neural features F ∈ R H×W ×C from input feature space, and then project features into each GLSSB, which will be described in Section 3.3. Besides, IRSK integrates original input, maximum prior L max ∈ R H×W and mean prior L mean ∈ R H×W as illumination prior L p ∈ R H×W ×5 ,EQUATIONWe first define F g is the output of GLSSB. Subsequently, the downsampling layer and following GLSSB achieve the feature extraction to acquire the deep feature, which can be denoted asF g ∈ R H 2 i × W 2 i ×2 i C, where i = 0, 1, 2. Moreover, the feature is later concatenated with the upsampling layer with a symmetrical structure. Finally, using a 3 × 3 convolution layer projects into F out ∈ R H×W ×3 and the enhanced image can be expressed as N = F out + L.As illustrated in Fig. 2 (b), GLSSB follows the LayerNorm, LESSM, LayerNorm and IRSK flow, motivated by Transformer [46] and Mamba [10] usage of similar structures in a basic block. Given the input feature, it first undergoes the LayerNorm and LESSM to capture the local-enhanced global information. the above process can be denoted as:EQUATIONAnd then, another LN and our proposed IRSK are used for Retinex-aware guidence. The above process can be formulated as:EQUATIONOverall, at the prior module of GLSSB, we capture global dependencies using a local-enhanced SSM. Because the SSM is better at learning global information, the subsequent module aims to handle more refined and complicated local dependencies.Local-Enhanced State Space Module. Existing state space models [8, 12, 10] excels at capturing the causal processing of input data in long range dependencies. However, the unidirectional scan manner encounters difficulties in vision data to modeling non-causal relationships. To accommodate vision data, [65, 31, 41] process the input data from different 2D scan directions. However, these methods ignore the local invariants of vision data as shown in Fig. 2(c ). In other word, the fixed scanning methods widen the distance between neighborhood data and snarl the causal relationships.The most SSMs [31] can be regarded as the continuous linear time-invarian systems, we further introduce the a e (L p ) augmented local bias, enhancing the original SSMs by preserving local 2D dependency. Following [58, 19] , we propose a novel global-then-local state space:EQUATIONwhere e (L p ) is independent of the hidden state space. Hence, the model can be computed in a simple way, given a feature F i-1 g ∈ R H×W ×C and illumination feature L p ∈ R H×W ×5 , we adopts the LayerNorm followed by our proposed LESSM to integrate the spatial long-term dependency. Following [10] , the input feature are chunk into F1 and F2 in two branches. The first branch projects the feature into a linear layer, followed by a depth-wise convolution, SiLU activation function, accompanied by our proposed augmented local bias and LayerNorm. In the second branch, the features is also projected to a linear layer followed by the SiLU activation function. Finally, features from the two branches are aggregated with the element-wise product and then are projected back to input feature space by linear layer. The entire process can be delineated as:EQUATION)Implicit Retinex-Aware Selective Kernel Module. In our framework, we propose a Retinex-aware kernel selective mechanism (IRSK), where two coupled Retinex-aware priors are used to select the spatial context regions, the maximum and mean values of RGB images can be regarded as a rough illumination prior. IRSK constructs a sequence of depth-wise convolutions with an alterable kernel to select the feature with different receptive field, using a spatial selection mechanism by illumination prior. Inspired by LSKNet [29] , for each of the feature maps from different selective kernel, a Sigmoid activation function is applied to obtain the individual illumination maps from illumination prior. Fig. 2 (d) shows a detailed conceptual illustration of IRSK module where we intuitively demonstrate how the implicit Retinex-aware module works. The above process can be formulated as:EQUATION)The output of the Retinex-aware maps are concatenated with the input features via residual connections, followed by a depth-wise convolution, GELU activation function and convolution layer.EQUATIONEQUATION4 Experiments 
benchmark datasets and implementation details: 
 Datasets. We employ five paired low light image datasets for evaluation, including LOL-V2-real [59] , LOL-v2-syn [59] , SMID [4] , SDSD-indoor [48] and SDSD-outdoor [48] datasets. Therein, LOL-V2-real contains 689 low-normal light paired images for training and 100 pairs for testing; LOL-V2-syn includes 900 paired images for training and the 100 pairs for testing; Besides, SMID is composed of the 15763 short-long exposure paired images for training and the remaining images for testing; SDSD-indoor and SDSD-outdoor are all subsets of SDSD dataset (the static version), which extract the paired images from 62 and 116 pairs for training, and the left 6 and 10 pairs for testing.Implementation Details. We implement MambaLLIE in PyTorch [40] on a server with the 4090GPUs. Random cropping the image pairs into 128 × 128 patches as training samples, data augmentation is performed on the training samples such as rotation and flipping. The batch size is 8. In terms of optimization procedure, Adam [24] is adopted as the optimizer with β 1 = 0.9 and β 2 = 0.999; The training iterations is set to 1.5 × 10 5 . The initial e learning rate is set to 2 × 10 -4 and steadily decreased by by the cosine annealing scheme. The loss criterion is mean absolute error (MAE), thus peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [52] is selected as the evaluation metrics for the paired datasets. Quantitative Comparison. As shown in Tab. 1, we evaluated the performance of our MambaLLIE against 11 SOTA image enhancement methods, including RetinexNet [53] , DeepUPE [49] , SID [4] , KinD [64] , MIRNet [63] , EnGan [21] , Restormer [62] , SNR-Net [56] , QuadPrior [51] , MambaIR [16] and Retinexformer [3] . Our MambaLLIE demonstrates superior performance than SOTA methods on the adopted benchmark datasets in terms of PSNR and SSIM, while achieves comparable results of SSIM with the SOTA methods in LOL-V2-real and SDSD-outdoor. Therein, when the parameters are roughly similar, our MambaLLIE achieves an average improvement of 0.2dB on benchmark datasets compared to the Transformer based SOTA method, i.e. Retinexformer. Compared with the earlier Transformer based SNR-Net, MambaLLIE outperforms it by average 1dB PSNR on the all datasets. When compared to the MambaIR, MambaLLIE achieves 1.7, 0.32, 2.19, 1.15 and 0.25dB PSNR improvements on the adopted datasets, respectively. Besides, Our MambLLIE gains the improvements over 7 dB on all datasets than traditional Retinex-based models, such as RetinexNet, DeepUPE and KinD.Qualitative Comparison. Figs. 3 & 4 report the vision results for comparing our method with latest the SOTA methods and traditional Retinex-based models. Existing methods suffer from insufficient illumination and fail to restore the details as shown in Fig. 3 . As we can see, color distortion and image degradation also affect the enhanced results of previous methods in Fig. 4 , yet our MambaLLIE not only enhances brightness but also faithfully preserves colors with reference to ground truth images, all while restoring the details.Enhancing low-light images in real-world scenarios is exceptionally challenging because, in addition to benefiting downstream tasks such as dark object detection, the enhanced images must also be visually pleasing to human perception. Low Light Object Detection. We utilized ExDark dataset [32] to compare the enhancement of preprocessing methods for high-level vision tasks. There are 7363 challenging low light images annotated with 12 bounding box classes, of which 5,890 for training and 1,473 for testing. Note that all supervised methods were pretrained on the LOL-V2-syn dataset, the low light image underwent different enhancement methods and then finetuned YOLOv3 [43] as the object detector. As shown in Tab. 2, our methods achieved the best average result compared with other models, and yielded the best results on Car, Chair, Cup, People and Table classes. Fig. 5 (a) further reported the visual comparison, compared with suboptimal preprocessing method SCI, detector through our MambaLLIE can detect the objects in extreme dark regions including two persons and a chair, while other methods failed.Face Detection We investigate the performance of low-light image enhancement methods on face detection in the dark. We use the DARK FACE dataset [60] and randomly sample 300 images for evaluation. The RetinaFace [44] is used as the face detector and fed with the results of different LLIE methods. We show the results of different methods in Fig. 5 (b) and Tab. 3. In general, MambaLLIE achieves the better mAP score and visual detection result. Please note that the effectiveness of face detection in low light conditions depends not only on the quality of the enhancement results but also on the specific face detection algorithm employed. We utilize the pre-trained RetinaFace model to assess the performance of different low light image enhancement methods to some extent.User Study. We conducted a user study to evaluate the human visual perception quality of the enhanced results in challenging scenarios. Due to the lack of the ideal reference for training, we selected the pretrained model from the benchmarks to enhance the photos. There are 7 random selected low light images from the benchmarks and ExDark datasets under different lighting conditions. Human perception primarily focuses on the presence of global visual effect, local detail, color distortion (noise), which significantly reflect the quality of the enhanced images. Thus, We assigned ratings on a scale of 1 (worst) to 5 (best), evaluating the quality of the enhancements in terms of overall rating, local detail and color distortion(noise), respectively. Overall, 70 participants were invited to assess the visual quality. The average scores are reported in Tab. 4, our MambaLLIE achieves the best score in the involved voting aspects. Fig. 5 (c) shows the toy example of user study, which display the input and the random enhanced results and local details by different algorithms.Perceptual Evaluation. We compare two non-reference perceptual metrics, MUSIQ [23] and NIMA [6] , on five unpaired datasets, including LIME [17] , VV [47] , NPE [50] , MEF [37] , and DICM [26] .Experimental evaluations in Tab. 5 show the superiority of our method over SOTAs with better perceptual evaluation in most comparisons, in terms of the NIMA scores, our method also achieves competitive results across all datasets. Experimental comparisons against the diffusion-based model DiffLL [20] also underline our method's robustness. These results are visually supported in Fig. 5(d) , where our method demonstrates qualitative improvements that align well with its perceptual metrics.Overall, the evaluations illustrate the superiority of our enhancement approach, achieving enhanced perceptual quality and setting a new LLIE benchmark on unpaired datasets.Implicit Retinex-Aware Framework. We compare the improvement of using a implicit Retinexaware model with the end-to-end and explicit Retinex-aware models. Specifically, Baseline-1 is a simple variant of our MambnaLLIE by removing Retinex-aware guidence, directly uses the standard vision state space module (VSSM) to process flattened vision data in our proposed UNet-shaped framework, following the Norm → VSSM → Norm → channel attention layer flow as referenced in [16] . Baseline-2 is designed to estimate the illumination map and then light up the low light image by element-wise multiplication, namely aims to estimate the illumination map instead of directly predicting the enhanced image, and then restores the enhanced result by N = R • Ĩ. Tab. 6 reveals our implicit Retinex-Aware framework significantly outperforms Baseline-1 with the improvement of 1.25dB in PSNR, while achieving a PSNR enhancement of 1.00 dB compared to Baseline-2 on SDSD-indoor dataset. Selective Kernel Behavior. We further investigate the kernel selection behavior in our MambaLLIE as shown in Fig. 6 . We find the implicit Retinex-aware selection pattern tend to learn two independent positive and negative illumination, resulting in complementary features. Compared with explicit Retinex-based methods, our IRSK can guide from a flexible deeper neural representation. The quantitative results are reported in Tab. 7. Different with LSKNet [29] , we put small kernels in front and larger kernels in higher levels. This is because object detection needs larger receptive field, thus adopts a sequence of depth-wise convolutions with growing kernel and increasing dilation, while has to introduce a lots of padding. But image enhancement may suffers from padding operation at the edge of the image, especially upsampling further expands the padding values. Thus, the the former The details of selective kernel behaviour, the LAM visualization [14] demonstrates influence of similar local information is higher than that of global dependence, our local-enhanced strategy underscores the feature. Besides, the larger receptive fields can provide globally consistent results.small kernels can quickly focus on local information and the the latter kernels contain larger receptive fields for better feature fusion.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Please refer to Section 4.1 Datasets and Implementation Details.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
limitation and discussion: 
 We adopt an implicit Retinex-Aware guidance within a global-then-local state space framework to address global insufficient illumination and local degradation for low light enhancement. However, our approach has several limitations. 1) Unlike end-to-end methods, our technique requires the design of a reasonable illumination prior, which relies on prior experience. 2) Most existing enhancement models, including ours, primarily focus on mean square error and use PSNR and SSIM to evaluate image quality. To mitigate inherent biases in these metrics, we conducted additional real-world experimental evaluations to reconcile the bias and further validate the effectiveness of our approach.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Our code is available at Project Page.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
conclusion: 
 In • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]  
 
 Appendix:  


Decision: The submission received scores of 4, 4, 6, 8. Reviewers recognize that the paper provides some relevant and well-constructed contributions, but there is disagreement about the level of novelty. In particular, reviewers ahrr (elaborated on in the AC-reviewer discussion) and DUEm considered the changes to be mainly incremental compared to MambaIR, while 1bAi considered that the modification follows the general trend of other transformer-based approaches in incprporating local dependencies. In terms of experiments and results, ahrr argued that the improvements are on par with (marginally better) other transformer-based methods, although comparisons were thorough and SOTA had been achieved. Reviewer 1rUY, who gave the score of 8, considered the application of Mamba to the LLE problem to be novel, with other important merits. The authors went into detail in making clarifications in the rebuttal phase.

After taking into account all the material and arguments presented, the AC considers the overall merit of the paper to be sufficient for acceptance. Although the critical comments of the reviewers are valid, and individually the novel components are broadly straightforward, as a whole there is sufficient non-trivial contribution in terms of an extension of Mamba to LLE, modifications that improve the VSSM receptive field (with visualization), and reasonable effectiveness in extending SOTA results.

Reviews: review: paper_summary: paper_summary: This paper introduces a low-light image enhancement module (MambaLLIE). This module has a U-shaped structure and each GLSSB block follows the Transformer-based design. LESSM is proposed to capture the spatial long-term dependency. IRSK is proposed to introduce large and selective kernels for enhancing feature-capturing ability. IRSK also introduces illumination guidance by utilising attention mechanisms.
strength_weakness: strength_weakness: 1. Could this paper state the difference between LESSM and the Vision State Space Module in [14]? It seems these two blocks have similar designs.

2. The core block GLSSB appears to be a combination of [14] and [23]. However, since [14] is proposed for image restoration tasks, this may limit the novelty of this paper.

3. The descriptions of baseline-1 and baseline-2 are not clear. 

4. In Section 4.4 Ablation Study, which dataset is used for the ablation study? How about other datasets? If the dataset used in the ablation study is SDSD-indoor, then the baseline-1 result (PSNR 28.87, SSIM 0.865) is quite similar to [14] (PSNR 28.97, SSIM 0.884), which may indicate that the better performance of this paper relies on Retinex-aware guidance rather than a larger receptive field. However, Retinex-aware guidance has already been explored in [3] and "Low-Light Image Enhancement with Multi-stage Residue Quantization and Brightness-aware Attention."
questions: questions: 1. Could this paper provide an analysis or explanation of why MambaLLIE has a larger receptive field in terms of global and local?
limitations: limitations: This paper discusses limitation in Sec 5 and give potential solutions.
paper_summary: paper_summary: The authors proposed a Mamba-inspired method for LLIE, which is designed to address some challenges of the existing method, MambaIR. By integrating GLSSB consisting of LESSM and IRSK, the authors could make the model capture a large local receptive field while preserving global understanding natures. The overall pipeline is explained in detail and motivated by sufficient reasoning. MambaLLIE achieved SOTA performance on several benchmarks, significantly outperforming prior works.
strength_weakness: strength_weakness: The proposed method is somewhat incremental in that it makes small (although relevant) changes to MambaIR. Since this is not the first work that introduces Mamba for LLIE, a more in-depth analysis and exploration is expected; however, the proposed is a slightly better engineered network structure. The authors are recommended to discuss MambaIR in more detail and clarify technical contributions over MambaIR.
questions: questions: Although the ERF visualization in Figure 1 is impressive, the authors are expected to provide some numeric results for quantitative performance comparisons. 
It is not clear why it is called "retinex-aware selection" in Figure 2(d). 
There are several methods dedicated for face detection in low light. It will be interesting to compare face detection performance compared to such methods,
limitations: limitations: The authors adequately addressed the limitations; but some complexity analysis will be helpful for readers to understand the method.
paper_summary: paper_summary: This paper presents MambaLLIE, an implicit Retinex-awre low light image enhancement framework with modified state space blocks. Specifically, a global-then-local state space block (GLSSB) is designed, which incorporates a local-enhanced state space module (LESSM) and an implicit Retinex-aware selective kernel (IRSK) module. By enhancing the original SSMs with local bias, the proposed MambaLLIE outperforms state-of-the-art CNN and Transformer-based methods.
strength_weakness: strength_weakness: 1. The novelty of this paper is limited. Like many Transformer-based works that embed local modeling capabilities in ViTs, the main contribution of this paper is incorporating local dependencies into SSMs. However, the advantages of the proposed GLSSB compared to ViTs and original SSMs have not been thoroughly analyzed. Furthermore, given the authors' design, the proposed GLSSB should have advantages in many other visual tasks, which need to be demonstrated through more experiments.

2. The writing of this paper needs improvement as there are many unclear descriptions. For example, how is the illumination prior in the paper obtained, how does the proposed IRSK module work specifically, and what are the specific settings for the different variants in the ablation study?

3. Some state-of-the-art works have not been compared (e.g., DiffLL[1]), and the experimental results in this paper do not seem to outperform them.

4. As the low-light enhancement task typically encounters issues like color distortion or other visual artifacts, the authors should compare more perceptual evaluation metrics. Merely reporting PSNR and SSIM is not sufficient.

[1] Jiang et al, Low-Light Image Enhancement with Wavelet-based Diffusion Models. Siggraph asia 2023.
questions: questions: Please refer to the `Weaknesses'.

Moreover, it appears that the color in the last column of Table 2 is incorrectly marked.
limitations: limitations: The authors have discussed the limitations in the paper.
paper_summary: paper_summary: This paper proposes a Mamba-based framework, namely MambaLLIE, for low-light image enhancement. Specifically, the authors claim that they have two technical contributions:

(i) A global-then-local state space block that integrates a local-enhanced state space module and an implicit Retinex-aware selective kernel module to capture intricate global and local dependencies.

(ii) An implicit Retinex-aware selective kernel mechanism to guide deeper neural representations and segregate them into independent positive and negative illumination components before integrating them.
strength_weakness: strength_weakness: (i) The motivation is not clear. To be specific, why use Mamba for low-light image enhancement? The authors claim that the Mamba can help capture long-range dependences. However, Transformer architecture can also model the long-range dependences. Why use Mamba instead of Transformer? Also, why design the space state module like this? The insight of the proposed Mamba should be analyzed in more detail.

(ii) A technique should be explained, i.e., what is the scanning operation in the state space module? The authors mention the scan in Mamba but I could not find any mathematical formula or text description in the paper to explain the computation process of this scan.

(iii) In table 1, although the performance of the proposed MambaLLIE is better than the state-of-the-art Transformer-based method Retinexformer, its computational complexity and memory cost are larger than those of Retinexformer. For example, the FLOPS and the Params are 33% and 38% higher. In addition, what about the comparison of performance on the MIT-Adobe-FiveK dataset (sRGB output mode)?

(iv) In Figure 4, it seems that Retinexformer achieves more visually pleasant results. For example, in the third line of Figure 6, Retinexformer reconstructs the complete edge of the redwood in the left zoomed-in patch but the proposed MambaLLIE fails. Thus, it is better to compare MambaLLIE and the state-of-the-art method on unpaired datasets such as the LIME, NPE, MEF, DICM, and VV datasets.

(v) The source code and pre-trained models have not been submitted. The reproducibility cannot be checked.
questions: questions: I am curious about the training time comparison between the proposed MambaLLIE and other methods.
limitations: limitations: Yes, the authors have analyzed the limitations in the main paper.

Comments: title: Response to the author rebuttal
comment: Thank you for your valuable time and consideration. We would like to clarify the novelty of our method through the following key points:

1. **First to Effectively Combine Mamba and Retinex**: Our method is the first to successfully integrate Mamba and Retinex theory and propose MambaLLIE for low-light enhancement task, achieving state-of-the-art results.

2. **Fundamental Improvement of the State-Space Model in MambaIR**: We introduce a novel state-space model, LESSM (Equation 9), specifically designed for low-light enhancement. This represents a fundamental improvement over the State-Space Model of MambaIR, which merely adds a convolution after the vanilla VSSM to restore neighborhood similarity. In contrast, our LESSM is both simple and effective, as demonstrated by its innovative integration of 2D dependencies into the directional scan (Equation 10), ensuring a closer arrangement of relevant local tokens. Experimental results show that our model improves PSNR by an average of **4.43%** across all datasets compared to MambaIR, while reducing FLOPs by **65.63%** and the parameter count by **46.98%**.

3. **Retinex-Aware Selective Mechanism (IRSK)**: Our IRSK mechanism expands the local receptive field through a Retinex-aware selective approach by coupling two illumination maps (Equation 12). This allows for *flexible selection* of the region of interest with different kernels, as illustrated in Figure 2(d). In contrast, [3] introduces a relatively fixed illumination feature into the attention mechanism, while *our selective maps could capture the effective interested regions with the normal and reversed feature and illumination maps*. We have discussed the advantages of our selective kernel behavior in **Lines 274-284*.

4. **Mamba for LLIE**: The low-light enhancement task is challenged by global color degradation and local noise disturbance. This led us to develop a LLIE model that fuses a global model and a local model. Since Mamba is a global-aware estimator with linear or near-linear scaling complexity, we chose it as our global baseline. Additionally, the CNN-based model preserves local 2D dependencies, refining local degradation to enhance LESSM and IRSK. Benchmark and real-world experimental evaluations indicate that our method **outperforms** previous approaches in both qualitative and quantitative comparisons.

We thank you again for your feedback and hope that our explanation can make you better understand our contribution and efforts.

Best regards,

The authors
Thank you for your detailed response. My primary concern is that, despite using Mamba to model global information, the improvement over [14] and [3] is limited. Moreover, the proposed method appears to make only minor changes to MambaIR. Furthermore, as shown in Table 4, the enhancement over [14] largely depends on using illumination guidance L_p in Eq. 10 and 12, which is not a novel approach in low-light image enhancement tasks. This raises the question of whether the Mamba architecture has genuinely contributed to low-light image enhancement. Therefore, I maintain my rating.
## Dear Reviewer 1bAi
Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. Additionally, if you have any further concerns or suggestions, please feel free to let us know. We eagerly await your response and look forward to hearing from you. Thank you for your valuable time and consideration!

Best regards,

The authors
## Dear Reviewer ahrr
Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. Additionally, if you have any further concerns or suggestions, please feel free to let us know. We eagerly await your response and look forward to hearing from you. Thank you for your valuable time and consideration!

Best regards,

The authors
We sincerely appreciate your recognition of our work and effort.  YES, You are right,  as discussed in [3], *"O(IG-MSA) of Retinexformer is linear to the spatial size, mainly comes from the k computations of the two matrix multiplication of attention."* Compared to the complexity of the global MSA used by previous Transformer methods, Retinexformer significantly reduces the computational complexity of attention while enhancing performance in LLIE tasks. Besides, our MambaLLIE inherits the linear complexity of SSM for visual representation learning, and a series of improvements in state space and Retinex-aware module are adopted to improve the performance, which have not significant and even negligible impact on overall complexity.

Finally, we once again extend our gratitude to the reviewer for the constructive feedback during the review and rebuttal stages, giving a final score of **strong accept** for our manuscript. The insightful comments on complexity remind us to maintain rigorous discussions and analyses in our future work.
Thanks for your response. Most of my concerns have been addressed except one.

You claim that `Mamba over Transformers due to its efficient long-range dependency modeling with linear complexity. Transformers can model long-range dependencies, but they typically involve higher computational complexity and memory costs.`

However, this is not true. Many Transformers also enjoy linear computational complexity, e.g., Retinexformer. It takes much shorter time to train than the Mamba-based methods.

Anyway, this is a good attempt to explore the potential of Mamba in LLE. Thus, I decided to raise my score to `strong accept` to support you. Thanks for your effort.