TinyLUT: Tiny Look-Up Table for Efficient Image Restoration at the Edge

Authors: huanan  li
juntao  guan
rui  lai
sijun  ma
lin  gu
zhangming  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:58:13

Abstract: Look-up tables(LUTs)-based methods have recently shown enormous potential in image restoration tasks, which are capable of significantly accelerating the inference. However, the size of LUT exhibits exponential growth with the convolution kernel size, creating a storage bottleneck for its broader application on edge devices. Here, we address the storage explosion challenge to promote the capacity of mapping the complex CNN models by LUT. We introduce an innovative separable mapping strategy to achieve over $7\times$ storage reduction, transforming the storage from exponential dependence on kernel size to a linear relationship. Moreover, we design a dynamic discretization mechanism to decompose the activation and compress the quantization scale that further shrinks the LUT storage by $4.48\times$. As a result, the storage requirement of our proposed TinyLUT is around 4.1\% of MuLUT-SDY-X2 and amenable to on-chip cache, yielding competitive accuracy with over $5\times$ lower inference latency on Raspberry 4B than FSRCNN. Our proposed TinyLUT enables superior inference speed on edge devices with new state-of-the-art accuracy on both of image super-resolution and denoising, showcasing the potential of applying this method to various image restoration tasks at the edge. The codes are available at: https://github.com/Jonas-KD/TinyLUT.

Main paper: 
 introduction: 
 With image sensors widely used in IoT applications, the demand for image restoration on edge devices has been stimulated. For decades, classic methods including interpolation [1, 2] , sparse coding [3, 4, 5] and domain transformation [6] have been proposed. In recent years, deep learning based algorithms [7, 8, 9, 10, 11] have made encouraging progress in restoration quality, while suffering from high computational load and long inference latency. These issues limit their extensive applications in resource-constrained IoT terminals. Thus it can be seen, the growing demand for edge devices (e.g. smartphones and Raspberry Pi) calls for an extremely lightweight solution for image restoration with high quality. Given this, researchers develop LUT-based methods [12, 13, 14] to supplant the intensive convolutional computations with direct memory access operation for accelerating the inference, and exhibiting enormous potential.In recent research, SRLUT [15] initially transfers the output values of a small trained deep network with receptive field (RF) size 4 to uniformly sampled LUT. To further improve the super resolution Table 1 : LUT size estimation when storing 8bit data with output entities r = 4. Compared with the full size, the sampled operation effectively reduces the storage consumption of LUT. Furthermore, separable mapping strategy(SMS) and trained dynamic discretization mechanism (DDM) achieve an exaggerated reduction result. quality, Li et al. [14] increases the RF size through the different kernel shape combination with multiple LUTs. Meanwhile, Ma et al. [13] constructs a series-parallel LUT framework based on multiple LUTs cascade to improve the prediction accuracy. However, the existing LUT-based schemes face the challenge of storage explosion when mapping 8bit image with larger than 2 × 2 kernel size. The storage explosion invalidates previous methods where the required storage of LUT grows exponentially as the number of indexing entries (i.e. input entities) increase. For example, 3 × 3 kernel, commonly used in CNNs, will need a tremendous storage up to (2 8 ) 9 × 8bit= 4ZB and (2 4 + 1) 9 × 8bit= 110GB in full LUT and uniformly sampled LUT with 8bit data respectively. Therefore, the issue of storage explosion presents severe challenge when mapping the complex models by LUT on edge devices.To overcome the storage explosion challenge, we propose the separable mapping strategy(SMS) and dynamic discretization mechanism (DDM) to decouple the kernel and activation, respectively. As reported in Table 1 , the reduction of LUT storage reaches exponential levels using SMS and further achieves a 4.48× times reduction by DDM when the RF size is 3 × 3 and storing 8bit data.From the perspective of kernel decomposition, considering the exponential relationship between the dimension of input entries and LUT size, we propose the innovative SMS to decouple the convolution kernel by spatial location relationship. SMS decomposes the convolution weight of n input entities into the parallelization of n component independent sub-operations, which are instantiated as 1 × 1 kernel size respectively to reduce the number of input entries in LUT-based inference, consequently reducing the dimension of nD LUT. Then according to the additivity of convolutions with compatible kernel sizes [16] , we reconstruct the feature space by averaging each output of n parallel layers. Finally, each branch with 1 × 1 kernel are individually mapped by 1-dimension LUT (1D LUT) with 1 input entity for s possible values. This approach reduces the size of nD LUT from s n to s × n and allows vanilla convolution to be LUT-mapped on resource-constrained devices.As for the activation compression, previous methods [15, 14, 17] uniformly sample the input values to reduce the possible values of s and apply interpolation to recover the inference accuracy. We analyze the advantages of SPLUT [13] and propose dynamic discretization mechanism (DDM) to decompose the activation into Most Significant Bits (MSBs) and Least Significant Bits (LSBs). Moreover, inspired by the activation quantization methods [18, 19] , DDM introduces the learnable clipping parameters to explicitly compress the quantization scale of MSBs and LSBs activation, thereby reducing the possible pixels values and decreasing LUT scales.As illustrated in Fig 1 , we take the classic image restoration task single image super resolution (SISR) as an example. Our proposed TinyLUT-F consumes only 4.1% storage of MuLUT-SDY-X2 [14] to achieve over 0.58dB PSNR increase in Set5 testset. Compared to other LUT-based methods, TinyLUT significantly reduces storage overhead while improving accuracy, expanding the application of LUT-based approaches on edge devices.The main contributions can be summarized as follows:• We reveal the scheme of storage explosion, being the key problem that limits the application of LUT in edge devices. Focus on this, we propose an innovative separable mapping strategy(SMS) to realize dimensional reduction of LUT and solve the storage explosion problem.• We design a dynamic discretization mechanism(DDM) to decompose the activation and compress the corresponding quantization scale, further shrinking the storage requirement without considerable accuracy loss.• Our comprehensive experiments demonstrate that TinyLUT, constructed by SMS and DDM, achieves significant restoration accuracy over previous LUT methods with minimal storage consumption, showing its potential for application on edge devices. 
related works: 
 Deep Learning for Image Restoration: In recent years, image restoration performance has been rapidly developed through deep learning methods [20, 8, 21, 22, 10, 7, 11, 23, 24, 12, 25, 26] and achieved outstanding accuracy. The authors of [24] presented a super resolution method using very deep networks and residual learning. Zhang et al. [7] proposed a deep convolutional neural network with residual learning to handle image restoration tasks such as denoising and SISR. FFDNet [10] achieves more efficient and flexible than DnCNN [7] by down-sampled method. Also it have the ability to robustly control the trade-off between noise reduction and detail preservation. As a fast and memory-efficient network, FMEN [23] is constructed by enhanced residual block and highfrequency attention block. Dong et al. [8] explored a more efficient network by re-designing the SRCNN structure. In [27] author proposed a deep convolutional network within a Laplacian pyramid framework for fast and accurate image super-resolution. Denoising results of FastDVDnet [28] feature remarkable temporal coherence, very low flickering, and excellent detail preservation. However, these CNN methods suffer form intensive computations of convolutional operator, which limits their deployment on edge devices.LUT-based methods are commonly used in hardware systems [29] and some embedded devices [30] to accelerate CNNs by pre-storing the results of complex functions with high efficient retrieval operations. In image restoration tasks, LUTs are commonly used in video coding [31, 32] , tone-mapping [33] and image enhancement [17] . Furthermore, deep learning methods based on LUTs have also emerged in SISR [15, 14, 13] and gradually garnered attention. SRLUT [15] establishes a spatial mapping from a local RF block to the corresponding highresolution pixel. In MuLUT [14] and SPLUT [13] , the authors proposed the collaborating structure by multiple LUTs to increase the model's RF and improve the inference accuracy. Furthermore, Liu et al. [34] found that the interaction of space and channel features in vanilla convolution allows previous methods to increase the RF at the cost of linearly increasing the LUT size. They proposed RCLUT based on the Reconstructed Convolution(RC) module to enlarge the RF, achieving significant performance with less storage. Realizing the dilemma between the performance improvement and storage of LUT-based models, Li et al. [35] introduce the Diagonal-First Compression(DFC) framework to achieve a better trade-off between storage size and performance for image restoration. Additionally, they also designed a new structure, SPF-LUT, to further improve the performance of LUT-based models. However, the RF and channels of LUT-based methods are limited to adapt the trade-off between storage and accuracy. In comparison, our work is focused on developing a lightweight LUT-based framework to improve the image restoration accuracy. 3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of the work are discussed in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview of tinylut: 
 The structure of TinyLUT and foundational components are depicted in Fig 2 . We design two parallel branches with cascaded LUTs to process Most Significant Bits (MSBs) and Least Significant Bits (LSBs) data respectively, and introduce separable mapping strategy and dynamic discretization mechanism.According to the previous LUT-based methods we follow two simple design rules:(i) to improve model accuracy, the RF size needs to be increased; and (ii) the number and storage requirement of LUTs need to be reduced to adapt edge devices. Therefore, our CNN network (Fig 2(a) ) is mainly inspired by the architecture of SRLUT [15] and DnCNN [7] . Specifically, the CNN model is built on standard convolution, depthwise convolution (DwConv) and PwBlock. The PwBlock 2(b) is a channel combining module based on SMS and described in Section 3.2. Based on SMS and DDM, the standard convolution, DwConv and PwBlock are mapped by LUT respectively to build standard convolution mapped LUT (ScLUT) , depthwise LUT (DwLUT) and pointwise LUT (PwLUT). In particular, we refer the philosophy of depthwise separable convolutions [36] to combine DwLUT and PwLUT to depthwise separable LUT(DSLUT). The mapping details for DwLUT and PwLUT are described in Section 3.2 and Section 3.3.As in depthwise separable convolutions [36] , the DwLUT optimizes the LUT size while maintaining a large effective RF. The PwLUT is used for cross-channel feature information integration and model channel number transformation. We introduce skip connections to fuse the real-value inputs and the retrieval outputs between DwLUT and PwLUT to avoid the degradation. The RF and output channels are set to 3 × 3 and 16. Due to each color channel having to be processed independently as in SRLUT [15] , the input channel of ScLUT is set to 1.According to the different number of DSLUT serial stacks, we build TinyLUT-F with 7 stacked. TinyLUT-S is built by ScLUT and double PwLUTs. As shown in Fig 2(c ), TinyLUT is constructed by DwLUT and PwLUT in each branch. An additional PwLUT is added at the end of each branch to adjust the number of output channels of different image restoration task. In super resolution, the output channel number of the last PwLUT is r 2 to adapt to the pixelshuffle [21] , where r is the upscaling factor. In order to further expand the RF size, we introduce rotational ensemble trick [15] in the inference.In Table 1 , we provide an estimation of the LUT size of four output channels and single input. It is evident that the SRLUT continues to face the issue of storage explosion, resulting in approximately EQUATIONThe x i,j represents the input feature data, and F out represents the output feature and w i,j stands for the convolution kernel, while n × n denotes the size. After transferring by original multiple dimensions LUTs, the retrieval process can be represented as:EQUATIONIn DwLUT, we utilize multiple 1 × 1 convolution kernels to build depthwise convolutions. As shown in Fig 3,  we decompose the convolution kernel into n 2 of sizeC out × 1 × 1 × 1.Mapping by 1D LUTs, the original LUT retrieval process can be reconstructed as:EQUATIONAfter decomposing and mapping, the storage consumption reduces from 256n 2 × C out × 8bit to 256 × n × n × C out × 8bit.Meanwhile, the output of the multiple 1D LUTs is averaged to achieve an approximation of the standard LUT inference.The pointwise convolution applies a 1 × 1 convolution to combine the outputs the depthwise convolution [36] . The operation of pointwise convolution with a kernel size of C out × C in × 1 × 1 is defined as:EQUATIONIn this context, x represents the input data and w represents weights. It still faces the problem of exponential size increase in the LUT mapping process due to multiple channels input simultaneously.Then we decouple the input channels in pointwise convolution. Subsequently, it is decoupled intoC in number of C out × 1 × 1 × 1 convolution kernel.The storage space required for the pointwise convolution, is reduced from(256 Cin ) × C out × 8bit to 256 × C in × C out × 8bit.The final output channel is set to C out . At this point, the original cross-channel per-pixel LUT retrieval operation is reconstructed as:EQUATIONAs depicted in Eq 5, the quantity of LUTs is denoted as C in . Additionally, the number of output channels for each LUT retrieval is represented as C out . According to Eq 4 and Eq 5, the PwBlock in Fig 2(b) consists of several parallel branches with the number of channels. Each branch includes 3 pointwise convolutions followed by ReLU except for the last layer. The channels in the pointwise convolution is set to 64 and that of the last layer is set to 16.In previous methods, in order to further reduce the volume requirement of LUT, it is common to sample the indexing entries. Related algorithms [15, 14, 37] use interpolation methods to restore accuracy after table retrieval. However interpolating in large RF and multiple channels will inevitably result in a significant amount of computation. To address this issue, we proposes the dynamic discretization mechanism (DDM). It adaptively explores the parameterized clipping level of quantization range for each channel based on the gradients with Straight-Through Estimator (STE) [38, 39] , which significantly optimize the equilibrium of accuracy and LUT size.As shown in Fig 2 , an 8bit input data is decomposed into MSBs and LSBs, then obtain output results through their respective branches and add them. It should be noted that the 8bit data stored in the LUT is defined in the domain of -128 to 127, while the data range of MSBs data is[-m, m -1],and the data range of LSBs data is [0, k]. Therefore, in Eq 3 mapping depthwise convolution to LUTs for the MSBs branch, the LUTs size can be obtained by:EQUATIONand the size of LUT in LSBs branch:EQUATIONThis operation reduces the index range of the LUT, realizes the discretization of the LUT and reduces the size.The index range is 2 4 due to the activation data range are equal to LUTs indexes range in the mapping process. Therefore, inspired by activation quantization [18, 19] , we use the learnable clipping parameter α to constrain the bit width of features. Specifically, α my and α ly are used to constrain the MSBs and LSBs of the layer y, respectively, to control the size of the LUT. For example, using F to denote feature data, this operation can be calculated as:EQUATIONThe LUT size is written as :EQUATIONIn Eq 9, n 2 denotes the input entries and r 2 denotes the output entries respectively. In order to reduce the index number of LUTs, we initialize α to 0.8 and then apply L2-regularization for α with the 0.1× regularization parameter λ used to compress the activation precision scale in each layer.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Upon publication, we intend to release the code for broader community access. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
implementation details: 
 Training Settings To demonstrate the effectiveness of our framework, we evaluate TinyLUT on multiple datasets for SISR. The CNN model of TinyLUT is trained in an end-to-end manner. We use DIV2K [40] dataset which has been widely applied in image processing tasks. TinyLUT model is trained for 200000 iterations with Pytorch [41] on Nvidia 3090 GPU. We employed the Adam optimizer [42] , where β 1 = 0.9, β 2 = 0.999. The learning rate is set at 5×10 -3 and was dynamically adjusted using a cosine annealing mechanism [43] . We randomly cropped degraded data into 48 × 48 MuLUT [14] SPLUT [13] FSRCNN [8] TinyLUT HR Figure 4 : Qualitative comparisons of bicubic interpolation, SRLUT [15] , MuLUT [14] , SPLUT [13] , FSRCNN [8] , our TinyLUT and HR images.patches with a batch size of 32. Data augmentation was performed through random rotations and flips. Throughout the experimental process, we employed PSNR and SSIM [44] as evaluation metrics for restoration accuracy. Besides, we measured and reported the runtime on multiple edge devices including Xiaomi 11 smartphones with a Qualcomm Snapdragon 888 CPU and Raspberry 4B.Evaluation Settings For single-image super resolution, assessed the effectiveness of our method on five widely used benchmark datasets: Set5, Set14, BSD100 [45] , Urban100 [46] , and Manga109 [47] . We compare our method with various SISR algorithms based on interpolation which include methods based on deep learning including FSRCNN [8] , VDSR [24] , RCAN [25] and CARN-M [26] , and SR methods based on LUTs, SRLUT [15] , MuLUT [14] , RCLUT [34] , SPF-LUT [35] and SPLUT [13] .In tables, the best result is highlighted in bold.Accuracy and Storage The quantitative results of image super resolution are shown in Table 2 . For single image super resolution task, the SSIM and PSNR values are computed at the Y-channel in the YCbCr color space. As reported in Table 2 , we build a series of TinyLUT models with volume from 37KB to 171KB to fit various edge devices. Our TinyLUT-F model achieves 0.58dB PSNR increase with nearly 24× storage reduction of MuLUT-SDY-X2 [14] . Even for the compact TinyLUT-S model, it only consumes 2.83% memory overhead and achieves 0.4dB PSNR and 0.02 SSIM improvement compared to SRLUT-S [15] . The qualitative results in Fig 4 of MuLUT-SDY-X2 [14] and SPLUT-L [13] have severe ringing artifacts in the white area. On the contrary, our method generates more natural textures and fewer artifacts. As shown in Table 2 , TinyLUT-F achieves much higher PSNR with 9× and 12× lower storage consumption than RCLUT [34] and SPF-LUT+DFC [35] , respectively. While TinyLUT achieves restoring clearer edges and 5× lower latency on Raspberry 4B compared with computation-heavy method(FSRCNN [8] ). In order to further show the capacity of the proposed TinyLUT framework, we remove the pixelshuffle and set the out channels of the last PwLUT to 1. Meanwhile, we adopt the residual learning formulation such as DnCNN [7] to accelerate model training. For additive white Gaussian noise (AWGN), we use Set12 and BSD68 benchmark datasets with noise level 15, 25 and 50. We compare our method with classical DNN method [7] and LUT methods [15, 48] .The quantitative results of image denoising is shown in Table 3 . Compare with LUT-based methods, our method significantly improves PSNR with the minimal storage in image denoising. Meanwhile, TinyLUT-S achieves the fastest inference speed with only 22KB storage overhead compare with DNN based denoising schemes. The denoising results in Fig 5 illustrate that our TinyLUT is able to obtain similar visual quality to DnCNN [7] . The quantitative and qualitative experiments prove the effectiveness of storage reduction and inference acceleration of our approach.In this subsection, image deblocking is used to further assess the generality of TinyLUT in image restoration tasks. Table 4 reports the comparison results of PSNR-B for LUT-based methods on Classic5 [49] and LIVE1 [50] . We also refer to the classical methods and DNN algorithms, including SA-DCT [51] and ARCNN [52] . TinyLUT-F achieves comparable PSNR-B results to the DNN method We perform several ablation experiments for single image super resolution to demonstrate that our contributions in TinyLUT provides quality improvements.The effectiveness of SMS We conduct an experiment with combinations of SMS and TinyLUT-S. In Table 5 , SMS denotes the TinyLUT-S includes SMS without DDM when constructed the model with 8bit data values. Original denotes the TinyLUT-S without SMS and evaluates using the quantized CNN model due to the excessive storage overhead of its LUT model. The accuracy of SMS is competitive with corresponding mapped neural network with 8bit data in SISR. In particular, SMS scheme achieves significant accuracy improvement compared to uniformly sampled method [15] with sampling interval size of 2 4 , while yielding over 7× storage reduction. This demonstrates the effectiveness of storage reduction using SMS with minimal loss of accuracy. LUT sampling method based on DDM has higher compression ratio of storage than the 8bit full sampling LUT, and achieves a better trade-off between storage and accuracy.As shown in Fig 7 , the precision range of the MSBs activation data is adjusted more adaptively by clipping parameter α in SISR, which reduces the possible value range of the input entities of PwLUT in TinyLUT-S, and then reduces the size in MSBs branch. As a comparison, the marked red line in the figure is the s value when the input entities range is fixed to 6bit during mapping in the MSBs of PwLUT. It can be observed from the figure that compressed by the DDM, not only the LUT size is reduced, but also all possible values of input entities are covered in the mapping process. Compared with the 43.6KB of TinyLUT-S-6M2L with a fixed combination of 6 MSBs and 2 LSBs, the size of TinyLUT-S-E is reduced to 36.6KB via the adjustment of DDM, achieving a reduction of 15%. Therefore, DDM yields additional storage reduction compared to decomposing data into fixed MSBs and LSBs [13] . More intuitively, the interpretability of DDM is also illustrated in Fig 7 . 
limitations and future direction: 
 The method in this paper achieves significant reduction in storage with high compatibility when mapping convolutional neural network. However, there are difficulties in mapping other models such as Mamba [53] and Transformer [54, 55] using TinyLUT. This can be further explored the unified mapping approach for other model. Exploring additional image restoration tasks on edge devices using LUTs would also contribute to the community. 
conclusion: 
 In this paper, we analyze previous successful LUT-based deep learning approaches and summarize the key problem of storage explosion, which limits the further popularization of LUT in image restoration on edge devices. To address the storage explosion, we propose the separable mapping strategy (SMS) and dynamic discretization mechanism(DDM) to decompose the kernel and activation, respectively.In particular, we design the TinyLUT framework based on SMS and DDM. By seamlessly integrating these innovations, TinyLUT-F sets a new record for SISR by achieving over 31dB PSNR on the Set5 dataset at just 171KB LUT storage. Overall, extensive experiments across seven benchmark datasets and two classic image restoration tasks demonstrate the effectiveness and efficiency of TinyLUT on resource-constrained devices. 
neurips paper checklist: 
 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main contributions have been clearly summarized in the abstract and Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Upon publication, we intend to supplement the discussion of the broader implications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The data and methods used in the paper are properly credited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Upon publication, we intend to release the well documented code and data for broader community access. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The paper considers lookup table based methods for accelerating CNNs for image reconstruction. The paper notes that the memory complexity of lookup table based methods for image reconstruction grows exponentially in the size of the convolutional kernel, which may create a storage issue. 

The paper received four reviews:
* Reviewer JnS6 (3: reject) main concern is that accelerating CNN-based methods (in particular DnCNN) is not convincing and that more modern architectures should be considered. I agree with the authors' response that CNN based methods are still relevant in particular when speed matters. The reviewer also notes that the paper is not well written, and I fully agree to this concern. 
* Reviewer G5Ti (6: weak accept after rebuttal) notes that the paper should compare to recent works that address the same problem, in particular the paper
[1] 2024 CVPR paper ``Look-Up Table Compression for Efficient Image Restoration'', 
which is about the exact same issue. In the rebuttal the paper compares to the paper [1]. 
* Reviewer 2kHn is also concerned about a lack of comparisons, that comparisons on more tasks (beyond denoising and super-resolution) are needed, and that the performance in terms of PSNR lacks behind state-of-the-art methods. The authors provided additional comparisons and notes that while the performance is slightly lower than state-of-the-art methods, the latency and storage are much better, which is a valid point. I also think that considering denoising and super-resolution is sufficient. 
* Reviewer MFFN (7: accept after rebuttal) has no major concerns after rebuttal.

In summary, the paper proposes a method to effectively reduce the storage complexity of lookup table based methods for accelerating CNNs, and the method is sufficiently validated.

Reviews: review: paper_summary: paper_summary: This paper proposes to reduce the size of LUT for image restoration and to make it applicable on edge devices. The main idea is using depthwise separable convolution to replace the vanilla convolution. When transfer to LUT, the storage is significantly reduced. Experiments show the effectiveness and efficiency of the proposed method.
strength_weakness: strength_weakness: The baseline DnCNN is somehow out-of-state. As described in Limitation, the proposed method based on depthwise separable convolution is suitable for CNN based architectures only, and the LUT-based compression on attention-based method is to be explored.

Quantitative comparisons on color image restoration tasks are needed, e.g., CBSD68. It is better to conduct experiments on more testsets. 

In runtime comparison, it is necessary to compare the complexity of the comparison algorithms. I am interested in the processing time of the proposed method on high-resolution images.

The description of the method is hard to understand. The writing should be improved.
questions: questions: See the weaknesses above.

Please follow the code of ethics in [1], Lena image in Fig 2 may violate this.

[1] https://conferences.ieeeauthorcenter.ieee.org/write-your-paper/improve-your-graphics/
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents TinyLUT, a method that significantly reduces LUT-based image restoration storage requirements for edge devices through separable mapping strategy and dynamic discretization mechanism, achieving competitive accuracy with over 5 times faster inference.
strength_weakness: strength_weakness: 1. The proposed SMS and DDM are used to built TinyLUT model, so how can these two modules help to improve other models? That is, it would be better if you can validate their generalization as general compression modules.
2. Should compare to more recent works, like [a].

[a]. Look-Up Table Compression for Efficient Image Restoration. Li. CVPR 2024
questions: questions: 1. How do the models deploy in edge devices? Which computing engine or accelerating library is used? I think it is helpful for readers to understand the efficiency of the LUT-based methods.
limitations: limitations: None
paper_summary: paper_summary: This paper introduced a separable mapping strategy that solves the storage issue of LUT-based methods. In addition, a dynamic discretization mechanism is designed to decompose the activation and compression quantization scales. Experimental results show the potential of this work for image restoration tasks.
strength_weakness: strength_weakness: 1. There is a lack of comparison with the state-of-the-art methods [1, 2], especially the literature [1] which also addresses the storage problem of LUT-based methods.

2. This paper claims to work on the image restoration problem, however only two restoration tasks were included in the experiments. Validation on more image restoration tasks is needed.

3. The proposed method does not perform well enough on the image denoising task. With similar inference efficiency, the performance of the proposed method lags far behind MULUT-SDY-X2 even more than 1 dB. I am concerned about the effectiveness of this work on a wider range of image restoration tasks.


> 1. Look-Up Table Compression for Efficient Image Restoration. CVPR 24.

>2. Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution. ICCV 23.
questions: questions: Please see Weaknesses.
limitations: limitations: Limitations were included.
paper_summary: paper_summary: To address the storage explosion challenge of LUT, this paper proposes a separable mapping strategy (SMS) and a dynamic discretization mechanism (DDM) to decompose the kernel and activation to reduce the storage consumption. Specifically, the SMS decomposes the convolution into independent sub-operations to reduce the input entries, consequently reducing the dimension of LUT. Additionally, the DDM explicitly compresses the quatization scales with learnable clipping parameters to decrease LUT scales.
strength_weakness: strength_weakness: 1. Although the separable mapping strategy (SMS) could significantly reduce the storage, it may lead to information loss and performance drop. As shown in Figure 3, the SMS obtains the final result using individually mapped indexes and neglects the relationship between the local indexes, which is important for convolutions. 
2. The motivations and benefits of activation decomposition in the DDM are unclear. Moreover, this paper does not sufficiently discuss the difference between DDM and former activation decomposition used in SPLUT.
3. The discussion of LUT-based methods in the related works is insufficient. The key ideas and limitations of these methods should be elaborated.
questions: questions: 1. To my understanding, the proposed strategy can not be applied to transformer-based model like SwinIR. Why is SwinIR introduced in the related works, and why is it categorized under CNN methods?
2. Different from previous methods, the proposed method is not contrained by the storage. Can the proposed method achieve better performance by increasing the RF and channels?
3. Why is the denoising experiment placed after the ablation experiments? The reviewer suggests combining it with the SR evaluation.
limitations: limitations: The limitations have been discussed in the paper.

Comments: title: Thank you for engaging in the discussion!
Thank you for engaging in the discussion!
Thank you for engaging in the discussion!
Thank you for engaging in the discussion!
Thanks for the reviewer's reply!
Reply
reply to the author's rebuttal
Post Rebuttal Comments
Correction of Improper Use of Lenna Images and Dataset Supplementation
Correction Regarding the Improper Use of the Lena Image
comment: We would like to thank the reviewer for engaging in the discussion and for reminding us of the violation regarding the Lena image. We will ensure that all edits mentioned in the rebuttal are incorporated when revising our paper. Thanks again for your participation in the discussion!
We greatly appreciate your time and dedication to providing us with your valuable feedback. We hope we have addressed the concerns, but if there is anything else that needs clarification or further discussion, please do not hesitate to let us know. We will ensure that all edits mentioned in the rebuttal are incorporated when revising our paper. Thanks again for your participation in the discussion!
We would like to thank the reviewer for engaging in the discussion. We will ensure that all edits mentioned in the rebuttal are incorporated when revising our paper. Regarding the inference efficiency you mentioned, it will be one of the directions for our future research. Thanks again for your participation in the discussion!
We would like to thank the reviewer for engaging in the discussion and increasing the score. We will ensure that all edits mentioned in the rebuttal are incorporated when revising our paper. Thanks again for your participation in the discussion!
Thanks for the reviewer's reply. As demonstrated by the reviewer, the accuracy of TinyLUT-S in denoising is indeed lower than MULUT-SDY-X2, although TinyLUT-S is more efficient in storage and inference latency. However, it should also be noted that the denoising accuracy on the larger TinyLUT-F version is significantly higher than all existing LUT models, with an average increase of 0.6dB compared to MULUT-SDY-X2 and 35% storage reduction. Meanwhile, TinyLUT-F achieves a 0.49dB accuracy increase over MULUT-SDYEHO-X2 with only 19% storage consumption. While the accuracy of methods such as MULUT reaches its upper limit due to storage limitations. This indicates that our method addresses the storage explosion challenge of LUT-based methods, which is also the most important contribution of this paper and achieves a better balance between storage and accuracy. 

Following the reviewer's insightful suggestions, we used multithreading and residual merging operations to accelerate the inference latency from 383ms to 254ms, and we noticed that TinyLUT-F still has potential for optimization in speed, such as referencing the proven efficiency enhancing LayerMerge [1] technique and other merging methods [2,3,4]. Thanks to the reviewer for the suggestions and questions regarding inference efficiency. This has allowed us to identify the shortcomings of TinyLUT in balancing inference efficiency and accuracy, which will also be an important direction for optimizing TinyLUT models in the future.

[1] Kim et al. “LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging”, ICML2024

[2] Kim et al. “Efficient latency-aware cnn depth compression via two-stage dynamic programming”, ICML2023

[3] Dror et al. “Layer folding: Neural network depth reduction using activation linearization”, BMVC2022

[4] Fu et al. “Depthshrinker: A new compression paradigm towards boosting real-hardware efficiency of compact neural networks”, ICML2022
Thanks for the reviewer's reply. We fully agree that ViT in edge device scenarios is a very promising emerging direction! The raised papers indeed demonstrate the growing efforts in breaking into this new field in academia in the last two years.


In this NeurIPS submission, we aim to share our recent progress on strategies that we believe, and as also echoed by reviewers, can directly benefit the existing edge computing community where CNN is still the majority.


Thanks to the kind suggestion of reviewer JnS6, we are pleased to cite the raised papers in the discussion section, as they not only enrich our manuscript but also serve to highlight and encourage further research efforts on ViT methods for edge devices.
Thanks to your response, some of my concerns were addressed. The authors' response addresses the ethical issues well. However, I remain concerned about the performance of this work on a wider range of image restoration tasks. The performance of the proposed method lags too far behind on the denoising task, even though it is more efficient. It is difficult to argue that the method achieves a better trade-off between performance and efficiency, especially with respect to the MULUT-SDY-X2 method. Therefore, I would like to keep the previous rating.
Thanks for the authors' reply. Part of my concerns are addressed. Thanks for the efforts of the Ethics reviewers. Although the authors conducted detailed experiments to demonstrate the advantages of the proposed TinyLUT and its variants, I still believe that accelerating the CNN-based image restoration methods is not very convincing, even considering the edge device scenario where domain-specific accelerator (DSA) is still an option. In high-level vision tasks, infering ViT-based models on edge devices has been investigated [1, 2, 3].

[1] Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In European Conference on Computer Vision, 2022
[2] Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Swiftformer: Efficient additive attention for transformerbased real-time mobile vision applications. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023
[3] Shashank Nag, Logan Liberty, Aishwarya Sivakumar, Neeraja J Yadwadkar, Lizy Kurian John. Lightweight Vision Transformers for Low Energy Edge Inference. ISCA 2024 workshop MLArchSys (Machine Learning for Computer Architecture and Systems).
Thank you for the careful explanation. After reviewing the rebuttal and considering the feedback from other reviewers, my concerns have been addressed. I have raised my score to Accept, as this work effectively tackles the storage explosion of LUTs and provides a valuable image restoration method for edge devices.
Thanks for the important reminder from Reviewer vpip. Regarding the replacement of Lenna images mentioned by the reviewer and an additional section to introduce the characteristics and licenses of the datasets used in the paper. 

We comply with the reviewer's suggestion and have replaced the Lena image in Fig. 2 with the Baby image from the Set5[2] test set, as described in the General Rebuttal:
> Regarding the violation of the Lena image mentioned by reviewer JnS6, we will use the Baby image from the Set5 test set to modify Figure 2 in our paper. We have ensured that Lena was not used in the training process and found that removing it from testing does not affect the conclusion of our paper.

and the Rebuttal to Reviewer JnS6:
> We comply with the reviewer's suggestion and replace the Lena image in Fig. 2 with the Baby image in the Set5 test dataset. Meanwhile, we confirmed that Lena was not included in the training set. We also removed the Lena image from the Set 14 test dataset and validated the accuracy of each method again. As the PSNR shown in Table 4, the results indicate that after removing Lena image, our TinyLUT still achieves competitive results and does not affect the ranking of our method.

Meanwhile, we have ensured that Lena was not used in the training process, and we found that removing it from testing does not affect the conclusions of our paper.

We comply with the reviewer's suggestion and have introduced the characteristics and licenses of the datasets used in papers. 
DIV2K[1] is a popular single-image super-resolution dataset that contains 1,000 images with different scenes and is split into 800 for training, 100 for validation, and 100 for testing. The licenses of DIV2K are Custom (non-commercial), and the specific terms of use on the DIV2K official website are: "Please note that this dataset is made available for academic research purposes only. All the images are collected from the Internet, and the copyright belongs to the original owners. If any of the images belong to you and you would like it removed, please kindly inform us, and we will remove it from our dataset immediately."

The Set5[2] is a dataset consisting of 5 images (“baby”, “bird”, “butterfly”, “head”, “woman”) commonly used for testing performance of Image Super-Resolution models. The licensing information for Set5 is for academic use only. 

The Set14[3] is a dataset consisting of 14 images commonly used for testing performance of Image Super-Resolution models with academic use only. 

The Urban100[4] contains 100 images of urban scenes and under CC license[4]. It commonly used as a test set to evaluate the performance of super-resolution models. 

BSD100[5] is a classical image dataset having 100 test images proposed by Martin et al. The licensing information for BSD100 is as follows in the official website: “You are free to download a portion of the dataset for non-commercial research and educational purposes. In exchange, we request only that you make available to us the results of running your segmentation or boundary detection algorithm on the test set as described below. Work based on the dataset should cite the Martin et al. (2001) paper”. 

Manga109[6,7] is composed of 109 manga volumes drawn by professional manga artists in Japan. The licenses of Manga109 is Custom (Manga109: research-only). The rules of using Manga109 is as follows in the official website: 
“The data set is to be used for academic purposes by non-commercial organizations. 
Data shall not be transferred to a third party.
When including manga material in an academic paper or video, users should include the relevant author’s name as “courtesy of [Author’s Name]”.

Thank you for your attention to this issue. We hope that this modification can meet the requirements of ethical review. If you have any other suggestions or feedback, please let us know.

[1] Agustsson et al. “Ntire 2017 challenge on single image super-resolution: Dataset and study”, CVPRW 2017

[2]Bevilacqua et al. "Low-complexity single-image super-resolution based on nonnegative neighbor embedding", BMVC 2012

[3] Zeyde et al. “On single image scale-up using sparse-representations”, Curves and Surfaces - 7th International Conference 2010

[4] Huang et al. “Single Image Super-Resolution from Transformed Self-Exemplars”, CVPR2015

[5]Martin et al. “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics”, ICCV2002

[6]Matsui et al. “Sketch-based Manga Retrieval using Manga109 Dataset”, Multimedia Tools and Applications 2017

[7]Aizawa et al. “Building a Manga Dataset ``Manga109'' with Annotations for Multimedia Applications”, IEEE MultiMedia 2020
Thanks for the important reminder from Reviewer HqQt. We comply with the reviewer's suggestion and have replaced the Lena image in Fig. 2 with the Baby image from the Set5[1] test set, as described in the General Rebuttal:
>Regarding the violation of the Lena image mentioned by reviewer JnS6, we will use the Baby image from the Set5 test set to modify Figure 2 in our paper. We have ensured that Lena was not used in the training process and found that removing it from testing does not affect the conclusion of our paper.

and the Rebuttal to Reviewer JnS6:

>We comply with the reviewer's suggestion and replace the Lena image in Fig. 2 with the Baby image in the Set5 test dataset. Meanwhile, we confirmed that Lena was not included in the training set. We also removed the Lena image from the Set 14 test dataset and validated the accuracy of each method again. As the PSNR shown in Table 4, the results indicate that after removing Lena image, our TinyLUT still achieves competitive results and does not affect the ranking of our method.

Meanwhile, we have ensured that Lena was not used in the training process, and we found that removing it from testing does not affect the conclusions of our paper.

[1]Bevilacqua et al. "Low-complexity single-image super-resolution based on nonnegative neighbor embedding", BMVC 2012