TinyLUT: Tiny Look-Up Table for Efficient Image Restoration at the Edge
Authors: Huanan LI
Juntao Guan
Lai Rui
Sijun Ma
Lin Gu
Zhangming Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image restoration
Deep Learning
Look-up Table
Super Resolution
Image Denoising
Primary area: machine_vision

Abstract: Look-up tables(LUTs)-based methods have recently shown enormous potential in image restoration tasks, which are capable of significantly accelerating the inference. However, the size of LUT exhibits exponential growth with the convolution kernel size, creating a storage bottleneck for its broader application on edge devices. Here, we address the storage explosion challenge to promote the capacity of mapping the complex CNN models by LUT. We introduce an innovative separable mapping strategy to achieve over $7\times$ storage reduction, transforming the storage from exponential dependence on kernel size to a linear relationship. Moreover, we design a dynamic discretization mechanism to decompose the activation and compress the quantization scale that further shrinks the LUT storage by $4.48\times$. As a result, the storage requirement of our proposed TinyLUT is around 4.1\% of MuLUT-SDY-X2 and amenable to on-chip cache, yielding competitive accuracy with over $5\times$ lower inference latency on Raspberry 4B than FSRCNN. Our proposed TinyLUT enables superior inference speed on edge devices with new state-of-the-art accuracy on both of image super-resolution and denoising, showcasing the potential of applying this method to various image restoration tasks at the edge. The codes are available at: https://github.com/Jonas-KD/TinyLUT.

TinyLUT: Tiny Look-Up Table for Efficient Image Restoration at the Edge
Huanan Li Juntao Guan 1.2.3 Rui Lail,2,, Sijun Ma Lin Gu Zhangming Zhu
Key Laboratory of Analog Integrated Circuits and Systems (Ministry of Education) School of Integrated Circuits, Xidian University, Xi'an, China
Hangzhou Institute of Technology. Xidian University, Hangzhou, China 'RIKEN AIP, Tokyo 1okyo103-0027, Japan
5 'The University of Tokyo, Japan
{huanamli. sinelliiiijnaaaaa jaaijuumaasstuu xidian. edu. cn {guanjuntad zhangmingzhuu/xiide rlai @mail. xidian edu. on, 1in, gueriken.
edu. cn
Abstract
Look-up tables(LUTs)) methods have recently shown enormous potential in image restoration tasks, which are capable of significantly accelerating the inference. However, the size of LUT exhibits exponential growth with the con- volution kernel size, creating a storage bottleneck for its broader application on edge devices. Here, we address the storage explosion challenge to promote the capacity mapping the complex CNN models by LUT. We introduce an innovative separable mapping strategy to achieve over 7x storage reduction, transforming the storage from exponential dependence on kernel size to a linear relationship. Moreover, design a dynamic discretization mechanism to decompose the activa- tion and compress the quantization scale that further shrinks the LUT storage by 4.48x x. As result, the storage requirement of our proposed TinyLUT is around 4.1% % of MuL.UT-SDDY-X2 and amenable to on-chip cache, yielding competitive accuracy with over 5x lower inference latency Raspberry 4B than FSRCNN. Our proposed TinyLUT enables superior inference speed on edge devices with tfte-offthe-aa accuracy both of image super-resolution and denoising, showcasing the potential of applying this method to various image restoration tasks at the edge. The codes are available
Introduction
With image sensors widely used in IoT applications, the demand for image restoration on edge devices has been stimulated. For decades, classic methods including interpolation [ 2]. sparse coding 4 and domain transformation [6] have been proposed. In recent years, deep learning based algorithms 7
have made encouraging progress restoration quality, while suffering from high computational load and long inference latency. These issues limit their extensive applications aisource-constraiie
IoT terminals. Thus can be seen, the growing demand for edge devices (e.g. smartphones and Raspberry Pi) calls for extremely lightweight solution for image restoration with high quality. Given this, researchers develop LUT-bbaeed methods 112 13 [4 to supplant the intensive convolutional computations with direct memory access operation for accelerating the inference, and exhibiting enormous potential.
In recent research, SRLUT [15] initially transfers the output values of a small trained deep network with receptive field (RF) size uniformly sampled LUT. To further improve the super resolution
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

31.4
31.2
TinyLUT-F
31.0
SPF-LUT-DFC
Figure 1: Performance comparison of our TinyLUT and other state-of-the art LUT--based algorithms. PSNR us. Vol- ume on Set5 benchmark dataset for x4 super resolution task. TinyLUT outper- forms the competitors accuracy with over 10x lower memory consumption.
30.8
PSNR (dB)
30.6
RCLUT
MULUT--DY-X2
30.4
SPLUT-L
MULUT-SDY
LinyLUT-S
SPLUT-M
30.0
SPLUT-S
SRLUT
29.6
Volume (KB)
Table 1: LUT size estimation when storing 8bit data with output entities r = 4. Compared with the full size, the sampled operation effectively reduces the storage consumption of LUT. Furthermore. separable mapping strategy(SMSS) and trained dynamic discretization mechanism (DDM) achieve an exaggerated reduction result.
RF
LUT
Full size
SRLUT
SMS
SMS DDM
pixels
4KB
272B
4KB
0.89KB
x 2 pixels
4D
64GB
1.27BB
16KB
3.57KB
x 3 pixels
90)
647BB
1767GB
36KB
8.0KKB
quality, Li et al. [14] increases the RF size through the different kernel shape combination with multiple LUTs. Meanwhile, Ma al. [13] constructs a series-parallel LUT framework based on multiple LUT's cascade improve the prediction accuracy. However, the existing LUT-bbaed schemes face the challenge of storage explosion when mapping Sbit image with larger than size. The storage explosion invalidates previous methods where the required storage of LUT grows exponentially as the number of indexing entries (i.e. input entities) increase. As reported Table 3 3 kemnel, commonly used in CNNs, will need tremendous storage up (28)9 8bit 16 64ZB and (2 + 1)9 x 8bitx 16 = 1767GB in full LUT and uniformly sampled LUT [5] with 8bit data respectively. Therefore, the issue of storage explosion presents severe challenge when mapping the complex models by LUT on edge devices.
kernel
To overcome the storage explosion challenge, we propose the separable mapping strategy(SMSS) and dynamic discretization mechanism (DDM) to decouple the kernel and activation, respectively. As reported in Table I the reduction of LUT storage reaches exponential levels using SMS and further achieves a 4.48x times reduction by DDM when the RF size is x 3 and storing 8bit data.
From the perspective of kernel decomposition, considering the exponential relationship between the dimension of input entries and LUT size, we propose the innovative SMS to decouple the convolution kernel by spatial location relationship. SMS decomposes the convolution weight of n input entities into the parallelization of n component independent kernel size respectively reduce the number of input entries in LUT-based inference, consequently reducing the dimension of rD LUT.. Then according to the additivity of convolutions with compatible kernel sizes [16), we reconstruct the feature space by averaging each output of parallel layers. Finally, each branch with 1 x kernel are individually mapped by 1-dimension LUT (ID LUT) with I imput entity for possible values. This approach reduces the size of nD LUT from s" x and allows vanilla convolution to be LUT-mapped on resource-constraing
operations,, which are instantiated
x 1
devices.
As for the activation compression, previous methods [5] 14
uniformly sample the input values reduce the possible values of and apply interpolation to recover the inference accuracy. We analyze the advantages of SPLUT [3] and propose dynamic discretization mechanism (DDM) to decompose the activation into Most Significant Bits (MSBs) and Least Significant Bits (LSBs). Moreover, inspired by the activation quantization methods 18, 19). DDM introduces the learnable clipping parameters to explicitly compress the quantization scale of MSBs and L.SBs activation, thereby reducing the possible pixels values and decreasing LUT scales.

---

As illustrated in Fig 1, we take the classic image restoration task single image super resolution (SISR) as an example. Our proposed TinyLUT-F consumes only 4.1% storage of MuLUT-SDY-X2[14 to achieve over 0.58dB PSNR increase in Set5 testset. Compared to other LUTT-based methods, TinyLUT significantly reduces storage overhead while improving aceuracy, expanding the application of LUUT-base approaches on edge devices.
The main contributions can be summarized as follows:
We reveal the scheme of storage explosion, being the key problem that limits the application of LUT in edge devices. Focus on this, we propose an innovative separable mapping strategy(SMSS realize dimensional reduction of LUT and solve the storage explosion problem.
We design dynamic discretization mcchanism(DDM) to decompose the activation and compress the corresponding quantization scale, further shrinking the storage requirement without considerable accuracy loss.
Our comprehensive experiments demonstrate that TinyLUT, constructed by SMS and DDM, achieves significant restoration accuracy over previous LUT methods with minimal storage consumption, showing its potential for application on edge devices.
2
Related Works
Deep Learning for Image Restoration: In recent years, image restoration performance has been rapidly developed through deep learning methods 20 g 21
22
10 7
II 23
24
26 and [2]
25
achieved outstanding accuracy. The authors of (24) presented a super resolution method using very deep networks and residual learning. Zhang et al. proposed deep convolutional neural network with residual learning to handle image restoration tasks such as denoising and SISR. FFDNet [IO] achieves more efficient and flexible than DnCNN 7 by down-sampled method. Also have the ability to robustly control the tradc-off between noise reduction and detail preservation. As a fast and memory-effiiient network, FMEN frequency attention block. Dong et al. 8 explored a more efficient network by re-designing the SRCNN structure. In (27 author proposed a deep convolutional network within a Laplacian pyramid framework for fast and accurate image super-resolution. Denoising results of FastDVDnet [28] feature remarkable temporal coherence, very low flickering, and excellent detail preservation. However, these CNN methods suffer form intensive computations of convolutional operator, which limits their deployment on edge devices.
23 is constructed by enhanced residual block and high-
Lookup Table for Image Restoration: LUT-based methods are commonly used in hardware systems 29 and some embedded devices (30 to accelerate CNNs by pre-storing the results of complex functions with high efficient retrieval operations. In image restoration tasks, LUT's commonly used video coding [I 32), tone-mapping (33] and image enhancement [17] Furthermore, deep learning methods based on LUTs have also emerged in SISR 14 [3] and gradually garnered attention. SRLUT(15) establishes a spatial mapping from a local RF block to the corresponding high- resolution pixel. In MuLUT and [PLUT(3], the authors proposed the collaborating structure by multiple LUTSs to increase the model's RF and improve the inference accuracy. Furthermore, Liu et al. (34] found that the interaction of space and channel features in vanilla convolution allows previous methods to increase the RF at the cost of linearly increasing the LUT size. They proposed RCLUT based on the Reconstructed Convolution(RC) module to enlarge the RF, achieving significant performance with less storage. Realizing the dilemma between the performance improvement and storage of LUT-based models, Li al. (35) introduce the Diagonal-Fire Compression(DFC) framework to achieve a better trade-off between storage size and performance for image restoration. Additionally, they also designed a new structure, SPF-LUT, to further improve the performance of LUTTbbased models. However, the RF and channels of LUT-based methods are limited to adapt the trade-off between storage and accuracy. In comparison, our work is focused on developing a lightweight LUT-base framework to improve the image restoration accuracy.
3

---

MS0s
Publockk
PooBock
Paitlock
Doc
Pixelahuull
Pec
Pax Cony
Pw Come
PaCon
Impit
3 Come
PwBBlokk
DicComu
Pou
PwBllnck
Osapui
Single
Relo
Reb
LSBs
(a) Network Architecture of TinyLUT
(b) PwBlock
Depthwise Separable
MSBs
ScLUT
PuLUT
DwLUT
PwI.UT
PwLUT
PixelshuMll
Inpur
SeLUT
Pwl LT
DwLUT
PWLUT
PwLUT
Ouipui
TinyLUT
Figure 2: (a) network overview of TUnyLU.. (b)The structure of PwBlock. (c) LUT framework overview TinyLUT.
3 Method
3.1
Overview of TinyLUT
The structure of TunyLUT and foundational components are depicted in Fig 2 parallel branches with cascaded LUT's to process Most Significant Bits (MSBs) and Least Significant Bits (LSBs) data respectively, and introduce separable mapping strategy and dynamic discretization mechanism.
We design two
According to the previous LUT-based methods we follow two simple design rules:(i) to improve model accuracy, the RF size needs to be increased; and (ii) the number and storage requirement of LUT's need to be reduced adapt edge devices. Therefore, our CNN network (Fig ((a) mainly inspired by the architecture of SRLUT [15] and DnCNN (7). Specifically, the CNN model is built on standard convolution, depthwise convolution (DwConv) and PwBlock. The PwBlock 2(b) is a channel combining module based SMS and described in Section 3.2 Based on SMS and DDM, the standard convolution, DwConv and PwBlock are mapped by LUT respectively to build standard convolution mapped LUT (ScLUT)) depthwise LUT (DwLUT) and pointwise LUT (PwLUT), In particular, we refer the philosophy of depthwise separable convolutions (36 to combine DwLUT and PwLUT to depthwise separable LUT(DSLUT). The mapping details for DwLUT and PwLUT described in Section 3.2 and Section 3.
As in depthwise separable convolutions [36. the DwLUT optimizes the LUT size while maintaining a large effective RF. The PwLUT is used for cross-channel feature information integration and model channel number transformation. We introduce skip connections to fuse the real-value inputs and the retrieval outputs between DwLUT and PwLUT to avoid the degradation. The RF and output channels set in SRLUT [5], the input channel of ScLUT set to
3 and 16. Due each color channel having to be processed independently as
According to the different number of DSLUT serial stacks, we build TinyLUT-F with stacked. TunyLUT-S is built by ScLUT and double PwLUT's. As shown Fig 2 TinyLUT constructed by DwLUT and PwL.T in each branch. An additional PwI.UT is added at the end of each branch to adjust the number of output channels of different image restoration task. In super resolution, the output channel number of the last PwLUT is r to adapt to the pixelshuffle (21). where is the upscaling factor. In order to further expand the RF size, we introduce rotational ensemble trick (15) in the inference.
3.2 Separable Mapping Strategy
In T we provide an estimation of the L.TT size of four output channels and single input. is evident that the SRLUT continues face the issue of storage explosion, resulting in approximately

---

Mapping
Separable Mapping Strategy(SMS)
(DDM)
MSB+
LSBs
10:001
ID LUTT
ID LXIT
LDT
IDT
Inlox
valez
rodes
voire
Isdex
Vilue
valle
[Pa]
(
[P4]
[P
[Py]
[0g]
(Pa)
(Fa)
[4r]
256
Figure 3: Left: Existing LUT-baaed method maps the input convolution to a 4-dimensional LUT (4D LUT). SMS decomposes the convolution kernel to reduce the storage from 256 x 8bit to 256 x 4 8hit. Right: The dynamic discretization mechanism (DDM) decouples the activation and compress the precision to reduce the storage to (am x 64 + og 4) x 8bit using Omm 0z E [0, 1].
1767GB of memory overhead when processing 3 x 3 pixels of input simultaneously. In contrast, the proposed separable mapping strategy (SMS) alleviates the memory bottleneck associated with large RF and reduces the required volume to just 36KB. Therefore, we enable the transfer of more deep convolutional network model into efficient LUT framework with fewer hardware cost.
The left image in Fig illustrates 4-dimensional LUT(4D LUT) mapping the convolution 2 x kemel with single output in existing methods [15 14 [3]
requires 256 8bit= 4GB and exceeds the storage capacity of edge devices. Therefore, we propose separable mapping strategy(SMS) to decouple 2 kernel into four 1 x 1 kernels which reduces the size of the LUT required to 256 x x Sbit= 1KB. Therefore, we propose the DwLUT. Firstly, from the original depthwise convolution operation, it can be obtained by:
Fout
4J
Wh,
i= 1=
The Fij represents the input feature data, and Font represents the output feature and waj stands for the convolution kernel, while denotes the size. After transferring by original multiple dimensions LUTs,, the retrieval process can be represented as:
Fout [UT f
In DwLUT, we utilize multiple x convolution kernels to build depthwise convolutions. As shown in Fig 3 we decompose the convolution kernel into n² of size Cout x x 1 1. Mapping by ID LUTs, the original LUT retrieval process can be reconstructed as:
Fowt - n=In-1 ΣΣ
After decomposing and mapping, the storage consumption reduces from 256" x Cout X 8bit to 256 x n x n x Cout x 8bit. Meanwhile, the output of the multiple 1D LUTs is averaged achieve an approximation of the standard LUT inference.
The pointwise convolution applies tion (36. The operation of pointwise convolution with a kernel size of Cout x Cin x x is defined
convolution combine the outputs the depthwise convolu-
Cin Cont Fout >>
To
We,t
c=0 1=
In this context, æ represents the input data and w represents weights. still faces the problem of exponential size increase in the LUT mapping process due to multiple channels input simultaneously. Then we decouple the input channels in pointwise convolution. Subsequently, is decoupled into Cin, number of Cout x x 1 x convolution kernel. The storage space required for the pointwise
5

---

convolution, is reduced from (256 x Cout X 8bit to 256 x Cin x Cout X 8bit. The final output channel set to Cou.. At this point, the original cross-channel per-pixel LUT retrieval operation is reconstructed as:
Civ
Pout
> LUTa(ae)] C
1
As depicted in Eq 5 the quantity of LUTs is denoted as Cin. Additionally, the number of output channels for each LUT retrieval is represented as Cout. According to and Eq the PwBlock in Fig [ consists of several parallel branches with the number of channels. Each branch includes 3 pointwise convolutions followed by ReLU except for the last layer. The channels in the pointwise convolution is set to 64 and that of the last layer is set to 16.
3.3 Dynamic Discretization Mechanism
In previous methods, in order to further reduce the volume requirement LUT, common to sample the indexing entries. Related algorithms [15] 14 37 use interpolation methods to restore accuracy after table retrieval. However interpolating in large RF and multiple channels will inevitably result in significant amount of computation. To address this issue, we proposes the dynamic dis- cretization mechanism (DDM). adaptively explores the parameterized clipping level of quantization range for each channel based the gradients with Straight-Through Estimator (STE) 38 39,, which significantly optimize the equilibrium of accuracy and LUT size.
As shown in Fig an 8bit input data is decomposed into MSBs and LSBs, then obtain output results through their respective branches and add them. should be noted that the 8hit data stored in the LUT is defined in the domain of - -128 128 to 127, while the data range of MSBs data is [-m, m 1]. and the data range of LSBs data [0, k.). Therefore, in Eq 3 mapping depthwise convolution to LUTs
for the MSBs branch, the LUTs size can be obtained by:
Size(MSB) = (2 x m) x n x
(6)
and the size of LUT in LSBs branch:
Size(LSB) = (k + 1) x n² x 22
This operation reduces the index range the LUT, realizes the discretization of the LUT and reduces the size.
The index range 2 due the activation data range are equal LUTs indexes range in the mapping process. Therefore, inspired by activation quantization [8 19 we use the learnable clipping parameter a to constrain the bit width features. Specifically, any and City are used to constrain the MSBs and LSBs of the layer y, respectively, to control the size of the LUT. For example, using F to denote feature data, this operation can be calculated as:
Fg round( F *ay)
(8)
The LUT size is written as
Size (max((FF) min(F,)) x n² 2
In Eq 9 n denotes the input entries and ,² denotes the output entries respectively. In order to reduce the index number of LUTs, we initialize to 0.8 and then apply 12-regularization for a with the 0.1x regularization parameter À used to compress the activation precision scale in each layer.
4
Experiments
4.1
Implementation Details
Training Settings To demonstrate the effectiveness of our framework, we evaluate TinyLUT on multiple datasets for SISR. The CNN model of TunyLUT is trained in an end-to-end manner. We use DIV2K (40) dataset which has been widely applied image processing tasks. TinyLUT model is trained for 200000 iterations with Pytorch 41] on Nvidia 3090 GPU. We employed the Adam optimizer 422, where 31 = 0.9. 82 = 0.999. The learning rate set at 5 10-3 and was dynamically adjusted using cosine annealing mechanism 43 We randomly cropped degraded data into 48 48

---

Table 2: Quantitative comparisons 5 standard SISR test for upscaling factor of 4. The size of TinyLUTss are smaller than other LUT methods and achieves better PSNR and SSIM average values with good margin. The storage overhead of weight parameters in the DNN. The inference latency evaluation environments are the same as [15 14
Method
Strage
Runtime
Set5
Set
Crban100
BSD100
Manga 109
Averaee
Xiaouá
Raspeery
SRLUTES
05
1504KB
157m
34 Tame
20 2082208447
27,0100
34.03016900
26. 53.3388553
26 86.00083300
26 P631
SPLUTLL
18432KB5
265ms
456mm
30 50.500
540 7520
24.660.7191
26 26.8700.00 2090
27, 20.88088581
27 27,4201 7802
MuLLUE-SDY-X
4159KB
176600755 7541
34.460.7194
26 86.660.1110
27-900.6633
27.480
RCLUTT 10
1519KB
250m
30 7200 8677
27 6700 7577
24,52/0.2259
26 26.9510.7154 S4
28 05/0 8665
27 59K1 796
2056KB
31055088755
24.81907387
27.08/0.7190
25.5640.7799
2708
TIT-S LIT-S
37KB
29
8Rmms
90 2222855555
27.3300 7450
26.71/0,7042 7066
27.2 20.558
27.1 N300700
TnnLUUT-
171KB
146ms
38 Tms
31.18988771
28,0100 7630
24.92/07.891
27.130.7184
28.830.8798
28.0100 7956
SRCNK 1H
2258B"
27448ms
30.48/086688
24.520.7221 27.50977513
26.9010.7101
27. 57.55775447
27.3000
VDSR
24
2660KB
106972am
81.3500.8531
28.02/076810
25. 25.18/07550
27 27.2901.7200
28. 28.500.8812
28.0700.002 8024
DNN
FSRCNN
350ms
30.71008656
27.609.7543
24.61007263
26.9610.7129
90.90056610
27.56017840
CARN-M M 26
1648KBB
3500ms
17609kms
31.82008998
28 28.240.7147
25.620.7664
27,42A1,7350
29,8500,8993
26 6100
Bicubiic
SRLUT L61
McLUT
SPLUT
13
ESRCMN
TIyLUT
Powaraak Paint
2002
Power 2002
Power
Power
Power
Power 2002
Power 2002
Power 2002
2002
2002
2002
Figure 4: Qualitative comparisons of bicubic interpolation, SRLUT [5]. MuLUT
14 SPLUT
13
FSRCNN (8), our TinyLUT and HR images.
patches with a batch size of 32. Data augmentation was performed through random rotations and flips. Throughout the experimental process, we employed PSNR and SSIM 44] evaluation metrics for restoration accuracy. Besides, we measured and reported the runtime on multiple edge devices including Xiaomi 11 smartphones with a Qualcomm Snapdragon 888 CPU and Raspberry 4B.
Evaluation Settings For single-image super resolution, assessed the effectiveness of our method on
five widely used benchmark datasets: Set5, Set14, BSD100 [5). Urban 100 [46), and Manga 109 (47).
We compare our method with various SISR algorithms based on interpolation which include methods
based on deep learning including FSRCNN 8.. VDSR (24, RCAN (25) and CARN-M (26), and SR
methods based LUT's, SRLUT [15. MuLUT (14, RCLUT In tables, the best result is highlighted in bold.
34 SPF-LUT
35 and SPLUT 13
4.2 Evaluation on Image Super Resolution
Accuracy and Storage The quantitative results of image super resolution shown in Table For single image super resolution task, the SSIM and PSNR values are computed at the Y-channel in the YCbCr color space. As reported in Table 2 we build series of TinyLUT models with volume from 37KB to 171KB to fit various edge devices. Our LUT-L model achieves 0.58dB PSNR increase with nearly 24x storage reduction of MuLUT-SDY-X* X2 14. Even for the compact LUT-L model, it only consumes 2.83% memory overhead and achieves 0.4dB PSNR and 0.02 SSIM improvement compared SRLUT-S [5]. The qualitative results of MULUT-SDY-X0 14 and SPLUT-
L [3] have severe ringing artifacts in the white area. On the contrary. our method generates more natural textures and fewer artifacts. As shown in Table TinyLUT-F achieves much higher PSNR with 9 and 12x lower storage consumption than RCLUT K4
and SPF-LUT+DFC (35). respectively. While TinyL.U achieves restoring clearer edges and 5x lower latency on Raspberry 4B compared with computation (ethho((SSCNNN8))
7

---

Table 3: Quantitative comparisons Set12 and BSD68. The size of TinyLUTs smaller than other LUT methods and achieves better PSNR and SSIM values with a good margin. *: The storage overhead of weight parameters in DNN. The evaluation environments are the same as 48
Runtime
Sct12
BSD68
Method
Storage
Average
Xisomi II
Raspberry 4B
15
25
50
15
25
50
SRLUT 1S
82KB
7ms
21ms
30.42
27.19
22.62
29.78
26.85
22.39
26.54
31.50
25.46
30.63
MULUT--DYEHO-X2 18
978KB
51ms
89ms
31.77
29.18
25:47
30.89
28.34
24.96
28:44
TiayLUT-S
22KB
20ms
27ms
31.10
28.26
24.29
30.24
27.48
23.83
27.53
TinyLUT-P
187KB
146ms
254ms
32.22
29.69
26.27
31,20
28.65
25.53
28.93
DnCNN
2
635ms
6859ms
32.86
30.44
27.18
31.73
29.23
26.23
29.61
GT
Noisy
DrCNN
TinyLUT
Figure 5: Qualitative comparisons of ground truth, noise image, DrCNN and our TinyLUT.
Running Time Inference speed is also an important impact for edge devices deployment. In the view T we report the runtime of LUT-base methods and other schemes. Results are obtained by averaging across 20 runs. Our approach achieves 4x and 10x acceleration on SISR compared to the fastest LUT-based and CNN method respectively. Meanwhile TinyLUT obtains absolute accuracy advantage compared LUT-based methods. Overall the experiments indicate that the SMS achieving sigmificant storage reduction while organizing the model in a more flexible manner, thus capable of improving accuracy. TinyLUT achieves accuracy close to CNN and exhibits the enormous application potential of LUTT-based method.
4.3 Evaluation on Image Denoising
In order to further show the capacity of the proposed TinyLUT framework, we remove the pixelshuffle and set the out channels of the last PwLUT to 1. Mcanwhile, we adopt the residual learning formulation such as DrCNN 7 to accelcrate model training. For additive white Gaussian noise (AWGN), use S and BSD68 benchmark datasets with noise level 15, 25 and 50. We compare our method with classical DNN method 7 and LUT methods [15 48).
The quantitative results of image denoising is shown in Table 3 Compare with LUT-bbased methods, our method significantly improves PSNR with the minimal storage in image denoising. Meanwhile, TinyLUT-S achieves the fastest inference speed with only 22KB storage overhead compare with DNN based denoising schemes. The denoising results Fig Fig illlustrate that our TinyLUT is able to obtain similar visual quality to DnCNN (7]. The quantitative and qualitative experiments prove the effectiveness of storage reduction and inference acceleration of our approach.
4.4 Evaluation on Image Deblocking
In this subsection, image deblocking used to further assess the generality of TinyLUT in image restoration tasks. Table reports the comparison results of PSNR-B for LUTT-based methods on Classie5 49) and LIVEI 50 We also refer to the classical methods and DNN algorithms, including
SA-DCTI51 and ARCNN 52 TinyL.UT-F achieves comparable PSNR-B B results to the DNN method

---

200
30.6
180
PSRES
Volume
30.3
80-
TRyWTS ThyLUTSS M812
ThyLTS-
30.0
70-
66
50
1.00
29.1
80
80
23.8
Inpat
60
28.5
28.2
27.9
10-
1/7 226
3/5
416
5/3
672
7/1
8/0
14 15
M58AL555
Figure 6: Illustration of the storage overhead and PSNR result for different combinations of MSBs and LSBS in TinyL.UT-S for single image super resolution task.
Figure Illustration of the input entities precision range clipping by parameter aMsB when mapping process TinyLUT-SSa in SISR. TinyLUT-S-E indicates the input entities range of PwLUT statistical result.
and better than other LUT--based methods. The results indicate the generality of our TinyLUT to deblocking.
Table 4: The quantitative comparisons of PSNR-B B standard benchmark datasets for image deblocking under quality factor 10.
Classical
DNN ARCNN
LUT-based
JPEG SA-DCT(ST
SRLUT 15
[4LUT[[4]
SPF-LUT(55
SPF-LUT+DFC 35
TuyLUT-F
Storage Classic5
415KB
81KB
489KB
3017KB
595KB
187KB
28.76
27.58
28.29
28.63
28.74
LIVE
25.33
28.01
28.77
27.69
28.39
28.62
28.61
28.67
4.5
Ablation Studies
We perform several ablation experiments single image super resolution demonstrate that our contributions in TinyLUT provides quality improvements.
The effectiveness SMS We conduct an experiment with combinations of SMS and TinyLUT- S. In Table 5] SMS denotes the TinyLUT-S includes SMS without DDM when constructed the model with 8bit data values. Original denotes the TinyLUT-S without SMS and evaluates using the quantized CNN model due to the excessive storage overhead of its LUT model. The accuracy of SMS competitive with corresponding mapped neural network with 8bit data SISR. In particular, SMS scheme achieves significant accuracy improvement compared to uniformly sampled method [I5] with sampling interval size of 24, while yielding over storage reduction. This demonstrates the effectiveness of storage reduction using SMS with minimal loss of accuracy.
Table 5: Impact of SMS. Ablation studies on TinyLUTSS for 4x SISR.
Model
Method
Set5
Set14
Urhan 100
BSD100
Manga109
LUT Size
Original
30.35
27.42
26.77
24.31
27.41
9.7 x 1034pB
TinyLUT-S
Uniformly Sampled 15
29.82
27.01
26.53
24.02
26.80
1,274MB
SMS
30.24
27.33
26.72
24.19
27.23
164KB
The effectiveness of DDM As shown in Fig 6 the PSNR results of super resolution and image denoising approximately equal the global maximum at 6 MSBs and LSBs (6M2L). Meanwhile the storage overhead much smaller than the combination of 5 MSBs and LSBs. Therefore we set the activation precision for the two branches of TinyLUT to 6 MSBs and L.SBs respectively.
As shown in Table compared with the full LUT with 2 entries for each input pixel, DDM reduces TunyLUT storage overhead while ensuring accuracy in super resolution. The results prove that the

---

Table 6: Effects of DDM. Ablation studies for DDM with 4x SISR.
Model
Method
Sct5
Sct14
Urban100
BSD100
Manga109
LUT Size
Full LUT
30.24
27.33
26.72
24.19
27.23
164KB
TinyLUT-S
DDM
30.22
27.33
26.71
24.19
27.21
37KB
LUT sampling method based on DDM has higher compression ratio of storage than the 8bit full sampling LUT, and achieves better trade-off between storage and accuracy.
As shown in Fig 7 the precision range of the MSBs activation data is adjusted more adaptively by clipping parameter or in SISR, which reduces the possible value range of the input entities of PwLUT in TinyLUT-S and then reduces the size in MSBs branch. As a comparison, the marked red line in the figure the value when the input entities range is fixed to 6bit during mapping in the MSBs of PwLUT. It can be observed from the figure that compressed by the DDM, not only the LUT size is reduced, but also all possible values of input entities are covered in the mapping process. Compared with the 43.6KB of TinyLUUT-S-6M of TUnTSS is reduced 36.6KB via the adjustment of DDM, achieving a reduction of 15%. Therefore, DDM yields additional storage reduction compared to decomposing data into fixed MSBs and LSBs
with fixed combination of 6 MSBs and 2 LSBs, the size
More intuitively, the interpretability DDM is also illustrated Fig
5 Limitations and Future Direction
The method in this paper achieves significant reduction in storage with high compatibility when mapping convolutional neural network. However, there are difficulties in mapping other models such Mamba (53) and Transformer [54] (53 using TinyLUT. This can be further explored the unified mapping approach for other model. Exploring additional image restoration tasks on edge devices using LUTs would also contribute the community.
6 Conclusion
In this paper, we analyze previous successful LUT-based deep learning approaches and summarize the key problem of storage explosion, which limits the further popularization LUT image restoration on edge devices. To address the storage explosion, we propose the separable mapping strategy (SMS) and dynamic discretization mcchanism(DDM) to decompose the kernel and activation, respectively. In particular, design the TinyLUT framework based SMS and DDM. By seamlessly integrating these innovations, TUn-L sets a new record for SISR by achieving over 31dB PSNR on the Set5 dataset at just 171KB LUT storage. Overall, extensive experiments across seven benchmark datasets and two classic image restoration tasks demonstrate the effectiveness and efficiency of TinyLUT on resource-constrained devices.
7
Acknowledgement
This work was supported in part by the National Science and Technology Innovation 2030-Major Projects under Grant 2021ZD01 14400, in part by Young Scientists Fund of the National Natural Science Foundation of China 62304162 1620 and in part by China Postdoctoral Science Foundation under Grant 2024M762532, and part by Postdoctoral Fellowship Program of CPSF under Grant GZCC20241313 and in part by Shaanxi Provincial Natural Science Foundation for Basic Research Program 2024JC-YBMS- and in part by Fundamental Research Funds for the Central Univer- sities under Grant XJSJ24090, Dr. Lin Gu was supported by JST Moonshot R&D Grant Number JPMJMS201 Japan.
References
[1] R. Keys. Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 155-)115 1981.
10

---

[2] R. G. Keys. Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 29, 2003.
[3] Radu Timofte, Vincent Desmet, and Luc Vangool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Springer International Publishing, 2014.
[4] Radu Timofte, Vincent De, and Lue Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In 2013 IEEE International Conference on Computer Vision, pages 1920 1927, 2013.
[5] Hong Chang, Dit Yan Yeung, and Yimin Xiong. Super-resolution through neighbor embedding. In IEEE Computer Society Conference on Computer Vision Puttern Recognition, 2004.
[6] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080- - 2095, 2007.
[7] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing, 26(7):3142-3155 2017.
[8] Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolu- tional neural network. Springer, Cham, 2016.
[9] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. arXiv e-prints, page aiv: 1ttl:::1000000200 September 2018.
[10] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for cnn- based image denoising. IEEE Transactions on Image Processing, 270))4408-4422,, September 2018.
[11] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) Workshops, pages 1833 1844, October 2021.
[12] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image December 2014. Deep Convolutional Networks. arXiv e-prints, page arXiv:1501.0009)
Using
[13] Cheng Ma, Jingyi Zhang, Jie Zhou, and Jiwen Lu. Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution. arXiv e-prints, page arXiv:2207. 12987, July 2022.
[14] Jiacheng Li, Chang Chen, Zhen Cheng, and Zhiwei Xiong. Mulut: Cooperating multiple look-up tables for efficient image super-resolution. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, pages 238-256, Cham, 2022. Springer Nature Switzerland.
[15] Younghyun Jo and Scon Joo Kim. Practical single-image super-resolution using look-up table. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 691-700, 2021.
[16] Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks, 2019.
[17] Hui Zeng, Jianrui Cai, Lida Li. Zisheng Cao, and Lei Zhang. Learning Image-adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time. arXiv e-prints, page arXiv:2009.10 14468, September 2020.
[18] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, I Jen Chuang, and Kailash Gopalakrish- nan. Pact: Parameterized clipping activation for quantized neural networks. 2018.
[19] Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Yuchao Li, Baochang Zhang, Fan Yang, and Rongrong Ji. Pams: Quantized super-resolution via parameterized max scale. 2020.

---

[20] Jiayi Qin, Lihui Chen, Seunggil Jeon, and Xiaomin Yang. Progressive interaction -larriiinneeiinng net- work for lightweight single-image super-resolution in industrial applications. IEEE Transactions on Industrial Informatics, 19(2):2183-219) 2023.
[21] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[22] Zheng Hui, Xiumei Wang, and Xinbo Gao. Fast and accurate single image super-resolution via information distillation network. In 2018 IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 723-731, 2018.
[23] Zongeai Du, Ding Liu, Jie Liu, Jie Tang, Gangshan Wu, and Lean Fu. Fast and memory-efficien network towards efficient image super-resolution. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 852-861, 2022.
[24] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate Image Super- Resolutinn Using Very Deep Convolutional Networks. arXiv e-prints, page 15 1.04577, November 2015.
[25] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, 2018.
[26] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, Accurate, and Lightweight Super- Resolution with Cascading Residual Network. arXiv e-prints, page 1XXiv:8003006660 March 2018.
[27] Wei-Sheng Lai, Jia-Bin Huang. Narendra Ahuja, and Ming-Hsuan Yang. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 41((1):2599-26 2613, 2019.
[28] Matias Tassano, Julie Delon, and Thomas Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[29] SHARIF, MD.. and HAIDAR. High-performance mathematical functions for single-core architectures. Journal of Circuits, Systems Computers, 2014.
[30] Hakki Can Karaimer and Michael S. Brown. A software platform for manipulating the camera imaging pipeline. In European Conference on Computer Vision, 2016.
[31] Jun Young Lee, Jae Jin Lee, and Seong Mo Park. New lookup tables and searching algorithms for fast h.264/ave cavle decoding. IEEE Trunsactions on Circuits Systems for Video Technology, 2007):100711077 2010.
[32] Sik Ho Tsang, Yui Lam Chan, and Wan Chi Siu. Region-based weighted prediction for coding video with local brightness variations. IEEE Transactions on Circuits and Systems for Video Technology, 23(3):549-550 2013.
[33] Ishtiaq Rasool Khan, Susanto Rahardja, Muhammad Murtaza Khan, Muhammad Mobeen Movania, and Fidaa Abed. A tone-mapping technique based on histogram using a sensitivity model of the human visual system. IEEE Transactions Industrial Electronics, 65(4):3469 3479, 2018
[34] Guandu Liu, Yukang Ding, Mading Li, Ming Sun, Xing Wen, and Bin Wang. Reconstructed con- volution module based look-up tables for efficient image super-resolution. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12183-12192, 2023.
[35] Yinglong Li, Jiacheng Li, and Zhiwei Xiong. Look-up table compression for efficient image restoration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26016-26022 June 2024.
[36] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam, MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv e-prints, page EXXi:1104.04880 April 2017.
12

---

[37] Canqian Yang, Meiguang Jin, Yi Xu, Rui Zhang, Ying Chen, and Huaida Liu. SepLUT: Separable Image-adaptive Lookup Tables for Real-time Image Enhancement. arXiv e-prints, page arXiv:2207.08351 July 2022.
[38] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. 2013.
[39] Nitsh Srivastava Geoffrey Hinton and Kevin Swersky. Neural networks for machine learning. Coursera, video lectures, 2012.
[40] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017.
[41] Anmol Chaudhary, Kuldeep Singh Chouhan, Jyoti Gajrani, and Bhavna Sharma. Deep Learning With PyTorch. Machine Learning and Deep Learning in Real-Time Applications, 2020.
[42] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Computer Science, 2014.
[43] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. 2016.
[44] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13(4):600- 612, April 2004.
[45] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference Computer Vision, 2002.
[46] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self--xemplars. In 2015 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5197-5206, 2015.
[47] Yusuke Matsui, Kota Ito, Yuji Aramaki, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch- based Manga Retrieval using Mangal09 Dataset. arXiv e-prints, page arXiv:1510.0433 October 2015.
[48] Jiacheng Li, Chang Wen Chen, Zhen Cheng, and Zhiwei Xiong. Toward dnn of luts: Learning efficient image restoration with multiple look-up tables. IEEE transactions pattern analysis and machine intellignnee, PP, 2023.
[49] Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Pointwise shape-adaptive det for high-quality denoising and deblocking of grayscale and color images. IEEE Transactions on Image Processing, 16(5):1395-141 2007.
[50] H
Sheikh.
Live image quality
assessment
database
release
2.
[51] Alessandro Foi, Vladimir Katkovník, and Karen Egiazarian. Pointwise shape-adaptive det for high-quality denoising and deblocking of grayscale and color images. IEEE Transactions Image Processing, 1665):1395-141, 2007.
[52] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by deep convolutional network. In 2015 IEEE International Conference Computer Vision (ICCV), pages 576-584, 2015.
[53] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
13

---

[55] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010. 1929, 2020.
14

---

8 NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions have been clearly summarized in the abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Deduction and experiments in Section ?? and Section demonstrate the validity of theoretical assumptions.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clarify the main setups of our method Section and settings of experiments in Section 4 for the reproducibility in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Upon publication, we intend to release the code for broader community access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubbliccguu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify the main training and test details Section 4 to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The benchmark result in Section Iis already obtained by multiple testsets and noise level, and other previous works didn't reported the full experiment with the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The main hardware devices used this paper are listed in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://awrrr...ccobbbb..ccccctaa
Answer: [Yes]
Justification: The research conduct in this paper conform, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Upon publication, we intend to supplement the discussion of the broader implications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The data and methods used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Upon publication, we intend to release the well documented code and data for broader community access.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "dqK3SCDy9j", "reviewer": "Reviewer_JnS6", "summary": "This paper proposes to reduce the size of LUT for image restoration and to make it applicable on edge devices. The main idea is using depthwise separable convolution to replace the vanilla convolution. When transfer to LUT, the storage is significantly reduced. Experiments show the effectiveness and efficiency of the proposed method.", "questions": "See the weaknesses above.\n\nPlease follow the code of ethics in [1], Lena image in Fig 2 may violate this.\n\n[1] https://conferences.ieeeauthorcenter.ieee.org/write-your-paper/improve-your-graphics/", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The approach is reasonable and the experiment result is provable.\nThe evaluation is complete. The efficiency of the proposed method on edge device is proved.", "weaknesses": "The baseline DnCNN is somehow out-of-state. As described in Limitation, the proposed method based on depthwise separable convolution is suitable for CNN based architectures only, and the LUT-based compression on attention-based method is to be explored.\n\nQuantitative comparisons on color image restoration tasks are needed, e.g., CBSD68. It is better to conduct experiments on more testsets. \n\nIn runtime comparison, it is necessary to compare the complexity of the comparison algorithms. I am interested in the processing time of the proposed method on high-resolution images.\n\nThe description of the method is hard to understand. The writing should be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness 1**  We appreciate the reviewer's meticulous work. Indeed, our proposed TinyLUT as well as existing LUT-based methods are mainly designed for CNN-based architectures that are widely deployed on resource-constrained edge devices. Primarily because the convolution operation is computationally efficient [1][2][3]. CNNs compete favorably with Transformers in terms of accuracy and scalability, and convolution remains much desired and has never faded[4][5]. In recent years, many CNN methods have still been proposed [4][5][6]. \n\nMeanwhile, as highlighted by the reviewer and acknowledged in our Limitations section, LUT-based compression for attention-based methods represents a critical direction for future research. This area will be a focal point in our subsequent investigations.\n\n**Response to Weakness 2** We thank the reviewer for the constructive suggestion. To address the reviewer's concern, we have conducted supplementary experiments for denoising task under 50 noise level on more color image test sets, inculding Kodak, CBSD68 and McMaster. The experiment results further indicate the effectiveness of our proposed TinyLUT, corroborating the assertions made by reviewers MFFN and G5Ti. The experimental results are shown as follows：\n\n**Table 1 Quantitative comparisons on color image denoising with noise level 50**\n|                 | Storage | Kodak | CBSD68 | McMaster | Average |\n|-----------------|---------|-------|--------|----------|---------|\n| SRLUT           | 82KB    | 22.64 | 22.45  | 23.41    | 22.83   |\n| MULUT-SDY-X2    | 489KB   | 25.64 | 24.95  | 26.38    | 25.66   |\n| MULUT-SDYEHO-X2 | 979KB   | 25.80 | 25.07  | 26.57    | 25.81   |\n| TinyLUT-F       | 189KB   | 26.43 | 25.53  | 27.14    | 26.36   |\n\nAs illustrated in Table 1, our TinyLUT-F achieved an average accuracy improvement of over 0.5dB as well as about 5x storage reduction compared to MULUT-SDYEHO-X2. Like the performance validation on color image data in super-resolution task, the quantitative comparisons on color image denoising are still needed for comprehensive assessment. Meanwhile, this additional investigation expands the scope of our work. \n\n**Response to Weakness 3** We fully agree with the reviewer's insightful comment. Following the complexity evaluation in Ref.[7], computational costs of addition and multiplication operations are employed as the metric to assess the model complexity. Following the existing literature, we conduct the complexity comparison on 4x super-resolution task with an output resolution of 1280x720. The results are presented in Table 2. As can be seen, our proposed TinyLUT-S achieves the lowest computational complexity compared to other competitors.  \n\n**Table 2 The statistics of addtion and multiplication operations**\n|              | Add    | Mul   |\n|--------------|--------|-------|\n| SRLUT-S      | 44.3M  | 19.4M |\n| MULUT-SDY    | 165M | 69.2M |\n| MULUT-SDY-X2 | 145.5| 73.6M |\n| TinyLUT-S    | 10.28M | 5.3M  |\n| TinyLUT-F    | 46.28M | 26.3M |\n\nTo address the reviewer's concern about latency, we conducted 4x super-resolution tests on a Raspberry Pi 4B platform. At present, the mainstream high-resolution images mainly include 1920×1080 (FHD) and 3840×2160 (UHD). Given this, we illustrated the processing time of our TinyLUT and competitive methods in Table 3. As can be seen, TinyLUT-S yields the lowest inference latency, while the latency of TinyLUT-F is comparable to SRLUT-S and far superior to MULUT on high-resolution images. This further validates the high efficiency of TinyLUT on edge devices mentioned by reviewers G5Ti and MFFN.\n\n**Table 3 Latency of high-resolution image on Raspberry Pi 4B**\n|              | 1920x1080(FHD) | 3840x2160(UHD) |\n|--------------|----------------|----------------|\n| SRLUT-S      | 554ms          | 2224ms         |\n| MULUT-SDY    | 853ms          | 3566ms         |\n| MULUT-SDY-X2 | 906ms          | 3834ms         |\n| TinyLUT-S    | 163ms          | 777ms          |\n| TinyLUT-F    | 614ms          | 2797ms         |\n\n**Response to Weakness 4** Thanks for the reviewer's suggestion. Although our article received positive feedback (2kHn) in terms of writing style and chapter division, there are still some grammar issues. We have reviewed the paper again and corrected some errors to improve the quality of the article.\n\n**Response to Question 1**  Thanks for the reviewer's important reminder. We comply with the reviewer's suggestion and replace the Lena image in Fig. 2 with the Baby image in the Set5 test dataset.\nMeanwhile, we confirmed that Lena was not included in the training set. We also removed the Lena image from the Set 14 test dataset and validated the accuracy of each method again. As the PSNR shown in Table 4, the results indicate that after removing Lena image, our TinyLUT still achieves competitive results and does not affect the ranking of our method.\n\n**Table 4 The 4x SR experiment for Set14 without Lena**\n|              | Set14v without Lena | Set14 |\n|--------------|---------------------|-------|\n| SRLUT-S      | 26.80               | 27.01 |\n| SPLUT-L      | 27.26               | 27.54 |\n| MULUT-SDY-X2 | 27.31               | 27.60 |\n| TinyLUT-S    | 27.02               | 27.33 |\n| TinyLUT-F    | 27.72               | 28.01 |\n\n[1] Shaker et al. \"SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications\", ICCV 2023\n\n[2] Howard et al. \"Searching for mobilenet-v3\", CVPR 2019\n\n[3] Sandler et al. \"Mobilenet-v2: Inverted residuals and linear bottlenecks\", CVPR 2018\n\n[4] Woo et al. \"ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders\", CVPR 2023\n\n[5] Liu et al. \"A ConvNet for the 2020s\", CVPR2022\n\n[6] Zhang et al. \"Efficient CNN Architecture Design Guiede by Visualization\", ICME 2022\n\n[7] Li et al. \"MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution\", ECCV2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VCNSH69dom", "reviewer": "Reviewer_G5Ti", "summary": "The paper presents TinyLUT, a method that significantly reduces LUT-based image restoration storage requirements for edge devices through separable mapping strategy and dynamic discretization mechanism, achieving competitive accuracy with over 5 times faster inference.", "questions": "1. How do the models deploy in edge devices? Which computing engine or accelerating library is used? I think it is helpful for readers to understand the efficiency of the LUT-based methods.", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed separable mapping strategy (SMS) and dynamic discretization mechanism (DDM) are efficient and effective for restoration tasks.\n2. The proposed TinyLUT achieves significant reduction in memory storage and also gains better accuracy-latency tradeoff over other methods.\n3. The paper demonstrates the great potential of LUT-based methods.", "weaknesses": "1. The proposed SMS and DDM are used to built TinyLUT model, so how can these two modules help to improve other models? That is, it would be better if you can validate their generalization as general compression modules.\n2. Should compare to more recent works, like [a].\n\n[a]. Look-Up Table Compression for Efficient Image Restoration. Li. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness 1** The reviewer's comment is very insightful. The proposed SMS and DDM methods are applicable to other LUT-based models such as SRLUT[1], SPLUT[2] and MULUT[3].\n\nOur proposed SMS enables decomposition of the input from RF and channels of these models for fewer LUT storage overhead. For example, the storage of SRLUT-S can be reduced from 1.27MB to 16KB by SMS. Moreover, the addition of DDM will further reduce the storage from 16KB to about 4.25KB to adapt to the deployment on edge devices. Due to time constraints during the rebuttal period, comprehensive validation of this aspect will be presented in a subsequent appendix.\n\n**Response to Weakness 2** The reviewer's suggestion is very valuable. To demonstrate the advantages of our method, we have conducted comparisons on image super-resolution and image denoising tasks with method [4] in Table 1. As described in your 'Strengths' section, these experimental outcomes further substantiate the considerable potential of LUT-based approaches. The method [4], being another LUT compression framework, was published after the submission of our manuscript. Method [4] based on DFC strategy preserves diagonal pairs to maintain the representation capacity, while non-diagonal pairs are aggressively subsampled to save storage.  As can be seen, TinyLUT-F achieves the highest PSNR with about 12x storage reduction compared to SPFLUT+DFC. \nSince DFC and TinyLUT are different technological routes, we can combine them to achieve greater storage reduction. \n\n**Table 1 Quantitative comparisons on 4x SR task**\n|            | storage  | Set5  | Set14 | BSD100 | Manga109 | Urban100 |\n|------------|----------|-------|-------|--------|----------|----------|\n| SPFLUT     | 17.284MB | 31.11 | 27.92 | 27.10  | 28.68    | 24.87    |\n| SPFLUT+DFC | 2.018MB  | 31.05 | 27.88 | 27.08  | 28.58    | 24.81    |\n| TinyLUT-F  | 171KB    | 31.18 | 28.01 | 27.13  | 28.83    | 24.92    |\n\nIn addition, we compare SPFLUT+DFC and our TinyLUT in the image denoising task with noise level 15 on Set12 and BSD68. As shown in Table 2, TinyLUT still yields about 0.2dB higher PSNR with 3× storage reduction compared to SPFLUT+DFC. The results further demonstrate that TinyLUT exhibits notable advantages in other image restoration tasks. These experiments corroborate the effectiveness of the proposed method, as noted by reviewers (G5Ti, MFFN).\n\n**Table 2 Quantitative comparisons on image denoise**\n|            | storage | Set12 | BSD68 |\n|------------|---------|-------|-------|\n| SPFLUT     | 3017KB  | 32.11 | 31.17 |\n| SPFLUT+DFC | 595KB   | 32.01 | 31.09 |\n| TinyLUT-F  | 187KB   | 32.22 | 31.20 |\n\n**Response to Question 1** The inference speed evaluation was conducted by deploying TinyLUT on two distinct platforms: Xiaomi 11 smartphone and Raspberry Pi 4B.\nFor the Xiaomi 11 platform, the TinyLUT was implemented in C++ and  invoked by Java Native Interface. Multi-thread parallel acceleration for look-up table was achieved through the Stream API.\nFor the Raspberry Pi 4B implementation, the TinyLUT model was also coded in C++. We employed the \"pthread\" library and \"omp\" library for multi-thread look-up table acceleration . \nThe source code will be made publicly available on GitHub to help the readers to understand the efficiency of the LUT-based methods.\n\n[1] Jo et al. \"Practical Single-Image Super-Resolution Using Look-Up Table\", CVPR2021\n\n[2] Ma et al. \"Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution\", ECCV2022\n\n[3] Li et al. \"MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution\", ECCV2022\n\n[4] Li et al. \"Look-Up Table Compression for Efficient Image Restoration\", CVPR2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XaNJmtkz7q", "reviewer": "Reviewer_2kHn", "summary": "This paper introduced a separable mapping strategy that solves the storage issue of LUT-based methods. In addition, a dynamic discretization mechanism is designed to decompose the activation and compression quantization scales. Experimental results show the potential of this work for image restoration tasks.", "questions": "Please see Weaknesses.", "limitations": "Limitations were included.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper was well written and organized.\n\n2. The performance improvement of SR task is remarkable.\n\n3. The proposed method is valuable for advancing image restoration on edge devices.", "weaknesses": "1. There is a lack of comparison with the state-of-the-art methods [1, 2], especially the literature [1] which also addresses the storage problem of LUT-based methods.\n\n2. This paper claims to work on the image restoration problem, however only two restoration tasks were included in the experiments. Validation on more image restoration tasks is needed.\n\n3. The proposed method does not perform well enough on the image denoising task. With similar inference efficiency, the performance of the proposed method lags far behind MULUT-SDY-X2 even more than 1 dB. I am concerned about the effectiveness of this work on a wider range of image restoration tasks.\n\n\n> 1. Look-Up Table Compression for Efficient Image Restoration. CVPR 24.\n\n>2. Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution. ICCV 23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1** The reviewer's reminder is very important and constructive. To address the reviewer's concern, we conduct a comparative study with the abovementioned two methods[1,2]. The comparison indicates that our TinyLUT is more effective than the latest storage saving LUT methods. Literature [1] proposed a LUT compression framework named DFC for efficient image restoration with a different technological route from our proposed TinyLUT. The DFC preserves diagonal LQHQ pairs to maintain representation capacity, while non-diagonal pairs are aggressively subsampled to save storage. Method [2] proposed a reconstructed convolution to decouple the calculation of spatial and channel-wise features for maintaining large RF with less LUT size. \nAs shown in Table 1, TinyLUT-F achieves much higher PSNR with 9× and 12× lower storage consumption than RCLUT [2] and SPFLUT+DFC [1], respectively. \n\n**Table 1 Quantitative comparisons on 4x SR**\n|                | storage  | Set5  | Set14 | BSD100 | Manga109 | Urban100 |\n|----------------|----------|-------|-------|--------|----------|----------|\n| RCLUT [2]      | 1.513MB  | 30.72 | 27.67 | 26.95  | 28.05    | 24.57    |\n| SPFLUT [1]        | 17.284MB | 31.11 | 27.92 | 27.10  | 28.68    | 24.87    |\n| SPFLUT+DFC [1] | 2.018MB  | 31.05 | 27.88 | 27.08  | 28.58    | 24.81    |\n| TinyLUT-F      | 171KB    | 31.18 | 28.01 | 27.13  | 28.83    | 24.92    |\n\nIn addition, we compare SPFLUT+DFC [1] and TinyLUT in image desnoing with noise level 15. Note that RCLUT [2] did not test on image denoising task. As shown in Table 2, TinyLUT still yields about 0.2dB higher PSNR with 3× storage reduction compared to SPFLUT+DFC. The results further demonstrate that TinyLUT exhibits notable advantages in other image restoration tasks. These experiments corroborate the effectiveness of the proposed method, as noted by reviewers (G5Ti, MFFN).\n\n**Table 2 Quantitative comparisons on image denoise**\n|            | storage | Set12 | BSD68 |\n|------------|---------|-------|-------|\n| SPFLUT     | 3017KB  | 32.11 | 31.17 |\n| SPFLUT+DFC | 595KB   | 32.01 | 31.09 |\n| TinyLUT-F  | 187KB   | 32.22 | 31.20 |\n\n**Response to W2** Thanks for the reviewer's insightful suggestion. As claimed in [3,4], image denoising and SR are classical yet still active topics in low-level vision since they are indispensable steps in many practical applications. Most of the image restoration models[3,4] adopted denoising and super-resolution as representative tasks. Upon the reviewer's insightful comment, we fully agree that the limitations of focusing solely on SR and image denoising cannot comprehensively demonstrate the generality in image restoration. \n\nTo address this concern, we incorporate the deblocking task widely employed in recent literature [1] into our experiment. The results of this experiment demonstrate the value of our method for advancing image restoration on edge devices. As shown in Table 3, TinyLUT-F still achieves remarkably higher PSNR-B than other LUT methods with much lower storage consumption. \n\n**Table 3 Quantitative comparisons on image deblock**\n|            | storage | Classic5 | LIVE1 |\n|------------|---------|----------|-------|\n| SRLUT      | 81KB    | 27.58    | 27.69 |\n| MULUT      | 489KB   | 28.29    | 28.39 |\n| SPFLUT     | 3017KB  | 28.63    | 28.62 |\n| SPFLUT_DFC | 595KB   | 28.62    | 28.61 |\n| TinyLUT-F  | 187KB   | 28.74    | 28.67 |\n\n**Response to W3** The reviewer's reminder is very important. In the image denoising task, our TinyLUT-S does lag far behind MULUT-SDY-X2 in accuracy, which is results from the significant reduction in storage for deploying on edge devices with very limited resource budgets, especially memory and storage [5,6]. As shown in Table 4, our proposed TinyLUT-S occupies merely 7.6% of the storage required by MULUT-SDY-X2 that is the most lightweight version of MULUT. \n\nAs for the inference efficiency, inspired by comments from reviewers G5Ti and 2kHn, we found that still some operations can be merged to reduce inference latency, such as residual addition between blocks. Meanwhile, we have introduced pthread and omp to accelerate inference in a multi-threaded parallel manner. As a result, the inference latency of TinyLUT-S is reduced to 27ms on Raspberry Pi 4B, yielding a real-time inference efficiency comparable to the current lightest SRLUT with about 1dB PSNR promotion and 4× lower storage. Hence, our TinyLUT achieves a better trade-off between accuracy and latency, as noted by reviewer G5Ti.\n\n**Table 4 Quantitative comparisons on denoise and latency**\n|                 | Runtime | Storage | Set12 | BSD68 | Set12 | BSD68 | Set12 | BSD68 | Average |\n|-----------------|---------|---------|-------|-------|-------|-------|-------|-------|---------|\n| SRLUT           | 21ms    | 82KB    | 30.42 | 29.78 | 27.19 | 26.85 | 22.62 | 22.39 | 26.54   |\n| MULUT-SDY-X2    | 44ms    | 289KB   | 31.50 | 30.63 | 28.94 | 28.18 | 25.46 | 24.97 | 28.28   |\n| MULUT-SDYEHO-X2 | 89ms    | 978KB   | 31.77 | 30.89 | 29.18 | 28.34 | 25.47 | 24.96 | 28.44   |\n| TinyLUT-S       | **27ms**    | 22KB    | 31.10 | 30.24 | 28.26 | 27.48 | 24.29 | 23.83 | 27.53   |\n\nIn addition, we compare our method and other LUT-based methods in image deblocking task under a quality factor of 10 in Table 3. The results demonstrate the effectiveness of our work on other image restoration tasks.\n\n[1] Li et al. \"Look-Up Table Compression for Efficient Image Restoration\", CVPR2024\n\n[2] Liu et al. \"Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution\", ICCV2023\n\n[3] Zhang et al. \"Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising\", TIP 2017\n\n[4]  Kim et al. \"Accurate Image Super-Resolution Using Very Deep Convolutional Networks\", CVPR 2016\n\n[5] Lin et al. \"MCUNet: Tiny Deep Learning on IoT Devices\",  NIPS 2020\n\n[6] Lin et al. \"MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning\", NIPS2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MkRFaCT1Xy", "reviewer": "Reviewer_MFFN", "summary": "To address the storage explosion challenge of LUT, this paper proposes a separable mapping strategy (SMS) and a dynamic discretization mechanism (DDM) to decompose the kernel and activation to reduce the storage consumption. Specifically, the SMS decomposes the convolution into independent sub-operations to reduce the input entries, consequently reducing the dimension of LUT. Additionally, the DDM explicitly compresses the quatization scales with learnable clipping parameters to decrease LUT scales.", "questions": "1. To my understanding, the proposed strategy can not be applied to transformer-based model like SwinIR. Why is SwinIR introduced in the related works, and why is it categorized under CNN methods?\n2. Different from previous methods, the proposed method is not contrained by the storage. Can the proposed method achieve better performance by increasing the RF and channels?\n3. Why is the denoising experiment placed after the ablation experiments? The reviewer suggests combining it with the SR evaluation.", "limitations": "The limitations have been discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper analyzes the storage explosion challenge of LUT and provides a solution by decomposing the convolution kernel and compressing the quantization scale.\n2. The proposed image restoration method is efficient and effective for edge devices, as demonstrated in the experiments.", "weaknesses": "1. Although the separable mapping strategy (SMS) could significantly reduce the storage, it may lead to information loss and performance drop. As shown in Figure 3, the SMS obtains the final result using individually mapped indexes and neglects the relationship between the local indexes, which is important for convolutions. \n2. The motivations and benefits of activation decomposition in the DDM are unclear. Moreover, this paper does not sufficiently discuss the difference between DDM and former activation decomposition used in SPLUT.\n3. The discussion of LUT-based methods in the related works is insufficient. The key ideas and limitations of these methods should be elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness 1** We thank the reviewers for their comprehensive review. It must be clarified that the proposed SMS strategy will not result in information loss or performance degradation due to neglecting the relationships between local indexes.  As for the original depthwise convolution operation, it can be obtained as follows:\n\\begin{equation}\n\tF_{out} = \\sum_{i=0}^{n-1}\\sum_{j=0}^{n-1} {I_{i,j}*w_{i,j}}\n\\end{equation}\nThe reconstruction of local index relationships is achieved through a summation process. This process involves the multiplication of each pixel within the receptive field with its corresponding weight. The resulting products are then aggregated to form the final output.\n\nAfter the CNN model training, we stored the results of each pixel within the receptive field multiplied with the trained weights in the corresponding LUT.\n\\begin{equation}\n\tLUT_{(i,j)}[I_{(i,j)}] = I_{i,j}*w_{i,j}\n\\end{equation}\nSimilarly, after obtaining individual input-corresponding results through separately mapped indices in SMS, the final result is constructed via summation of all outputs. The relationships between local indices are reconstructed in this process. This process is formally expressed in the next equation of our manuscript:\n\\begin{equation}\n\t\\hat F_{out} = \\frac{1}{n^{2}} \\sum_{i=0}^{n-1}\\sum_{j=0}^{n-1} LUT_{(i,j)}[x_{(i,j)}]\n\\end{equation}\nFollowing the discrete retrieval in SMS, a summation of individual results is necessitated to reconstruct the relationships among local indices. To prevent misinterpretation, we will refine Figure 3 to explicitly illustrate this crucial step in the process.\n\n**Response to Weakness 2** We appreciate the reviewer's insightful question. Our motivation for DDM stems from the limitations of the previous SPLUT approach, which was constrained to a 4 MSBs and 4 LSBs symmetric decomposition scheme for 8-bit inputs.  SPLUT uses a fixed quantization scale for each layer. These constraints inherently limited model performance.\n\nOur DDM proposed an innovative asymmetric activation decomposition strategy and adaptively finds the right quantization scale for each layer, thus significantly boosting the inference accuracy and achieving a better balance between accuracy and storage. \nNotably, the DDM strategy is not exclusive to our method. It can be applied to other LUT-based methods to further reduce the storage consumption.\n﻿\n\n\n\n**Response to Weakness 3** The reviewer's reminder is very important and insightful. In response, we will augment the related work section with an explication of the methodologies presented in [1] and [2], the key ideas and limitations are also will be elaborated. In addition to summarizing the key ideas and limitations of SRLUT, SPLUT and MULUT, we will also summarize RCLUT's proposal of the plugin module to increase the RF of the model with slightly increasing storage overhead. SPFLUT[1] proposed a LUT compression framework to balance the performance improvement and storage growth. The incorporation of these references will enhance the comprehensiveness of our literature review.\n\n**Response to Question 1** We extend our gratitude for the reviewer's meticulous examination. The inclusion of SwinIR in the related work section was motivated by its significant performance. However, upon careful consideration of the reviewer's astute observation, SwinIR does not align with the CNN-centric theme of our work, and the inappropriate description is removed from the manuscript.\n\nFurthermore, we restructure the categorization of methods in the related work section. Specifically, we will supplement [3] and [4] under the CNN methods subsection to provide a more comprehensive overview of CNN approaches. \nWe reiterate our appreciation for the reviewer's thorough and insightful feedback, which has significantly contributed to enhancing the coherence and relevance of our manuscript.\n\n**Response to Question 2** We appreciate the reviewer's suggestion for improvement. We validated the suggestions of the reviewer in the 4x SR task. As shown in Table 1, the improvement resulting from using a 5x5 RF and 32 feature channels shows an increase in accuracy with only a minor linear growth in storage overhead compared to the baseline.\n\nWe extend our sincere gratitude to the reviewers for their insightful suggestions, which have illuminated additional avenues for exploration in our method.\n\n**Table 1 Quantitative comparisons on 4x SR**\n|                     | Storage | Set5  | BSD100 | Manga109 |\n|---------------------|---------|-------|--------|----------|\n| TinyLUT-S           |    37KB     | 30.22 | 26.71  | 27.21    |\n| TinyLUT-5x5 RF      |   60KB      | 30.30 | 26.75  | 27.24    |\n| TinyLUT-32 channels |    136KB     | 30.35 | 26.77  | 27.39    |\n\n**Response to Question 3** The reviewer's suggestion is very important and insightful. We will revise the structure of the experimental section and combine the denoising experiments with the SR evaluation. This reorganization will significantly improve the clarity and flow of our research presentation.\n\n[1] Li et al. \"Look-Up Table Compression for Efficient Image Restoration\", CVPR2024\n\n[2] Liu et al. \"Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution\", ICCV2023\n\n[3] Lai et al. \"Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks\", TPAMI 2019\n\n[4] Tassano et al. \"FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation\", CVPR2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their valuable and professional comments. We are thankful that most of the reviewers shared positive feedback for our paper. Meanwhile, we are glad to read that our work was efficient (JnS6, G5Ti, MFFN), effective (G5Ti, MFFN),  valuable (2kHn), potential (G5Ti), well written and organized (2kHn). 

We appreciate the reviewers' acknowledgement of our proposed method's potential and contribution to the LUT-based approaches. As reviewer MFFN noted, our study "analyzes the storage explosion challenge of LUT and provides a solution by decomposing the convolution kernel and compressing the quantization scale". Our method solves the storage explosion problem encountered in recently proposed SRLUT[1], SPLUT[2] and MULUT[3]. Our work has promoted the development of LUT-based methods, as expressed by reviewer G5Ti: "The paper demonstrates the great potential of LUT-based methods," and as reviewer 2kHn also expressed: "The proposed method is valuable for advancing image restoration on edge devices."

However, the reviewers have some concerns about our paper. The reviewer expressed concern that our proposed method is only suitable for CNN-based architectures (JnS6). Meanwhile, the reviewers suggest that we should compare with the latest LUT-based methods, especially the SPFLUT[4] mentioned by G5Ti and 2kHn. The last concern is the violation of the Lena image mentioned by reviewer JnsS6. 

Regarding the concern that our method is only suitable for CNN-based architectures. Indeed, our proposed TinyLUT as well as existing LUT-based methods are mainly designed for CNN-based architectures. This is because CNN are still the preferred choice for real-time deployment on mobile devices, primarily because the convolution operation is computationally efficient[5,6,7]. CNNs compete favorably with Transformers in terms of accuracy and scalability, and convolution remains much desired and has never faded[8,9]. In recent years, many CNN methods have still been proposed[8,9,10]. 

Regarding the comparison with the latest LUT-based methods, the [4] proposed SPFLUT+DFC method, was published in CVPR2024 after we submitted the manuscript. To demonstrate the advantages of our method, we have conducted the comparisons on image super-resolution and image denoising tasks with SPFLUT+DFC in Table 1. As can be seen, TinyLUT-F achieves the highest PSNR with about 12x storage reduction compared to SPFLUT+DFC. As described in the 'Strengths' section by G5Ti, these experiments further demonstrate the value of our method for image restoration on edge devices.

**Table 1 Quantitative comparisons on 4x SR task**
|            | storage  | Set5  | Set14 | BSD100 | Manga109 | Urban100 |
|------------|----------|-------|-------|--------|----------|----------|
| SPFLUT     | 17.284MB | 31.11 | 27.92 | 27.10  | 28.68    | 24.87    |
| SPFLUT+DFC | 2.018MB  | 31.05 | 27.88 | 27.08  | 28.58    | 24.81    |
| TinyLUT-F  | 171KB    | 31.18 | 28.01 | 27.13  | 28.83    | 24.92    |

In addition, we compare SPFLUT+DFC and our TinyLUT in image denoising task with noise level 15 on Set12 and BSD68. As shown in Table 2, TinyLUT still yields about 0.2dB higher PSNR with 3× storage reduction compared to SPFLUT+DFC. The results further demonstrate that TinyLUT exhibits notable advantages in other image restoration tasks. These experiments corroborate the effectiveness of the proposed method, as noted by reviewers (G5Ti, MFFN).

**Table 2 Quantitative comparisons on image denoise**
|            | storage | Set12 | BSD68 |
|------------|---------|-------|-------|
| SPFLUT     | 3017KB  | 32.11 | 31.17 |
| SPFLUT+DFC | 595KB   | 32.01 | 31.09 |
| TinyLUT-F  | 187KB   | 32.22 | 31.20 |

Regarding the violation of the Lena image mentioned by reviewer JnS6, we will use the Baby image from the Set5 test set to modify Figure 2 in our paper. We have ensured that Lena was not used in the training process and found that removing it from testing does not affect the conclusion of our paper.

[1] Jo et al. "Practical Single-Image Super-Resolution Using Look-Up Table",  CVPR2021

[2] Ma et al. "Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution",  ECCV2022

[3] Li et al. "MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution",  ECCV2022

[4] Li et al. "Look-Up Table Compression for Efficient Image Restoration",  CVPR2024

[5] Shaker et al. "SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications", ICCV 2023

[6] Howard et al. "Searching for mobilenet-v3", CVPR 2019

[7] Sandler et al. "Mobilenet-v2: Inverted residuals and linear bottlenecks", CVPR 2018

[8] Liu et al. "A ConvNet for the 2020s",  CVPR2022

[9] Woo et al. "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders", CVPR 2023

[10] Zhang et al. "Efficient CNN Architecture Design Guiede by Visualization",  ICME 2022

Meta-review: The paper considers lookup table based methods for accelerating CNNs for image reconstruction. The paper notes that the memory complexity of lookup table based methods for image reconstruction grows exponentially in the size of the convolutional kernel, which may create a storage issue. 

The paper received four reviews:
* Reviewer JnS6 (3: reject) main concern is that accelerating CNN-based methods (in particular DnCNN) is not convincing and that more modern architectures should be considered. I agree with the authors' response that CNN based methods are still relevant in particular when speed matters. The reviewer also notes that the paper is not well written, and I fully agree to this concern. 
* Reviewer G5Ti (6: weak accept after rebuttal) notes that the paper should compare to recent works that address the same problem, in particular the paper
[1] 2024 CVPR paper ``Look-Up Table Compression for Efficient Image Restoration'', 
which is about the exact same issue. In the rebuttal the paper compares to the paper [1]. 
* Reviewer 2kHn is also concerned about a lack of comparisons, that comparisons on more tasks (beyond denoising and super-resolution) are needed, and that the performance in terms of PSNR lacks behind state-of-the-art methods. The authors provided additional comparisons and notes that while the performance is slightly lower than state-of-the-art methods, the latency and storage are much better, which is a valid point. I also think that considering denoising and super-resolution is sufficient. 
* Reviewer MFFN (7: accept after rebuttal) has no major concerns after rebuttal.

In summary, the paper proposes a method to effectively reduce the storage complexity of lookup table based methods for accelerating CNNs, and the method is sufficiently validated.
Decision: Accept (poster)