Expanded Convolutional Neural Network Based Look-Up Tables for High Efficient Single-Image Super-Resolution

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:00:57

Abstract: Advanced mobile computing has led to a surge in the need for practical super-resolution (SR) techniques. The look-up table (LUT) based SR-LUT has pioneered a new avenue of research without needing hardware acceleration. Nevertheless, all preceding methods that drew inspiration from the SR-LUT framework invariably resort to interpolation and rotation techniques for diminishing the LUT size, thereby prolonging the inference time and contradicting the original objective of efficient SR. Recently, a study named EC-LUT proposed an expanded convolution method to avoid interpolation operations. However, the performance of EC-LUT regarding SR quality and LUT volume is unsatisfactory. To address these limitations, this paper proposes a novel expanded convolutional neural network (ECNN). Specifically, we further extend feature fusion to the feature channel dimension to enhance mapping ability. In addition, our approach reduces the number of single indexed pixels to just one, eliminating the need for rotation tricks and dramatically reducing the LUT size from the MB level to the KB level, thus improving cache hit rates. By leveraging these improvements, we can stack expanded convolutional layers to form an ECNN, with each layer convertible to LUTs during inference. Experiments show that our method improves the overall performance of the upper limit of LUT based methods. For example, under comparable SR quality conditions, our model achieves state-of-the-art performance in speed and LUT volume.

Main paper: 
 related work 2.1 traditional super-resolution: 
 Interpolation-based methods are widely used, such as nearestneighbor interpolation, bilinear interpolation, and bicubic [9] interpolation. These methods obtain SR results by taking the weighted average of pixels near the target location. Interpolation-based methods are simple and efficient. However, they only consider positional information and do not fully consider the arrangement of different pixel values, so they cannot effectively recover high-frequency signals. Sparse coding-based methods [27] infer HR images by learning the sparse representation of patches. However, computing sparse representations significantly increases inference time.Deep neural networks (DNNs) have revolutionized the field of SR with their powerful fitting capabilities. Since the pioneering work of Dong et al. with SRCNN [5] , the landscape of SISR has been enriched by a plethora of innovative models, each introducing novel architectural features and learning strategies [1, 6, 10, 11, 25, 13, 14, 15, 23] . The trend towards increasingly complex models [17, 28, 33] has led to significant performance gains. This is exemplified by the introduction of residual learning strategies and attention mechanisms [22, 32] , which have become integral components of modern SR frameworks. The EDSR [17] and RDN [33] models, for instance, employ extensive residual blocks and have set new benchmarks in terms of image quality. Despite these advancements, the deployment of such models on edge devices remains a challenge due to their computational intensity and memory requirements.Techniques like quantization and pruning offer some relief by compressing the models without a substantial loss in performance. However, the need for specialized hardware like GPUs, DSPs, or NPUs for efficient inference underscores the gap between research prototypes and real-world applicability. Overall, while DNNs have significantly pushed the boundaries of SISR, the quest for practical, deployable solutions that balance performance with efficiency continues to be a driving force in the field.Look-up tables (LUTs) do not require special hardware implementation and only need some extra storage space to accelerate algorithm running speed by trading space for time. Jo et al. [8] were the first to apply LUTs to SR tasks and proposed SR-LUT. SR-LUT trains an SR network and then finds an equivalent LUT. The LUT replaces the complex DNN in the inference phase, significantly reducing the computational burden and making SR-LUT more practical. Subsequently, Li et al. [16] proposed MuLUT, which improved SR performance by using multiple LUTs to enlarge the receptive field (RF). Ma et al. [20] developed SP-LUT with a similar idea, using multiple cascaded LUTs to enlarge the RF. Note that SP-LUT quantized the intermediate features to 4 bits by using two parallel branches of MSB and LSB, thereby avoiding the interpolation operation. Meanwhile, they also discarded the rotation operation. This was an important attempt to break out of the SR-LUT framework. However, each module in SP-LUT only enlarges the RF by a small part in the horizontal or vertical direction, and the method leads to the need to stack many modules to enlarge the RF, which seriously affects efficiency. Liu et al. [19] proposed an RC module, which significantly enlarged the RF and improved SR performance with only a little extra storage. However, RCLUT followed the SR-LUT framework, so it had no advantage in inference time. Recently, Yin et al. [30] introduced an expanded convolution method that increases the RF by expanding the output window size in the spatial dimension rather than the input window size. This novel approach has piqued our interest. Nonetheless, the performance of EC-LUT in terms of SR quality and LUT volume remains unsatisfactory. 
method: 
 Figure 2 shows the overall workflow of ECNN and its differences from the EC-LUT [30] method. EC-LUT enlarged the RF by expanding the output window size of convolution in the spatial dimension, thereby maintaining the SR quality without interpolation and gaining an advantage in inference speed. To completely avoid the negative effects of interpolation and rotation operations, we improved the expanded convolution (EC) by further extending the expansion to the channel dimension and reducing the number of single indexed pixels to just one. Now, EC looks like an inverted vanilla convolution, which generates multiple output values from a single input value. The output values are placed in the corresponding positions and in-place addition is performed. To enhance the mapping ability, we construct an ECNN by connecting multiple EC layers.Conventional convolution operations compute the product of each pixel in an image with the corresponding elements of a filter in its neighborhood, followed by a summation to produce the convolution output. EC-LUT introduces an EC method that feeds pixels within the convolution input window into a neural network, which then outputs 𝑛 × 𝑚 values arranged into an output window. Each element of the output window is then added to the corresponding pixel in the output image. In other words, during the convolution process, data is read from the input image by the sliding input window and written into the output image by the output window (in-place addition operation). In EC, enlarging both the input and output windows can increase the RF. However, the size of the LUT is exponentially related to the input window size and linearly related to the output window size. Therefore, EC-LUT reduces the LUT size by enlarging the output window and shrinking the input window, whereas other LUT-based methods reduce the LUT size by sampling the LUT. Notably, the sampled LUT requires interpolation techniques during the inference stage to fill in missing dictionary values, significantly increasing computational complexity. It is precisely because interpolation is not needed that EC-LUT achieves faster inference speeds. Following the concept of EC-LUT, we further reduce the EC input window to the size of a single pixel. At this point, the size of the LUT is significantly reduced from GB or MB levels to KB levels. The subsequent challenge is how to improve SR quality. Previous studies have improved SR quality by cascading multiple LUTs. However, we observed that even when cascading multiple LUTs and employing different branches to process the input image separately, these methods are merely mapping one feature plane to another. In contrast, CNN networks first map the input image to a high-dimensional latent space and perform feature extraction in that space. Inspired by this, this paper further extends the EC feature fusion operation from the spatial dimension to the channel dimension. Furthermore, given the presence of multiple channels, it is logical to stack multiple EC layers to form an ECNN. Figure 3 shows the details of sub-network in the EC layer. The sub-network consists of three 1 × 1 convolutions and two ReLU activation functions. To shorten training time, the hidden layer channel count of the sub-network is set to 64. The EC layer can be formally described as:EQUATIONwhere (𝑖, 𝑗, 𝑐) is the pixel index in the feature map, F 𝑖𝑛 (𝑖, 𝑗, 𝑐) represents the input window at location (𝑖, 𝑗, 𝑐), X represents the corresponding three-dimensional sliding output window, Φ represents the learnable sub-network, 𝜃 represents the parameters of Φ, F 𝑜𝑢𝑡 represents the final output feature obtained at the target position, and 𝜒 denotes the sliding window sets that cover the target position.As shown in Figure 2 , EC can be applied as a regular convolution operation. Hence, our ECNN model consists solely of EC layers, and it extracts the feature maps in the LR space to lower the computational burden. Then, the model appends an up-sampling module at the final stage of the network to enhance the resolution. Furthermore, inspired by VGG [24] , instead of using a single large-scale RC module to expand the receptive field as in RCLUT, we stack several 3 × 3 EC layers to increase efficiency. The first layer of ECNN is responsible for ascending the input channel from 𝑐ℎ 𝑖𝑛 to 𝑐ℎ 𝑚𝑖𝑑 . The intermediate hidden layers use a 3× 3 EC layer with input and output channels of 𝑐ℎ 𝑚𝑖𝑑 . Previous LUTbased SR methods have used multiple branches to enhance mapping capabilities, but the ends of these branches are always merged together. In other words, when these branches are considered as a whole, they always map one plane to another plane. However, on a single feature plane, the information contained in local patches is limited. ECNN uses multiple channels, which contain richer information in local patches, resulting in higher SR quality.The final layer of ECNN is an up-sampling module, which directly splits each value of the input features into 𝑟 × 𝑟 values by non-linear mapping, as shown in Figure 2 . Finally aggregates them along the channel axis to compress back to the low-dimensional RGB space. Actually, this procedure can also be viewed as setting the output window size of the EC layer to 𝑟 × 𝑟 , and then when reordering the output, there is no overlap in the spatial dimension, and only overlap in the channel dimension.It should be noted that another major distinction between ECNN and the SR-LUT framework is ECNN's ability to simultaneously process all three RGB channels. When there are ample channels in the feature maps extracted within the network, sharing a single set of extracted features for all three RGB channels can reduce computational requirements by approximately one-third, thereby further accelerating inference speed. However, when there are fewer intermediate feature channels, this shared feature approach may fail to adequately represent the information of all three RGB channels, compromising SR quality. This study continues to employ the approach of channel separation.Converting a neural network into a LUT necessitates computing the output values for all possible input values. These input values serve as indices for the LUT, with the corresponding output values stored at those indices. In the case of SR-LUT, the input is a LR image with a data bit-width of 8 bits, thus obviating the need for quantization operations. However, when cascading multiple LUTs, the intermediate feature tensors act as inputs to the hidden layers, which typically utilize 32-bit floating-point data during training. As shown in Equation 1, the LUT size is exponentially related to the input value's bit-width, it becomes necessary to quantize the feature maps. Previous studies, such as SP-LUT, directly incorporate quantization nodes into the training process. However, feature maps produced by randomly initialized neural networks span a wide numerical range, and the truncation functions used in quantization nodes can impede gradient propagation. Experiments have demonstrated that this approach fails to achieve optimal performance. Alternatives like MuLUT and RCLUT employ Tanh or Sigmoid activation functions to map feature maps to a finite range. However, these methods still affect the model's convergence rate. To address this issue, we analyzed the results of standard network training, excluding quantization nodes. As illustrated in Figure 4 , we observed that in a normally trained network, the output value range of each layer falls within [-1, 1]. Therefore, truncating the data to [-1, 1] will no longer hinder gradient propagation. Based on this finding, we propose decoupling quantization from model training, as depicted in Figure 5 . The model is first trained normally, and upon convergence, quantization nodes are introduced for finetuning to ensure the model's performance remains unaffected. This fine-tuning process is referred to as quantization-aware training (QAT).The computation of the quantization node can be formulated as follows:𝑠 = 2 𝑏 -1 -1, 𝑍 = 𝑐𝑙𝑎𝑚𝑝 (𝑟𝑜𝑢𝑛𝑑 (𝑥 • 𝑠), -𝑠, 𝑠), 𝑦 = 𝑍 𝑠 (3)where 𝑠 is the scaling factor, which is determined by the quantization bit number 𝑏, 𝑥 is the input floating-point number, 𝑍 is the integer after quantization, and 𝑦 is the output of quantization node. We attempted to quantize the feature maps to 4 bits, but this resulted in a drastic performance drop, so we eventually opted for 8 bits quantization.Note that QAT is performed on the input (LUT index) of the sub-network in the EC layer, whereas the output (values stored in LUT) of the sub-network remains a floating-point number prior to the in-place addition operation. In order to further reduce the LUT volume, the values stored in LUT were further quantified, but this step was postponed until after fine-tuning. For this step, experiments indicate that post-training quantization (PTQ) can be conducted directly, and quantization above 12 bits hardly affects performance. Thus, we quantize it to 12 bits at the LUT generation stage. It is worth noting that our PTQ does not require any statistical data about the model, which means that this step of quantization is zero-burden. 
experiment 4.1 implementation details: 
 Datasets and Metrics. We use the DIV2K dataset [26] for training. This dataset has 800 images for training, 100 for validation, and 100 for testing. In addition, there are five commonly used benchmark test datasets, namely Set5 [3] , Set14 [31] , B100 [2] , Urban100 [7] , and Mang109 [21] . We report our results on these five datasets and compare them with previous studies. The quantitative evaluation metrics are PSNR (peak signal-to-noise ratio) on the Y channel of YCbCr space and structural similarity index (SSIM) [29] . In addition, We evaluate the computation efficiency by recording and presenting the rumtime of generating 1280 × 720 output images on mobile devices. To be consistent with previous studies, according to [17, 33] , we use Matlab's imresize function to perform bicubic interpolation on HR images to obtain LR images.Training Details. The sub-network within the EC layer uses a total of 3 grouped 1 × 1 convolution layers, with the number of groups set to 𝑐ℎ 𝑖𝑛 , and the number of hidden layer channels set to 64 × 𝑐ℎ 𝑖𝑛 , where 𝑐ℎ 𝑖𝑛 denotes the input channels of the EC layer. The sub-network in the up-sampling module of the last layer of ECNN adopt the same settings. We use Adam optimizer [12] with an initial learning rate of 1 × 10 -4 for a total of 20000 epochs, halving the learning rate every 4000 epochs. The loss function is mean-squared error (MSE). We randomly crop the LR image into patches of size 48 × 48 with a mini-batch size of 16 and augment the data by random rotation and flipping. We train the ECNN model with Pytorch [4] on Nvidia 2080Ti GPU.We compared ECNN with other SR methods, including three common interpolation methods (nearest neighbor, bilinear, bicubic [9] ), five LUT-based methods (SR-LUT [8] , EC-LUT [30] , MuLUT [16] , SP-LUT [20] , RCLUT [19] ), and three neural network-based methods (FSRCNN [6] , CARN-M [1] , RRDB [28] ). To compare the evaluation metrics of different dimensions more clearly, we implemented three different complexities of ECNN: ECNN-L4C4, ECNN-L6C6, ECNN-L8C8, where the number after 'L' indicates the number of EC layers, and the number after 'C' indicates the number of channels in the hidden layer. The quantitative comparison results are shown in Table 1 . The results indicate that among the LUT-based SR methods, ECNN-L8C8 achieved the highest PSNR and SSIM quality. Compared with the earliest method SR-LUT, ECNN-L8C8 improved PSNR by 1.22dB on the Set5 dataset, and was close to SR-LUT regarding running time and LUT size. It is worth noting that the lower complexity ECNN-L6C6 surpassed all previous LUT-based methods across metrics, except for running slower than EC-LUT-V, thus demonstrating the superiority of ECNN. Compared with ECNN-L8C8, ECNN-L6C6 compromised only a small part of SR quality, but gained twice the running speed, showing better overall performance. Among all the LUT-based methods, ECNN-L4C4 had the shortest running time, yet experienced a more significant decrease in SR quality.Overall, the single LUT size of ECNN is at the KB level. ECNN-L4C4 and ECNN-L6C6 even have a total LUT size of less than 1MB. This reduced LUT size not only minimizes storage space utilization but also enables the KB-level LUT to be seamlessly loaded into the L1 cache, greatly enhancing the cache hit rate and subsequently improving inference speed. Moreover, despite having a smaller RF, ECNN-L6C6 and ECNN-L8C8 exhibit superior SR quality compared to RCLUT, demonstrating the superiority of a multi-channel approach. Additionally, all of our methods have faster inference speed than RCLUT, indicating that the strategy of utilizing multiple smallsized convolution modules instead of a single large-sized one is indeed effective. Specifically, comparing the first row, only ECNN-L6C6 and ECNN-L8C8 produced a good transition at the circular arc in the lower left corner of the mirror, while all other methods produced jagged edges. In the second and third rows, all other methods produced oblique stripe-like artifacts, whereas ECNN-L6C6 and ECNN-L8C8 effectively suppressed such artifacts. In the fourth row, there are two obvious black lines in the picture, which other methods restored as wavy lines, and there are also some reverse stripes in the middle.Ours-L6C6 Ours-L8C8 GT Our ECNN-L8C8 restored the lines effectively. In the last row, the white highlight on the edge of the clothes, ECNN generated a sharp edge, whereas other methods produced jagged edges. These results show that ECNN can better restore the high-frequency details of the image, while suppressing artifacts, and produce better visual effects.The effectiveness of proposed quantization strategy. We conducted ablation experiments on ECNN models with different complexities under various quantization strategies. The experimental results in Table 2 demonstrate the effectiveness of the quantization strategy proposed in this paper from two perspectives. Firstly, after undergoing both the QAT and PTQ stages (our strategy), the performance degradation of the three models is minimal, virtually negligible. Secondly, directly incorporating quantization nodes during the training process (a strategy employed in previous studies) has a minimal impact on the ECNN-L4C4 model, whereas the performance degradation of the ECNN-L8C8 model is significant. This indicates that when there are a large number of cascaded LUTs, the DQ strategy fails to achieve optimal performance, whereas our strategy remains effective regardless. The impact of PTQ quantization bit width. Although using 16-bit low-precision floating-point numbers as the storage format for LUT data is acceptable, as demonstrated by SP-LUT, fixed-point numbers still excel in computational efficiency. Therefore, it remains necessary to quantize this portion of data to further enhance performance. According to Table 3 , PTQ effectively offers acceleration. Meanwhile, as shown in Table 4 , when quantized to 12 bits, the model performance hardly suffers any decrease. But as the quantization level is reduced below 10 bits, the model performance experiences a significant drop. Notably, when quantized to 8 bits, the PSNR decreases by 1.13dB on the Set5 dataset. To balance both LUT size and SR quality, we opted to quantize to 12 bits.To compare the performance difference between the proposed ECNN and the ordinary convolutional neural network, experiments were conducted under the same conditions of layer number, channel number and equivalent convolution kernel size. As shown in Table 5 , the three ECNN models with different complexities all surpassed the ordinary CNN network in PSNR quality. This is because the ordinary convolution operation is simply a linear mapping that multiplies the weights and inputs and then sums them up, and its nonlinear mapping ability depends on the nonlinear activation function. ECNN first performs nonlinear mapping on the input data and then sums them up, which can approximate a more abstract representation of the latent knowledge. This is similar to the idea of Network In Network [18] . As shown in Figure 7 , compared with vanilla CNN, ECNN extracts more high-frequency signals in both the first and last layers. This explains why ECNN models with the same complexity perform better than vanilla CNNs. In addition, ECNN converted to LUT completely avoids multiplication and floating-point operations, and has more advantages in computational efficiency than ordinary convolution. 
conclusion: 
 In this paper, we propose the complete expanded convolution, which can be used to build ECNN models like vanilla convolution.After the ECNN training, we add quantization nodes to fine-tune the model and generate the equivalent LUT for inference stage acceleration. Our method only uses a single value as an index, which greatly reduces the size of each LUT and increases the cache hit rate. More importantly, we do not need the interpolation and rotation operations, which significantly improves the inference speed. In addition, ECNN increases the receptive field and enhances the mapping ability by using multi-channel multi-layer convolution, significantly improving SR quality. Experiments show that our models offer superior SR quality, faster inference, or smaller LUT volume compared to comparable LUT-based methods. In the future, we will explore arbitrary-scale SR methods based on ECNN to improve the practicality of ECNN further.  
: 
 Size (MB) 0.0 2.5 5.0 7.5Figure 1 : Illustration of PSNR on Set14 benchmark dataset for ×4 SR, model size and runtime which measured by generating 1280 × 720 image. We compare our methods (red) with common interpolation based methods (black), prior LUT-based methods (blue) and deep learning based methods (green). Our models offer superior SR quality, faster inference, or smaller LUT volume compared to comparable LUT-based methods.[9], and A+ [27] . Interpolation-based SR methods often produce vague results. Sparse encoding methods, however, suffer from slow inference speeds. Recently, deep learning-based SR methods [11, 25, 17, 28] have achieved significant performance improvements. However, these methods have dramatically increased computational costs and even require large amounts of memory, hindering their practical applications on resource-limited edge devices like smartphones and smartwatches. Look-up table (LUT) based methods store the outputs of complex computations in a LUT, which can be directly retrieved without recomputing when needed. The low computational cost and hardware-independent characteristics of LUT make it suitable for implementing efficient SR methods. However, obtaining a LUT requires enumerating the input values, which leads to an exponential growth of the LUT size with the increase of the index pixel number. When using a single pixel index, the LUT size is usually at the KB level. When using two pixel indices, the LUT size grows to the MB level. Although sampling techniques can be used to reduce the LUT size, on one hand, sampling techniques are limited in their ability to increase the index pixel number, typically only allowing for up to four indices. On the other hand, interpolation is required at the inference stage after sampling, which increases the computational costs. Cascading methods can make the LUT size grow linearly with the receptive field (RF) size, but these methods still remain within the confines of the SR-LUT [8] framework and still require interpolation and rotation techniques.Recently, a novel study [30] proposed an expanded convolution method to improve LUT-based SR methods' inference speed. By expanding the output window size in the spatial dimension thereby increasing the RF, using merely two pixels as indices, the performance outperforms SR-LUT that employs four pixels as indices. Meanwhile, due to avoidance of interpolation operations, it was much faster than SR-LUT. However, we found some problems in EC-LUT. First, the method still used rotation techniques without conferring any significant advantage. Second, it still used two pixels as indices, resulting in a LUT size of 9MB, which requires more storage space and reduces the cache hit rate. Finally, a bigger problem was that the expanded operation was performed in the HR space, significantly increasing the computational burden.To address these issues, in this paper, we further improve the expanded convolution (EC) and generalize it to more common cases. At the same time, we completely abandon the SR-LUT framework and directly use EC as a vanilla convolution to build an expanded convolutional neural network (ECNN). Specifically, based on the previous version of EC, we further extend the expanded operation to the channel dimension while reducing the input window size to 1 × 1. Now, the input of EC is a single value, and the output size is 𝑘 𝑠 × 𝑘 𝑠 × 𝑐ℎ 𝑜𝑢𝑡 , where 𝑘 𝑠 denotes equivalent kernel size and 𝑐ℎ 𝑜𝑢𝑡 denotes the number of output channels. This is exactly opposite to the vanilla convolution. In this way, the LUT size becomes:2 𝑏 1 × 𝑘 𝑠 × 𝑘 𝑠 × 𝑐ℎ 𝑜𝑢𝑡 × 𝑏 2 bit (1)where 𝑏 1 denotes bit width of feature map, 𝑏 2 denotes bit width of the value stored in LUT. We no longer need sampling, interpolation operations and rotation techniques while improving the cache hit rate. Moreover, we perform the EC operation in the LR space and transform to the HR space by directly splitting each value of the input features into 𝑟 × 𝑟 values. Finally, we aggregate these values along the channel axis to compress them back to the original lowdimensional RGB space. In summary, our contributions can be summarized as follows:• We propose ECNN, a novel LUT-based approach for SR task. By reducing the number of single indexed pixels to just one, ECNN eliminates the interpolation and rotation tricks, thereby reducing the LUT size and breaking the inference speed bottleneck of the SR-LUT series methods. • We further extend feature fusion to the feature channel dimension to enhance mapping ability. The smaller LUT size and multi-channel processing capability enable us to stack EC layers to construct ECNN, which significantly improves SR quality.• Extensive experiments show that our method improves the overall performance of the upper limit of LUT based methods. For example, under comparable SR quality conditions, our model achieves state-of-the-art performance in terms of speed and LUT volume. 
 
 Appendix:  


Decision: The final ratings are 1 weak accept, 1 borderline accept, and 1 borderline reject. The reviewers found that the paper is well written and the results are promising. Overall, the paper seems to have merits for acceptance. Nevertheless, there is a recurring question on clarification of the 1x1 conv vs. receptive field; it is suggested that the authors elaborate to resolve the issue.

Reviews: review: paper_summary: paper_summary: This paper presents a novel expanded convolutional neural network (ECNN) that enhances feature fusion in the channel dimension and reduces the number of single indexed pixels, significantly decreasing the LUT size and improving inference speed. Experimental results demonstrate that this method achieves state-of-the-art performance in both speed and LUT volume under comparable SR quality conditions.
strength_weakness: strength_weakness: 1.This paper is well-written with a complete structure, and I did not observe any significant grammatical or spelling errors.

2.The authors have taken a practical approach, inspired by reference [30], identifying issues with SR-LUT, and proposing a new method, ECNN, to address the limitations related to "SR quality".

3.The introduction is logically structured. The authors progressively approach the core problem from an application perspective.

4.The authors have validated the rationale of their quantization technique through extensive testing and ablation studies, confirming the effectiveness of ECNN.
limitations: limitations: 1.Despite the many strengths of this paper, there are areas for improvement. Firstly, regarding motivation, from line 125, the authors introduce the main problem they are addressing but fail to explain or justify these challenges. For example, the authors mention that the rotation technique in EC-LUT is "without conferring any significant advantage." However, in my understanding, rotation is a data augmentation technique that helps the model learn features from different directions, which is independent of the network structure itself. Are the authors implying that in the context of LUT methods, it increases overhead? I did not find an explanation or validation of the negative impacts of rotation in the paper. Additionally, I noticed that by line 281, the discussion of rotation changes from offering no advantage to having a "negative impact," which further confused me.

2.This paper’s references to prior work enhance its technical foundation. However, as a reviewer, I do not recommend dedicating too much space to describing previous work. The authors state in the abstract that this work is an incremental improvement based on EC-LUT, which naturally leads readers to look for distinctions between the two. I can see the authors' efforts in this regard, as they have used several paragraphs in Sections 1, 2, and 3 to introduce EC-LUT. After reading the paper, even without a detailed understanding of EC-LUT, I have a clear grasp of its motivation. The balance between receptive field size (affecting performance) and LUT size is crucial for LUT-based SR techniques, and the introduction of EC mitigates the performance impact of reducing the receptive field to 2x2. Since the motivation for previous work is already well established, it seems that an excessive introduction of prior work reduces the space available to highlight the novelty of the proposed method.

3.The paper mentions reducing the convolution size to 1x1 to decrease the LUT size to the KB level. Based on my knowledge, 1x1 convolutions are typically used for dimensionality reduction of feature maps and have weaker feature extraction capabilities compared to larger kernels, which is why previous LUT work used 3x3 or 2x2. The technique illustrated in Figure 3 seems similar to the PixelShuffle upsampling method, but with the deconvolution kernel replaced by LUT and multiple output channels summed. This makes ECNN's EC part resemble upsampling with pixel-level fusion, which indeed reduces LUT size, but what are the trade-offs? I am uncertain about the reliability and effectiveness of this approach.

4.The advantage of LUT-based SR methods lies in converting computational tasks of CNNs into LUT queries. This paper appears to retain the complete CNN and replaces a single convolution process with LUT. This significant change requires more detailed text to describe its motivation.

5.Although I am reluctant to overemphasize innovation, as it is difficult to quantify, there are concerns in this area. The key "E" (Expanded) in the proposed ECNN comes from another work in the same field (EC-LUT). Despite potential incremental improvements, the lack of a sufficiently distinct differentiation in the algorithm name from "Expanded Convolution" used previously is worrisome.
paper_summary: paper_summary: The paper proposes a novel super-resolution technique called expanded convolutional neural network (ECNN) to address the limitations of existing methods. The ECNN approach extends feature fusion to enhance mapping ability and reduces the LUT size, improving cache hit rates. The experiments demonstrate that the ECNN outperforms previous methods in terms of speed and LUT volume while achieving state-of-the-art performance in super-resolution quality. Overall, the paper presents a promising solution to the challenges faced inefficient super-resolution approaches.
strength_weakness: strength_weakness: - The proposed approaches reduce the LUT size effectively.
- The experiment results show that ECNN is more effective than previous approaches.
limitations: limitations: - More state-of-the-art approaches should be compared to as CARN (2016) , FSRCNN (2018) and RRDB (2018) are proposed quite a long ago.
- The paper would benefit from additional refinement and expanded content to provide a more thorough explanation of the ECNN.
- Many expressions require additional polish.
e.g. At this point, the size of LUTs has been significantly reduced from the GB or MB levels to the KB levels.
paper_summary: paper_summary: The authors propose an expanded CNN (ECNN) based look-up table for highly efficient single-image super-resolution. The authors extend feature fusion to the feature channel dimension to enhance mapping ability. In addition, the authors reduce the number of single indexed pixels to just one, reducing the LUT size from the MB level to the KB level. With the improvements, the authors stack expanded convolutional layers to form an ECNN, with each layer convertible to LUTs during inference.
strength_weakness: strength_weakness: 1) The authors propose ECNN, a novel LUT-based approach for SR task, by reducing the number of single indexed pixels to just one.
2) The authors extend feature fusion to the feature channel dimension to enhance mapping ability.
3) The proposed ECNN improves the overall performance of the upper limit of LUT based methods.
limitations: limitations: 1) Receptive field is important for the SR performance. The authors use expanded convolution with input window size 1x1. How do the authors implicitly improve the receptive field? Please explain, because 1x1 receptive field cannot result in good performance.
2) It is not clear how the authors up-sample the feature maps in the final layer. Fig. 2 and the explanation in Sec. 3.2 are confusing. How do the authors aggregate the feature maps along the channel axis to compress back to the low-dimensional RGB space?
3) The authors choose to quantize the LUTs to 12bit. However, since there is no 12-bit data type, is it better to quantize to 16 bits?