S$^2$-CSNet: Scale-Aware Scalable Sampling Network for Image Compressive Sensing

Authors: chen  hui
haiqi  zhu
shuya  yan
shaohui  liu
feng  jiang
debin s zhao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:41:01

Abstract: Deep network-based image Compressive Sensing (CS) has attracted much attention in recent years. However, there still exist the following two issues: 1) Existing methods typically use fixed-scale sampling, which leads to limited insights into the image content. 2) Most pre-trained models can only handle fixed sampling rates and fixed block scales, which restricts the scalability of the model. In this paper, we propose a novel scale-aware scalable CS network (dubbed S$^2$-CSNet), which achieves scale-aware adaptive sampling, fine granular scalability and high-quality reconstruction with one single model. Specifically, to enhance the scalability of the model, a structural sampling matrix with a predefined order is first designed, which is a universal sampling matrix that can sample multi-scale image blocks with arbitrary sampling rates. Then, based on the universal sampling matrix, a distortion-guided scale-aware scheme is presented to achieve scale-variable adaptive sampling, which predicts the reconstruction distortion at different sampling scales from the measurements and select the optimal division scale for sampling. Furthermore, a multi-scale hierarchical sub-network under a well-defined compact framework is put forward to reconstruct the image. In the multi-scale feature domain of the sub-network, a dual spatial attention is developed to explore the local and global affinities between dense feature representations for deep fusion. Extensive experiments manifest that the proposed S$^2$-CSNet outperforms existing state-of-the-art CS methods.

Main paper: 
 introduction: 
 Compressive sensing (CS) is an effective signal processing technique, which has generated significant research interest in the signal and image processing communities [3, 77] . Mathematically, supposing that 𝑥 ∈ R 𝑛×1 is an input signal, CS can achieve fast imaging by sampling far fewer measurements than that required by Nyquist sampling [10, 14, 31] , i.e., 𝑦 = Φ𝑥, where 𝑦 ∈ R 𝑚×1 are the observed measurements, Φ ∈ R 𝑚×𝑛 with 𝑚 ≪ 𝑛 is the sampling matrix and 𝑚 𝑛 is defined as the sampling rate (or CS ratio). Due to the simple and fast sampling, CS has been widely deployed including snapshot compressive imaging [40, 71] , medical imaging [54, 56] , image encryption [35] , and compressive learning [50, 66, 87] .In the development of image CS, block-based CS (BCS) [6, 19, 30 ] has emerged as a classical method and has been widely adopted by most research efforts [5, 29, 58, 80, 88] . In BCS, images are divided into non-overlapping blocks of fixed scale and sampled block by block. Following BCS, some representative sampling matrices have been proposed, including the structural matrix [8, 20] , the random matrix [63, 68] and the binary matrix [2, 44] . Corresponding to these sampling methods, some model-based reconstruction methods [13, 21, 86] have been presented, which usually utilize various iterative solvers to reconstruct images. However, due to extensive fine-tuning, these methods incur high computational costs.Driven by the powerful learning ability of deep neural networks, deep network-based CS has demonstrated superior performance compared to traditional sampling and reconstruction methods. In these works, they explore the use of fully connected layers [1, 42] and the convolutional layers [12, 39, 58] to model the sampling matrix, and propose network-based deep reconstruction methods. Some earlier studies adopt block-by-block reconstruction [34, 51, 72, 78] , which ignores the correlation between different blocks in the image, leading to serious blocking artifacts. To address this issue, some recent works [15, 58, 60, 84] feed all measurements of all blocks jointly into the reconstruction network and effectively eliminate the blocking artifacts. However, one of the weaknesses of these deep CS networks is that the sampling process ignores the image content and applies a spatially uniform sampling rate to the entire image. Considering that the meaningful information in an image is usually not uniformly distributed, some researchers have proposed adaptive CS [5, 53, 80, 88] , which can adaptively allocate the sampling rate according to the content features of the image.By treating each block independently and dynamically allocating the sampling rate, adaptive CS achieves content-aware sampling and provides further intrinsic insight into the image content. However, the existing adaptive CS methods suffer from the following issues: 1) Model scalability: Following [5, 57] , we define fine granular scalability as a model that can handle arbitrary CS ratios and allocate the CS ratio adaptively according to the image content. By mapping image content features to sampling rates, some adaptive CS [53, 88] employ a multi-channel solution with each channel handling a single sampling rate, which limits the scalability of the model. 2) Image prior computation: Some ideal solutions [80, 88] directly use the original image to calculate prior features for guiding the adaptive sampling, which is not always possible to access the complete image before CS sampling [5, 29] . 3) Block-based sampling: Existing adaptive CS schemes mechanically divide the image by a fixed scale, such coarse blocking leads to the mixing of information with different content features (e.g., texture regions and smooth regions), which will cause uneven sampling rate allocation and potentially degrade the performance of adaptive CS.To overcome above issues, we propose a novel scale-aware scalable CS network (S 2 -CSNet). Specifically, to enhance the scalability of the model, a structured learnable matrix with a predefined order is first designed, which can sample multi-scale image blocks with arbitrary sampling rates. Then, to address the issue of image prior computation, a distortion estimation method based on the universal sampling matrix is introduced, which estimates the real reconstruction distortion in the measurement domain. Then, a scale-aware sampling scheme is presented to achieve multi-scale sampling, which calculates the reconstruction distortion under different scale partitions of the same region and selects the partition mode with the minimum distortion for sampling. Finally, a multi-scale hierarchical sub-network under a well-defined compact framework is put forward to efficiently reconstruct the image. In the multi-scale feature domain, a dual spatial attention mechanism is suggested to explore the local and global affinities between dense feature representations for deep feature fusion.The main contributions are summarized as follows:(1) We propose a novel scale-aware scalable CS network S 2 -CSNet, which adopts a different approach from the traditional BCS and achieves scale-aware adaptive sampling and fine granular scalability without direct access to the original image.(2) In S 2 -CSNet, to achieve scale-aware sampling, a universal sampling matrix is designed, which can sample multi-scale image blocks with arbitrary sampling rates. Based on this sampling matrix, a distortion-guided scale-aware scheme is presented, which can evaluate the reconstruction distortion under multi-scale partitioning and select the optimal sampling scale.(3) A multi-scale hierarchical sub-network under a well-defined compact framework is put forward to reconstruct the image, in which a dual spatial attention mechanism is developed to explore the local and global affinities between dense feature representations to further enhance the reconstruction ability. 
related work 2.1 block-based image cs sampling: 
 As a lightweight sampling method, BCS effectively reduces computational complexity and is widely used in image CS [37, 80, 88] . In image BCS, the sampling process can be described as 𝑦 𝑖 =Φ 𝑠,𝐵 𝑥 𝐵 𝑖 , where 𝑥 𝐵 𝑖 denotes the 𝑖-th image block with spatial size 𝐵 and channel number 𝑙, Φ 𝑠,𝐵 ∈R 𝑚×𝑙𝐵 2 with 𝑚=⌊𝑠×𝑙𝐵 2 ⌋ is a predefined sampling matrix for sampling rate 𝑠. Based on the above idea, some researchers propose deep network-based CS methods [1, 5, 12, 29, 42, 58] , which use convolutional layers to represent the sampling matrix and achieve remarkable results. However, these methods are content-independent and cannot adaptively allocate the sampling rate according to the image content features.In adaptive CS, the sampling rate𝑠 𝑖 ∈ {𝑚/Q } Q 𝑚=1 with Q = 𝑙𝐵 2 ofan image block can be an arbitrary value between 0 and 1 [5, 29] . If non-adaptive CS is applied to solve this problem, e.g., [58, 76, 84, 88] , they need to train Q models to satisfy arbitrary sampling rates. To sample images at different sampling rates in a single model, some works propose to train multiple sampling matrices [62, 73, 76] . However, such an operation will impose a large storage burden, with a total memory cost of [5, 29] if Q sampling matrices need to be trained. To address this problem, some researchers divide the sampling matrix into different hierarchies to handle different sampling rates [43, 57, 83] . Based on these works, some recent CS methods [5, 29, 53, 80, 88] implement adaptive sampling rate allocation. In particular, [5] and [29] employ an auxiliary lightweight branch to compute prior features from measurements and train a universal sampling matrix to handle arbitrary sampling rates. Although these methods achieve adaptive sampling rate allocation, they all mechanically use fixed-scale blocking, which affects the performance of adaptive sampling.Q 𝑚=1 (𝑚 Q)=[ Q 2 ( Q+1)/2] ∈ O ( Q 3 )Recovering the original signal 𝑥 from a extremely small number of measurements 𝑦 is a classical ill-posed problem. The traditional CS method solves this problem in an optimized manner:EQUATIONwhere 𝛿 λ(𝑥) is a prior term with regularization parameter 𝛿. Based on Eq. ( 1), many effective methods have been proposed, such as gradient descent methods [16] , greedy methods [45, 67] , and convex optimization methods [70] . For image CS, more complex structures are designed, including prior-based methods [85] , projected Landweberbased methods [18] , and denoiser-based methods [55, 82] . However, these traditional methods require constant iteration, leading to a high computational complexity. Different from traditional CS methods, deep network-based CS exhibits greater potential. Specifically, early works [34, 74] often rely on block-based CS, where the target image is reconstructed block by block, and then all reconstructed image blocks are concatenated together to form the final image. However, connecting all the blocks in such a coarse manner will lead to serious blocking artifacts, especially at low sampling rates [27, 58] . To solve this problem, some post-processing methods [34, 42, 74] F the blocking artifacts. In addition, some literatures [7, 57, 58, 63] attempt to explore global deep prior, which concatenates all measurements into a complete image and feeds it into a well-designed network for optimization. Recently, some deep unfolded networks (DUNs) have been applied to image CS to provide better theoretical basis and inferential interpretability [5, 60, 61, [78] [79] [80] 84] . Specifically, DUN usually unfolds certain optimization solvers into deep network forms [4, 59, 69, 75] , such as multi-scale block CS (MS-BCS) algorithm [18] , proximal gradient descent (PGD) algorithm [52] and approximate message passing (AMP) algorithm [11] . In general, none of the aforementioned deep network-based image CS methods consider improving performance at the scale-aware sampling level, which leads to limited insights into the image content.i i s B i y ¡ F ( ) D X ) Reshape + Concat ( ) X RD Map i i B B í i B B i i i i B B B B i i i i Reshape i B i x 1.0, i i B B i x F Block Measurements2 2 i i B B 2 2 i i B B 2 2 2 2 i i i i B B B B 2 2 2 2 i i i i ' , i i s B M F F i i B B í m , i i s B F T , i i s B3 Proposed Method 3.1 Overview of S 2 -CSNet Fig. 1 shows the algorithmic workflow and the main structure of S 2 -CSNet. Considering the diversity of image contents, a scaleaware adaptive partitioning (SAP) sub-module divides the image into sub-blocks of multiple scales. Specifically, in SAP, to handle the input image block with arbitrary possible sampling rates and different scales, a universal sampling matrix is first designed, which is a structural sampling matrix with a predefined order. Based on this sampling matrix, a distortion-guided scale-aware scheme is presented, which can evaluate the reconstruction distortion (RD) of the image under different scales of sampling and select the optimal partitioning manner. In the sampling sub-module, image blocks of different scales are sampled with arbitrary assigned sampling rates to obtain the compressed measurements (see Eq. ( 2)). In the reconstruction sub-module, an initial reconstruction sub-network and a multi-scale hierarchical deep reconstruction sub-network with a dual spatial attention mechanism (see Fig. 3 ) are proposed to reconstruct the original image from the sampled measurements. 
scale-aware scalable sampling: 
 This section describes the implementation details of the scale-aware scalable sampling. Specifically, given the input image 𝑋 and the overall sampling rate 𝑅, it will be divided into blocks of multiple scales, i.e., { (𝑥𝐵 1 1 ,𝑠 1 ),•••,(𝑥 𝐵 𝑖 𝑖 ,𝑠 𝑖 ),•••,(𝑥 𝐵 𝐾 𝐾 ,𝑠 𝐾 ) },where 𝑠 𝑖 is the sampling rate assigned to each block based on the reconstruction distortion prior and 𝐾 is the total number of blocks.1) Design of the universal sampling matrix. In scale-aware sampling, a specific-size sampling matrix Φ 𝑠 𝑖 ,𝐵 𝑖 ∈R 𝑚 𝑖 ×𝑙𝐵 𝑖 2 is designed to perform the sampling operation for image blocks 𝑥 𝐵 𝑖 𝑖 with scale 𝐵 𝑖 and sampling rate 𝑠 𝑖 . In order to handle arbitrary sampling rates, the non-adaptive BCS methods need to train 𝑙𝐵 2 sampling matrices [5, 29] . If there are 𝑞 variable scales, then 𝑞𝑙𝐵 2 sampling matrices are required to realize S 2 -CSNet. To reduce the number of sampling matrices, like [5, 29, 80] , we propose to design a universal sampling matrix Φ 1.0,B ={Φ 1.0,𝐵 1 ,Φ 1.0,𝐵 2 ,...,Φ 1.0,𝐵max } to handle multiple scales B={𝐵 1 ,𝐵 2 ,••• ,𝐵 max } and arbitrary sampling rates. For a block 𝑥 𝐵 𝑖 𝑖 , the sampling process can be formulated as follows:EQUATIONwhere 𝑀 𝑠 𝑖 ,𝐵 𝑖 ∈R 𝐾 ×𝑙𝐵 𝑖 2 is a measurement mask used to extract the measurements at sampling rate 𝑠 𝑖 (details are shown in Fig. 1 ), ⊙ is the element-wise product, and G is a sampling rate allocation function. It should be noted that G is not required for the original image to guide the sampling rate allocation, which is related to Eq. ( 4) in Section 3.2.2, i.e., G allocates the sampling rate based on the distortion calculated by Eq. ( 4). Furthermore, to fully train the sampling matrix to handle arbitrary sampling rates, we define a decreasing trend of measurement base (i.e., each row of the matrix) importance from the first row to the last row [5] . In terms of handling sampling at multiple scales, the different sub-sampling matrices in Φ 1.0,B are restricted to satisfy different scales of sampling by solving the following optimization problem: arg minEQUATIONwhere 𝐼 and 𝐷 are the initial and deep reconstruction, respectively, 𝑁 𝑠 is the number of samples and 𝑋 𝑗 is a validation sample.2) Scale-aware adaptive sampling. The difficulty of CS reconstruction varies depending on the image content [29] , while existing BCS mechanically divides the image by a fixed scale, which leads to the mixing of information with different content characteristics and affects the performance of CS sampling. To solve this problem, we propose a scale-aware sampling strategy, which can efficiently distinguish the image content according to the RD prior of the image. In this subsection, we will describe both the calculation of the RD prior and RD-based scale-aware sampling. Calculation of the RD prior: During the actual sampling process of CS, we cannot fully acquire the input image in some cases [5, 29] . In order to obtain the content properties of the original image, some ideal calculation methods, such as [80, 88] , which employ multiple detectors to compute the salience directly on the original image. Different from these methods, inspired by [29, 53] , we propose a distortion estimation method in the measurement domain that only uses the sampling matrix to perfom linear operations on the measurements, which can be defined as:EQUATIONwhere x𝐵 𝑖 𝑖 is the initial reconstruction recovered from the measurements 𝑦 𝑖 of the image block 𝑥 𝐵 𝑖 𝑖 (see Section 3.3 for details), Φ 𝑅 𝑏𝑠 ,𝐵 𝑖 is a submatrix truncated from Φ 1.0,𝐵 𝑖 with a matrix mask 𝑀 ′ 𝑅 𝑏𝑠 ,𝐵 𝑖 , 𝑅 𝑏𝑠 is the base sampling rate, which is usually a portion of 𝑅. Δ𝑦 𝑖 𝐵 𝑖 is the estimation of the real reconstruction distortion of the image block 𝑥 𝐵 𝑖 𝑖 , and || • || is the absolute operator. It is worth noting that the sampling of S 2 -CSNet consists of two parts, the base sampling for Eq. ( 4) and the scale-aware sampling for Eq. ( 2). In Eq. ( 2), if a mask 𝑀 ′ is used to generate the sub-matrix Φ 𝑠 𝑖 ,𝐵 𝑖 for sampling, S 2 -CSNet will require two times of sampling operations. Instead, Eq. (2) only requires one sampling operation, and the measurements are used for both the base sampling and scale-aware sampling. To verify the accuracy of predicting ||Δ𝑥 || from ||Δ𝑦||, Fig. 2 shows the correlation [26, 28] and the statistical diagram of these two variables. It can be observed that they are approximately linearly correlated, i.e., the bigger the ||Δ𝑥 ||, the bigger the corresponding ||Δ𝑦||. Therefore, it is plausible to deduce ||Δ𝑥 || from ||Δ𝑦||.Scale-aware sampling: As shown in Fig. 1 , scale-aware scalable sampling consists of two steps: scale-aware partitioning and adaptive block sampling. Given an image 𝑋 ∈R H×W , and a group of sampling scalesB={𝐵 1 ,𝐵 2 ,••• ,𝐵 max } with 𝐵 1 <𝐵 2 <•••<𝐵 max ,we first compute the RD for each sampling scale 𝐵 𝑖 based on Eq. ( 4):EQUATION)where Ψ is a repeat operation that expands the computed distortion value for each block into a two-dimensional matrix R 𝐵 𝑖 ∈R (H×W) by filling in the distortion values at the corresponding locations of 𝑋 . Then, the image 𝑋 is uniformly divided into multiple blocks{𝑥 𝐵max 1 ,𝑥 𝐵max 2 ,••• ,𝑥 𝐵max 𝐾} based on the largest scale 𝐵 max . In the aggregation process, a recursive comparison function Γ will compare the distortion values of different scales in each sub-block 𝑥 𝐵max 𝑖 and return the scale 𝐵 𝑠 with the minimum distortion:EQUATIONSpecifically, Γ compares all division schemes under sub-block 𝑥 𝐵max 𝑖 , i.e., 𝑥 𝐵max 𝑖 is partitioned into multiple sub-scales from 𝐵 1 to 𝐵 max -1 . Next, Γ chooses the scale 𝐵 𝑠 in B with the minimum distortion, i.e.,| |Δ𝑦 𝐵𝑠 𝑖 | |< | |Δ𝑦 𝐵 𝑗 𝑖 | |,where 𝐵 𝑗 is an arbitrary scale and 𝐵 𝑗 ≠ 𝐵 𝑠 . Taking the scale B={𝐵 1 =16,𝐵 max =32} as an example, the image is uniformly divided into 32×32 sub-blocks according to the maximum scale 32. For each sub-block 𝑥 𝐵 max 𝑖 , we compare whether to further divide it into 4 sub-blocks in a 16×16 manner or to keep the size of 32×32 to minimize the RD. The RD has been calculated using Eq. ( 4), and is recorded in the RD maps R 𝐵 1 and R 𝐵 max . Therefore, it is sufficient to extract the distortion values at the position of 𝑥 𝐵 max 𝑖 from the R 𝐵 𝑖 of the corresponding scale. Then, Γ will compare the distortions under all division modes (i.e., 𝐵 1 and 𝐵 max ) of 𝑥 𝐵 max 𝑖 and choose a scale with the smallest distortion as the optimal division scale. Finally, we mark the corresponding distortion value to the position of 𝑥 𝐵max 𝑖 in RD map R, and perform the CS sampling 𝑦 𝑖 =Φ 𝑠 𝑖 ,𝐵𝑠 𝑥 𝐵max 𝑖 .1) Initial reconstruction. For the sampled measurements 𝑦 𝑖 ∈R 𝑚 𝑖 ×1 of an image block 𝑥 𝐵 𝑖 𝑖 , there are usually two stages of reconstruction, including an initial reconstruction and a deep reconstruction [5, 58, 88] . Specifically, 𝑦 𝑖 will first be upsampled (i.e., Φ ⊤ 𝑠 𝑖 ,𝐵 𝑖 𝑦 𝑖 ) to produce a 𝐵 𝑖 2 ×1 vector. Next, a reshape operation (Υ) is used to transform all vectors into 𝐵 𝑖 ×𝐵 𝑖 tensor blocks (i.e., x𝐵 𝑖 𝑖 ). Fig. 1 illustrates the flow of the detailed initial reconstruction for x𝐵 𝑖 𝑖 , which can be summarized as x𝐵 𝑖 𝑖 =Υ (Φ ⊤ 𝑠 𝑖 ,𝐵 𝑖 𝑦 𝑖 ). At last, all tensor blocks are concatenated to output the complete reconstructed image X .2) Dual spatial attention-based deep reconstruction. Since X is a coarse reconstruction, a dual spatial attention-based hierarchical network is proposed to finely optimize X . The reconstruction process for X consists of two parts of inputs, one is the basis measurements 𝑦 𝑏𝑠 used to compute the RD (see Section 3.2.2), and the other is the remaining measurements 𝑦 𝑒𝑠 from the adaptive sampling. The overall measurements 𝑦 are composed of 𝑦 𝑏𝑠 and 𝑦 𝑒𝑠 , and with the exploitation of RD information [5] , the reconstruction process ℏ(𝑦) can be represented as:EQUATIONwhere 𝜀 is a feature extractor consisting of several convolutions and residual blocks [5] , | is the operation that concatenates 𝐼 (•) and 𝜀 (•), and ∪ denotes the union of sets. Fig. 3 illustrates the overall framework of the deep reconstruction network, which involves a series of horizontal and vertical branches to construct a hierarchical grid architecture [7, 17, 29] . Specifically, the horizontal branch consists of multiple deep reconstruction blocks (DRBs), which are responsible for feature extraction and local and global affinity mining in a certain scale space. The vertical branch is composed of upsampling and downsampling submodules, which are used to integrate the intermediate feature maps of different horizontal branches. For a DRB positioned in the 𝑖-th row and 𝑗-th column, its output feature map ⌣ 𝑥 𝑖,𝑗 can be summarized as the following two equations:EQUATIONEQUATIONwhere Θ 𝐷𝑅𝐵 , Θ 𝐷𝑆 and Θ 𝑈 𝑆 denote the mapping operations of DRB, downsampling submodule and upsampling submodule, respectively, and 𝜃 is the trainable parameter. It should be noted that Eq. ( 8) mainly integrates downsampled features and Eq. ( 9) mainly integrates upsampled features, e.g. they can calculate the outputs 𝑖,𝑗 and 𝐹 𝑔𝑓 𝑖,𝑗 respectively to deeply fuse local and global features. Specifically, for the inputs Q and K of SA, a 3×3 convolution is first used to extract the feature to generate Q'∈R H×W×(C/2) and K'∈R H×W×(C/2) . To get the input tokens { Q, K, Ṽ}, a global pooling is applied to reduce the dimensionality of Q' to get Q. For K', it is flattened into the dimension of HW×(C/2) to obtain K. In addition, the input V∈R H×W×C is left unchanged to generate Ṽ.F G T Matrix Multiplication H W C H W C H W C H W C H W C H W C H W C/2 H W C/2 C/2 1 C/2 1 C/2 1 H W 1 H W 1 H W 1 H W C H W C ¡ Reshape ¡ LFEB H W C GFEB H W C C (g) DSAMNext, we use the softmax function to reweight Q⊤ and conduct matrix multiplication with K to generate the transposed attention map 𝑊 𝑖,𝑗 ∈R HW×1 , i.e., 𝑊 𝑖,𝑗 = K⊗Softmax 1 ( Q⊤ ). Here, 𝑊 𝑖,𝑗 actually represents the weight of the feature Ṽ, which is calculated from the query feature map Q and the key feature map K. To match the dimension of Ṽ, we reshape 𝑊 𝑖,𝑗 to size H×W×1 and use Softmax2D (i.e., Softmax 2 in Fig. 3 ) to generate the normalized weights W𝑖,𝑗 . The weighted aggregation of W𝑖,𝑗 and Ṽ can be computed as:EQUATION)In LSAM (i.e., (g.1) in Fig. 3 ) and GSAM (i.e., (g.2) in Fig. 3 ), the inputs Q, K and V of SA can be respectively defined as: where the feature map K is a rough fusion of In the training of S 2 -CSNet, the learnable parameters include the sampling matrix Φ 1.0,B and the reconstruction network. Given the training set {𝑋 𝑗 } 𝑁𝑝 𝑗 =1 , we employ the following ℓ 2 -loss like [58] to train S 2 -CSNet, which can be defined as:L (Φ 1.0,B , 𝜃 𝑑 ) = 1 𝑁 𝑝 ∑︁ 𝑁𝑝 𝑗 =1 ∑︁ 𝐵max 𝐵 𝑖 =𝐵 1 | | F (Φ 𝑠 𝑖 ,𝐵 𝑖 𝑋 𝑗 , 𝑛 𝑗 , 𝜃 𝑑 ) -𝑋 𝑗 | | 2 2 (14)where 𝜃 𝑑 is the parameter of the reconstruction network, 𝑁 𝑝 is the number of training images, 𝑛 𝑗 is a non-negative integer, and the output of F ( •) is the reconstructed image X𝑗 . In the input of F, 𝑛 𝑗 is a randomly generated integer from {1,2,••• ,Q ′ }, which is utilized to define the number of measurements, i.e., 𝑠 𝑗 =𝑛 𝑗 /Q ′ , whereQ ′ =𝑙𝐵 max 2 .In each training epoch, a random 𝑛 𝑗 is selected to train the ability of the sampling matrix to cope with arbitrary sampling rates [5] . Furthermore, during the optimization process, Eq. ( 14) ensures the training of sampling matrices for different scales in Φ 1.0,B by partitioning the image into different scales. 
experiments and analysis 4.1 implementation and training details: 
 In block-based CS methods [5, 58, 61, 80] , the block size is usually set to 16 or 32. Following these works, in the scale-aware sampling of the proposed S 2 -CSNet, we set the number of sampled scales to 2, i.e., B = {16, 32}. We implement S 2 -CSNet with PyTorch on two NVIDIA RTX 3060 GPUs, employ Adam [32] optimizer with a batch size of 20. S 2 -CSNet is trained with a learning rate of 1e-4 for 300 epochs, and as the number of epoch increases, the learning rate is multiplied by a factor of 0.5 every 50 epochs. The training set is the same as [5] , which contains 25600 randomly cropped 128×128 image patches from T91 [9] and Train400 [81] . For testing, we utilize two widely used datasets: Set11 [34] and CBSD68 [46] . The sampling rate allocation is similar to [88] , supposing that 𝜐 denotes the amount of distortion information embodied in image 𝑋 . One has 𝜐 = 1 𝑛 𝑗 ∈R 𝑙 𝑗 , where 𝑛 is the total number of pixels on image 𝑋 and 𝑙 𝑗 is the distortion value of location 𝑗 on RD map R. For block 𝑥 𝑖 of image 𝑋 , its sampling rate can be calculated as 𝑠 𝑖 = 𝜐 𝑖 𝜐 × 𝑅, where 𝜐 𝑖 represents the distortion information of 𝑥 𝑖 , and 𝑅 is the given sampling rate. In addition, two Swin Transformer Blocks [41] are used in the GFEB of the deep reconstruction network, and the number of multi-head self-attention block is set to 4, the shift distance is set to 4 and the window size is set to 8.1) Exploring the architecture of deep reconstruction network in Fig. 3 : We first explore the setting of the number of horizontal branches (𝑁 𝐵 ) and the number of DRBs (𝑁 𝐷 ) on each horizontal branch. Fig. 4 reveals the relationship between these two hyperparameters and the image reconstruction quality, from which it can be seen that as these two hyperparameters increase, the reconstruction quality becomes less and less sensitive to them. Therefore, we set 𝑁 𝐵 = 4 and 𝑁 𝐷 = 4. Correspondingly, the number of sets of vertical branches is set to 5, including 3 sets of downsampling branches and 2 sets of upsampling branches as displayed in Fig. 3 . To simplify the model and reduce complexity, we apply DSAM only in the first horizontal branch. On this branch, each DRB contains a set of FEM and DSAM, with the number of RBs in FEM set to 6. The DRBs of the remaining horizontal branches consist of RBs, and the number of RBs contained in the DRBs of the different branches is set to 2, 3 and 4, respectively. In addition, we set the number of input and output channels of DRB to 64.2) Setting of the base sampling rate to acquire the RD: Scale-aware partitioning relies on the RD prior, in Eq. ( 4), we set a small base sampling rate 𝑅 𝑏𝑠 to compute the reconstruction distortion of the image. 𝑅 𝑏𝑠 is part of the overall sampling rate 𝑅, i.e., 𝑅 𝑏𝑠 = 𝛾 × 𝑅. To test the image reconstruction quality under different proportion coefficients 𝛾, Fig. 6 shows the average PSNR curve on Set11 and CBSD68 at seven sampling rates. It can be observed that when 𝛾 approaches 0 or 1, S 2 -CSNet degrades into a uniform sampling version due to the lack of prior computation or sufficient space for adaptive sampling rate allocation. Therefore, we set 𝛾 = 0.15, where it reaches the peak of reconstruction quality.3) Analysis of the learned universal sampling matrix Φ 1.0,B : Like [5, 29] , we analyze the properties of the learned sampling matrix from three aspects, including: orthogonality, coefficient distribution, and frequency view. For orthogonality: It can be seen from Fig. 8 that the property ΦΦ ⊤ = 𝜇I is approximately satisfied for both 𝐵 = 16 and 𝐵 = 32 sampling matrices without additional constraints, where I is the identity matrix. For coefficient distribution: We show the histograms of the learned matrices and Gaussian random matrices in Fig. 8 . It can be observed that the learned matrices exhibit a wider and sparser distribution. For the frequency view: we reshape one row of the sampling matrix to 16 × 16 and 32 × 32, respectively, and present their spatial and frequency domain views. Fig. 9 shows the views of the first ten rows of the learned sampling matrices, indicating that they exhibit structured and anisotropic spatial patterns different from traditional manually defined filters. In addition, the rows at the front of the sampling matrix have narrower frequencies, which implies that they pay more attention to low-frequency information at low sampling rates.1) Overall Comparisons: In Table 3 , we compare S 2 -CSNet with thirteen representative state-of-the-art CS algorithms. Here, we categorize these methods into two groups: CS with adaptive CS ratio allocation (ACRA) and CS with fixed CS ratio allocation. For CS based on ACRA, such as BCSNet [88] , AMSNet [80] , and CASNet [5] , they adaptively allocate sampling rates based on the content of the image. However, methods like [88] and [80] require the original image to compute prior features to guide the allocation of sampling rates. Furthermore, BCSNet [88] only implements a limited number of sampling rate allocations, lacking scalability. CASNet [5] employs a universal sampling matrix to organically achieve both model scalability and ACRA. Overall, there is currently no algorithm that considers scale-aware adaptive sampling.2) Comparisons of Reconstruction Quality: Visual quality is an important measure of the algorithm [22-25, 38, 47-49] . In Table 1 and Table 2 , we present the comparison of different image CS methods on the Set11 and CBSD68 datasets in terms of PSNR and SSIM metrics. Compared with representative non-ACRA methods (CSNet + [58] , AMP-Net [84] and CAT-Net [33] ), a) on the dataset Set11, the proposed S 2 -CSNet achieves on average 2.62dB, 1.51dB, 0.40dB and 0.0358, 0.0239, 0.0089 gains in PSNR and SSIM compared against these three methods at the given sampling rates. b) On the dataset CBSD68, our proposed algorithm achieves on average 0.76dB, 0.69dB, 0.51dB and 0.0265, 0.0303, 0.0123 gains in PSNR and SSIM under different sampling rates. Compared with representative ACRA methods (AMSNet [80] and CASNet [5] ), a) On the dataset Set11, the proposed S 2 -CSNet achieves on average 0.47dB, 0.49dB and 0.0207, 0.0028 gains in PSNR and SSIM compared with these ACRA methods under the given sampling rates. b) On the dataset CBSD68, our proposed method achieves on average 0.16dB, 0.10dB and 0.0099, 0.0011 gains in PSNR and SSIM in terms of different sampling rates. The visual comparisons in Fig. 5 shows that our S 2 -CSNet is able to recover high-quality results with more details.3) Comparisons of Complexity: The computation cost and model parameters are important in many practical applications [36, 64, 65] . To verify the efficiency of the proposed S 2 -CSNet, in Table 3 , we compare the number of parameters of different CS methods and the speed of sampling and reconstructing a 256×256 image at sampling rates 𝑅 ∈ {0.01, 0.04, 0.10, 0.25, 0.30, 0.40, 0.50} on a 1080Ti GPU. Due to the additional prior computation and adaptive sampling rate allocation required by ACRA-based CS (e.g., AMSNet [80] , CASNet [5] ), their speed is generally slower than non-ACRA-based CS (e.g., CSNet + [58] , AMP-Net [84] ). Compared with recent ACRA-based methods (AMSNet [80] and CASNet [5] ), the proposed S 2 -CSNet basically maintains a similar complexity with them. Specifically, S 2 -CSNet has fewer parameters than CASNet [5] but higher than AMSNet [80] . In terms of running speed, S 2 -CSNet is faster than AMSNet [80] and CASNet [5] .1) Effect of Scale-Aware Adaptive Partitioning (SAP): Fig. 7 displays some examples of scale-aware sampling, and it can be seen that the proposed S 2 -CSNet can effectively distinguish image contents and achieve finer sampling rate allocation. Specifically, we observe that S 2 -CSNet tends to use large-scale blocks for sampling in content-consistent regions such as the background, which will allow the convolution kernel to obtain a broader field of view in convolution-based sampling. In regions with complex content such as boundaries, small sampling scales can separate different content features more efficiently. Furthermore, in Fig. 9 , we note that the frequency domain distribution of the sampling matrix with scale 16 × 16 is wider than that of the sampling matrix with scale 32 × 32, which suggests that the sampling matrix with scale 16 × 16 is more adept at processing high-frequency information. This is consistent with the example in Fig. 7 , i.e., in high-frequency regions with complex content such as texture, S 2 -CSNet will use small-scale sampling to achieve better reconstruction results. In contrast, in regions with simple and consistent content, S 2 -CSNet prefers to use large-scale sampling to reconstruct low-frequency information. In Table 4 , we present the reconstruction results of S 2 -CSNet without the SAP module, including fixed-scale sampling with 𝐵 = 16 (1st row) and 𝐵 = 32 (2nd row). It can be seen that SAP can bring average PSNR gains of 0.33dB and 0.39dB for S 2 -CSNet compared with the above two fixed-scale sampling.2) Effect of Adaptive CS Ratio Allocation (ACRA): As one of the main ideas of S 2 -CSNet, the allocation scheme based on reconstruction distortion can perform adaptive CS ratio allocation for blocks of different scales. Compared with the fixed CS ratio allocation scheme, ACRA can more effectively integrate the allocation of the sampling ratio into the content characteristics of the image and bring better reconstruction quality. The effectiveness of ACRA is also verified in Table 4 , from which it can be seen that the average PSNR under fixed CS ratio sampling is 0.44dB lower than the sampling under the ACRA strategy.3) Effect of DSAM in Deep Reconstruction Network: Compared with the native multi-scale reconstruction network [7, 29] , the introduction of DSAM aims to enhance the mining of local and global features of the deep reconstruction network. In Table 4 , we compare the native multi-scale network (i.e., composed of multiple sets of cascaded residual blocks) with our DSAM-based multi-scale network. It can be observed that DSAM can bring about 0.26dB gain compared to the non-DSAM strategy. 4) Effect of RD Map R: In the process of deep reconstruction, we introduce the reconstruction distortion (RD) map (see Fig. 3 ) to perceive the distribution of distortion information. The RD map contains the reconstruction distortion of each block, which is used to guide the CS ratio allocation of S 2 -CSNet. As shown in Table 4 , the introduction of RD map results in an average PSNR gain of 0.15dB compared to inputting only the initial reconstructed image in the deep reconstruction. 
conclusion: 
 In this paper, by analyzing the drawbacks of block-based sampling, a scale-aware scalable network (dubbed S 2 -CSNet) for image compressive sensing is proposed, which achieves scale-variable adaptive sampling and fine granular scalability without direct access to the original image. Specifically, to adapt to multi-scale adaptive sampling, a structural sampling matrix with a predefined order is presented, which can sample multi-scale image blocks with arbitrary sampling rates. Then, we design a distortion-guided scale-aware scheme, which is used to evaluate the reconstruction distortion under different scale samplings and select the optimal scale for sampling. Furthermore, to reconstruct the measurements with high quality, a multi-scale hierarchical sub-network is developed, in which a dual spatial attention is embedded to mine the local and global affinities between dense feature representations. Extensive experiments on both objective metrics and subjective visual qualities demonstrate that the proposed S 2 -CSNet outperforms existing state-of-the-art CS methods by large margins. 
 
 Appendix:  


Decision: 1 Accept, 1 Weak Accept, 2 Borderline Reject.
All reviewers provide positive comments to this paper. However, several mineral issues are to be addressed in the final version.
-- how to generate RD map $\mathcal{R}$
-- What and how to determine the predefined order
-- how the Scale-aware scalable sampling module works
-- differences and contrasts between this work and existing works
-- detailed example demonstrating how this framework could be applied in practical applications
-- more experiments

Reviews: review: paper_summary: paper_summary: This paper proposes a novel scale-aware scalable CS network, dubbed as $S^2$-CSNet, which achieves scale-aware adaptive sampling, fine granular scalability and high-quality reconstruction with a single model. The proposed $S^2$-CSNet makes use of a universal sampling matrix to achieve scalable sampling. It also makes use of a distortion-guided scale-aware scheme to select the optimal division scale for sampling. In $S^2$-CSNet, a multi-scale hierarchical sub-network is designed to reconstruct the image. The multi-scale hierarchical sub-network uses a dual spatial attention mechanism to mine the local and global affinities between dense feature representations. Extensive experiments on both objective metrics and subjective visual qualities demonstrate that the proposed $S^2$-CSNet outperforms existing state-of-the-art CS methods by large margins.
strength_weakness: strength_weakness: The proposed $S^2$-CSNet is the first deep network-based image CS that provides scale-aware adaptive sampling and fine granular scalability without direct access to the original image.
limitations: limitations: It is unclear how to generate RD map $\mathcal{R}$, which is an important input of deep reconstruction sub-module in the proposed $S^2$-CSNet. 

According to this paper, the universal sampling matrix is a structural sampling matrix with a predefined order. What does the predefined order mean? How to decide this predefined order?

It is a little difficult to understand how the Scale-aware scalable sampling module works. The presentation of Sec 3.2 should be improved. Moreover, in Sec 3.2, Some formulas are incorrectly referenced, such as, at line 375-376, "the sampling of $S^2$-CSNet consists of two parts, the base sampling for Eq. (4) and the scale-aware sampling based on Eq. (4)."
paper_summary: paper_summary: This paper proposes a novel scale-aware CS network (dubbed S$^2$-CSNet), which achieves scale-aware adaptive sampling, fine granular scalability and high-quality reconstruction with one single model. However, the novelty is limited.
strength_weakness: strength_weakness: The paper indicates a completed story.
limitations: limitations: 1)	Lns 38 and 39: What is the difference between "image compression" and "Image compressive sensing" in KEYWORDS?
2)	Notation used in the paper is very confused. Some expressions in this paper are ambiguous. Ln338: “Np is the number of samples”, Ln553: “Np is the number of training images”. Ln554: “  represents the reconstructed image  ”. Ln 189: “Q= lB2”,  Ln497: “the query feature map Q”, Ln487: “for the inputs Q and K”, Ln489: “To get the input tokens {˜Q,˜K,˜V}”. SA takes the query feature map Q as input, and the inputs Q indicates to query feature map or Token. It's hard to understand. What is the difference between the output of Eqs. (8) and (9)? Ln457: “e.g. they can calculate the outputs   and   of the DRBs, respectively.” This description is inconsistent with Eqs. (8) and (9). Figure 9: “first ten rows (1-10)”?
3)	Lns 383-386: “It can be seen that they are nearly linear relationship, i.e., the larger the ||Δx||, the larger the corresponding ||Δy|. Therefore, it is reasonable to infer ||Δx|| from ||Δy||.” As defined in Eq.4: “we propose a distortion estimation method in the measurement domain that only uses the sampling matrix to perfom linear operations on the measurements.” “linear operations” is predefined. Please provide a detailed theoretical analysis of "it is reasonable to infer ||Δx|| from ||Δy||".
4)	Ln 405: “Γ will select the scale Bs with the minimum distortion”. Please explain in detail the specific operation and selection strategy of Γ at scale Bs.
5)    References [70] and [71] are repeated.
paper_summary: paper_summary: This paper presented a network for image compressive sensing, which mainly consists of three components. To be specific, a learnable matrix with a predefined order for mutli-scale sampling with arbitrary rates,  a distortion estimation method for guiding the sampling rate allocation, and a sampling scheme for completing multi-scale sampling.
strength_weakness: strength_weakness: 1. In contrast to existing methods, the proposed method could achieve both variable sampling sizes and rates.
2. The proposed method achieved the moderate performance improvements compared to the baselines in the paper.
limitations: limitations: 1. This work sets the number of scales to 2 (i.e., 16 and 32) by following the previous works. However, one of the advantages emphasized in the paper are handling multiple scales. The paper does not conduct experiments to verify the scalability of the method on 3 or more scales.
2. The proposed method handles multiple scales by respectively handling each of them, and selecting the scale with minimum reconstruction distortion. Such a way usually induces a poor scalability to model. How does the complexities of the model change with the number of block scales?
3. The differences and contrasts between this work and existing works were not clearly stated in the section of relate works. For example, what are the major differences between them in adaptive allocations of sampling rates? And why does the proposed strategy make sense and outperform the others?
4. Some typos exist in the paper, such as "reconstructiona".
paper_summary: paper_summary: This work introduces a novel approach named S2-CSNet for image compressive sensing (CS), addressing the two challenges of maximizing information preservation in linear CS measurements and accurately reconstructing original images. S2-CSNet utilizes a novel compressive sampling paradigm called scale-aware scalable sampling, where images are divided into non-overlapping blocks of sizes 16 and 32. These blocks are sampled using learnable universal sampling matrices in a self-adaptive, non-uniform manner while achieving the target sampling rate. A reconstruction distortion (RD) prior map, estimated from initial measurements, guides the adaptive CS ratio allocation in the subsequent sampling stage. The method employs a hierarchical grid network architecture that projects measurements back to the image domain and integrates them with the RD prior map for multi-scale reconstruction. Extensive experiments demonstrate that S2-CSNet achieves state-of-the-art performance across a wide range of sampling rates, validating the effectiveness of the proposed scale-aware adaptive sampling, dual spatial attention module (DSAM), and RD map.
strength_weakness: strength_weakness: 1. The scale-aware adaptive sampling method introduces fine-grained scalability and is a novel approach in the community. It leverages different block sizes to enhance image information preservation within compressed measurements, significantly improving the efficiency and effectiveness of the sampling process. Currently, no existing algorithms offer similar insights into adaptive image content sampling.

2. S2-CSNet significantly outperforms existing methods, with gains of 0.4dB and 0.1dB PSNR on the Set11 and BSD68 datasets, respectively. These gains are achieved while maintaining comparable complexity in terms of parameter number and inference time.

3. The detailed analysis of the learned universal sampling matrices, as shown in Figures 7-10, provides valuable insights into the structural properties and distribution of matrix elements. These findings are not only interpretable but also beneficial for further research in the field.
limitations: limitations: Despite my overall positive assessment of this work, I recommend addressing the following concerns to strengthen the manuscript:

1. The manuscript lacks a detailed discussion on the practical implementation of scale-aware adaptive CS sampling in real-world scenarios, such as single-pixel cameras, digital image encoding, or encryption. The authors should provide a detailed example demonstrating how this framework could be applied in practical applications to ensure its relevance and utility.

2. The description of the scale-aware sampling process is complex and challenging to follow. I suggest adding a structured "Algorithm" part for the sampling process and relocating detailed network block designs to the supplementary material. This adjustment will help focus the main text on the primary contributions, improving readability and comprehension.

3. The method shows a more significant performance advantage at lower CS ratios. An explanation of why this occurs would provide deeper insights into the method's efficiency and limitations at different compression levels.

4. To aid in understanding and reproducing the results, I recommend providing access to the code and pre-trained models. This is crucial given the complexity of the proposed sampling and reconstruction methods.

5. The current manuscript omits several relevant and recent works that provide additional context and comparison for the proposed method. These include:

[1] Deep Physics-Guided Unrolling Generalization for Compressed Sensing

[2] Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system

[3] Deep Memory-Augmented Proximal Unrolling Network for Compressive Sensing

[4] SAUNet: Spatial-Attention Unfolding Network for Image Compressive Sensing

[5] ISTA-NET++: Flexible Deep Unfolding Network for Compressive Sensing

[6] D3c2-net: Dual-domain deep convolutional coding network for compressive sensing