Selective Sensing: A Data-driven Nonuniform Subsampling Approach for Computation-free On-Sensor Data Dimensionality Reduction
Authors: Zhikang Zhang
Kai Xu
Fengbo Ren
Venue:  (2021)
Keywords: Compressive sensing
nonuniform subsampling
machine learning

Abstract: Designing an on-sensor data dimensionality reduction scheme for efficient signal sensing has always been a challenging task. Compressive sensing is a state-of-the-art sensing technique used for on-sensor data dimensionality reduction. However, the undesired computational complexity involved in the sensing stage of compressive sensing limits its practical application in resource-constrained sensor devices or high-data-rate sensor devices dealing with high-dimensional signals. In this paper, we propose a selective sensing framework that adopts the novel concept of data-driven nonuniform subsampling to reduce the dimensionality of acquired signals while retaining the information of interest in a computation-free fashion. Selective sensing adopts a co-optimization methodology to co-train a selective sensing operator with a subsequent information decoding neural network. We take image as the sensing modality and reconstruction as the information decoding task to demonstrate the 1st proof-of-concept of selective sensing. The experiment results on CIFAR10, Set5 and Set14 datasets show that selective sensing can achieve an average reconstruction accuracy improvement in terms of PSNR/SSIM by 3.73dB/0.07 and 9.43dB/0.16 over compressive sensing and uniform subsampling counterparts across the compression ratios of 4-32x, respectively. Source code is available at https://figshare.com/s/519a923fae8f386d7f5b

Under review as a conference paper at ICLR 2021
SELECTIVE SENSING: A DATA-DRIVEN NONUNIFORM SUBSAMPLING APPROACH FOR COMPUTATION-FREE ON-SENSOR DATA DIMENSIONALITY REDUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Designing an on-sensor data dimensionality reduction scheme for efficient sig- nal sensing has always been a challenging task. Compressive sensing a state-of-the-art sensing technique used for on-sensor data dimensionality reduc- tion. However, the undesired computational complexity involved in the sens- ing stage of compressive sensing limits its practical application in resource- constrained sensor devices or high-data-rate sensor devices dealing with high- dimensional signals. In this paper, we propose selective sensing framework that adopts the novel concept of data-driven nonuniform subsampling to reduce the dimensionality of acquired signals while retaining the information inter- est in computation-free fashion. Selective sensing adopts co-optimization methodology co-train selective sensing operator with a subsequent infor- mation decoding neural network. We take image as the sensing modality and reconstruction as the information decoding task to demonstrate the 1st proof-of- concept of selective sensing. The experiment results on CIFAR 10, Set5 and Set1 datasets show that selective sensing can achieve an average reconstruction ac- curacy improvement in terms of PSNR/SSIM by 3.73dB/0.07 and 9.43dB/0.16 over compressive sensing and uniform subsampling counterparts across the di- mensionality reduction ratios of 4-32x, respectively. Source code is available
INTRODUCTION
In the era of Internet-of-thing (IoT) data explosion (Biookaghazadeh et al. 2018). efficient in- formation acquisition and on-sensor data dimensionality reduction techniques are in great need. Compressive sensing is the state-of-the-art signal sensing technique that is applicable to on-sensor data dimensionality reduction. However, directly performing compressive sensing in the digital do- main as linear transformation of signals can be computationally costly, especially when the signal dimension n is high and/or a data-driven sensing matrixMousavi et al.] (2017) 2018); Lohit et al. (2018); Wu al. 2018) is used. To mitigate this problem, several approaches have been proposed to reduce the computational complexity of compressive sensing by constraining the sensing matrices be sparse, binary, or ternary (Wang al., 20166; Nguyen al. 2017 Zhao al.) 2018 Hong et al. 2019). While these approaches can reduce the computational complexity by a constant factor (O(nn)) ). where can be as low 10 10-2), such reduced computational complexity can be still too high to be affordable for eesource-constraiig al. 2018). or high-data-rate sensor devices dealing with high-dimensinn -igenhimmenion signals, e.g.. LiDAR
sensor devices, e.g., low-cost IoT sensors (Djelouat
and depth map (Chodosh et al. 2019). Other approaches (Duarte al. 2008 Robucci et al. 2010) propose to implement compressive sensing in the analog domain instead, eliminating or reducing the computation cost of compressive sensing through custom hardware implementation. However, such custom hardware implementation inevitably increases the cost of the sensor and is often specific to the sensor design, thereby cannot be generally applied to other sensors applications.
In this paper, we propose a selective sensing framework to address the above- mentiondd problem reduce the dimension-
by adopting the novel concept of data-driven nonuniform subsampling ality of acquired signals while retaining the information of interest in a computation fashion. Specifically, the data dimensionality reduction in selective sensing nonuniform subsampling (or

---

Under review as a conference paper at ICLR 2021
selection) process that simply selects the most informative entries of signal vector based an optimized, stationary selection index vector informed by training data. Since no computation is in- volved for any form of data encoding. the computational complexity of the selective sensing operator is simply O(1), leading to the computation-free data dimensionality reduction during the selective sensing process [] Selective sensing adopts co-optimization methodology to co-train selective sensing operator with a subsequent information decoding neural network. As the trainable param- eters of the sensing operator (the selection index) and the information decoding neural network are discrete- and continuous-valuue mixed
respectively, the co-optimization problem in selective sensing is a optimization problem that is inherently difficult solve. We propose a feasible solution to solve it by transforming the mixed cescrree-continuonn optimization problem into two continuous optimization subproblems through interpolation and domain extension techniques. Both of the subproblems can then be efficiently solved using madient-descent-bane take images as the sensing modality and reconstruction as the information decoding task to demon- strate the 1st proof-of-coneep of selective sensing. The experiments on CIFARIO 10, Set5 and Set14 datasets show that the selective sensing framework can achieve an average reconstruction accuracy improvement in terms of PSNR/SSIM by 3.73dB/0.07 and 9.43dB/0. 16 over compressive sensing and uniform subsampling counterparts across the dimensionality reduction ratios of 4-32x, respec- tively.
algorithms. We
The contributions of this paper are summarized as follows:
1. We propose a new on-sensor data dimensionality reduction method called selective sensing. Se- lective sensing efficiently reduces the dimensionality of acquired signals in computation-free fash- ion while retaining information of interest. The computation-free nature of selective sensing makes it a highly suitable solution for performing on-sensor data dimensionality reduction on resource- constrained sensor devices or high-data-rate sensor devices dealing with high -igh-dimensional signals.
2. We propose and apply the novel concept of duta-driven nonuniform subsampling, Specifically, we first formulate the problem of co-optimizing selective sensing operator with a subsequent in- formation decoding neural network as mixed discrete-continood optimization problem. Further- more, we propose a viable solution that transforms the problem into two continuous optimization subproblems that can be efficiently solved by gradient-descent-based algorithms, which makes the co-training feasible.
3. We empirically show that data-driven nonuniform subsampling can well preserve signal informa- tion under the presence of a co- co-trained information decoding network.
2
RELATED WORK
2.1 NONUNIFORM SUBSAMPLING
Model-based nonuniform subsampling has been proposed in Chepuri et al. (2016) in the name of sparse sensing. Sparse sensing requires a hand-crafted sparsity model of signal as prior knowledge. Differently, selective sensing requires no prior knowledge about the sparsity model of signal, as all the necessary information needed for reconstruction can be learned from data through the train- ing process. Therefore, selective sensing has a much broader range of applications, especially in IoT, than sparse sensing, considering a vast majority of IoT signals are not well studied nor under- stood yet, but huge amounts of loT data are already available for training and learning. Dadkhahi & Duarte 2014) proposes generate an image mask that can preserve the manifold structure pre- sented in image data. Differently, we focus on the task of single image sensing and reconstruction in this paper. [Balassaare et al. (2016); Weiss al. (2019); Gözeü et al. (2018); (ahadii et al.) (2019) 2020) propose to perform MRI image nonuniform subsampling in k-space (frequency domain). As many spatial-domain signals are much sparser in the frequency domain, e.g., natural images and MRI images, the existing nonuniform subsampling approaches performed in k-space are insufficient
'For temporal signals, the selection operation can be simply implemented in the digital domain with counter and a mux that already exists in the control logic of most sensors. We consider such operations as control rather than data computation as no data is computed during the selective sensing process. For spatial signals such as images, the selective sensing operator can also be implemented as a low-cost masked sensor array with no computation involved. In addition, addition,MMayberry et al. (2014); [Centeye] (2020) present image sensor architectures for embedded systems that can provide pixel-level control of image sensors.
2

---

Under review as a conference paper at ICLR 2021
for dealing with dense signals directly in the spatial domain. In addition, the complex computation of or the custom hardware Maccadeen et al. (017)) for implementing Fourier transformation required in these methods is a real-breaker for resource-constraing constrained sensor devices and/or high-data-rane sen- sor devices dealing with high-dimensional signals. Differently selective sensing works directly in the spatial domain and the selective sensing operators require computation upon the sensor data at all. Huijben et al.J (2019) propose to co-optimize a probabilistic subsampling mask and a sub- sequent task-specific neural network in an end-to-end fashion. The sensing mask is dynamically generated from a random distribution with respect to each signal. The computation of generating such masks is hardly affordable for anoource-constraing and/or high-data-rate sensor devices. Dif- ferently, selective sensing uses static sensing mask learnt through the co-training algorithm. Once a sensing mask depolyed to sensor devices, no computation is needed to update the existing mask. Therefore, selective sensing is extremely friendly to resource-constraieed or high-data-rate sensors.
2.2 SENSING MATRIX SIMPLIFICATION METHODS
The computational complexity of the linear transformation in compressive sensing is O( Zhao al. (2018) Hong al. (2019) proposes model-based methods to construct sparse sensing ma- trices. Wang et al. () Nguyen et al. (2017) propose data-driven methods to build binary or ternary sensing matrices. However, all these approaches could only reduce the computational com- plexity by constant factors, i.c. O(en²), where c can be as low as 10 10-3). A key differentiator of selective sensing is that by adopting the novel concept of data-driven non-uniform subsampling, is computation-free free and has computational complexity of (11).
2.3 DATA-DRIVEN COMPRESSIVE SENSING
Kulkarni et al. (2016) Mousavi & Baraniuk (017 Yao et al. 2019) propose to directly learn the in- verse mapping of compressive sensing through the training of reconstruction neural network models. In addition, MMusavi et al. (2017) 2018); Lohit et al. (2018): Wu et al. (2018) propose to co-train customized sensing scheme with reconstruction neural network to improve the reconstruction accuracy. It should be noted that such co-training algorithms are specific to the reconstruction net- work proposed in the corresponding literatures. To the best of our knowledge, there no general co-training algorithm that can be applied to various reconstruction networks in the domain of data- driven compressive sensing. These approaches inspire us to develop a framework that co-trains a selective sensing operator and a subsequent information decoding network. Co-trained signal sens- ing and reconstruction frameworks can be viewed as a specific type of autoencoders autoencoders GGodffelow eet al. 2016). The main difference between such frameworks and a general autoencoder model is that the sensing (encoder) part of such frameworks must be implemented on sensors for on-sensor data dimensionality reduction. Therefore, the computation complexity of the encoder has to be extremely low in order to be affordable for sensor devices.
2.4
IMAGE SUPER-RESOLUTIIO
The problem of beural-networf based image super- super-resolution has been studied in recent years et al. [017]] Dong et al., 2013) Yang et al. (2019). The image super-resolution task is fundamentally different from the image reconstruction task of selective sensing in following two aspects. First, images in super-resolution tasks are uniformly subsampled in the training phase, while images in selective sensing are nonuniformly subsampled. Therefore, the existing network structures for image super-resolution cannot be directly applied to perform the image reconstruction task in selective sensing. Second, the downsizing factor of images in super-resolution tasks is only up to 4x to the best of our knowledge in the existing literature. Differently, in selective sensing and reconstruction tasks, the nonuniformly subsampling factor (compression ratio) of images can have a much larger range (4-32x in this paper).
METHODOLOGY
In this section, we first formulate the co-optimization of selective sensing operator and subse- quent information decoding network as a mixed cescrete-oomtiuuuus optimization problem. Then, by applying continuous interpolation and domain extension the integer variables, we reformulate the
3

---

Under review as a conference paper at ICLR 2021
mixed discrete-continen optimization problem into two continuous optimization problems, both of which can be solved by conventional sendient-descent-oo mulation, extend the conventional fackpropagation(BP) algorithm to derive general co-training algorithm to co-optimize a selective sensing operator and a subsequent information decoding net- work. At last, by taking images as the sensing modality and using reconstruction the information decoding task, we propose a practical approach, referred to as SS+Net, to compose selective sens- ing framework for image selective sensing and reconstruction.
algorithms. Based on the new for-
In this paper, a lowercase letter denotes scalar or a scalar-valued function, and a uppercase letter denotes vector, a matrix, a tensor, or a vector-valued function. We use brackets index the element of a vector, a matrix, or a tensor. For example, assume X denotes a n-dimensional vector X [o -.. 1n then [ = I3 for i = 0, 1.
3.1 PROBLEM FORMULATION
Consider the original signal X is an r--dimensional vector, the subsampling rate is m and the subsampled measurement is m-dimensional vector. The selective sensing of X is a nonuniform subsampling or a selection process that can be formulated as
Y S( = [X[ X[/[m 1]]..
where S(X, /) is a function that stands for the selective sensing operator. I is a m-dimensional vector denoting the selection set, which contains the indices (integer values between and n - 1) of the elements to be selected. Consider N(Y, θ) is subsequent information decoding network and θ is the trainable parameters. o is a differentiable objective function that measures the information loss throughout the entire selective sensing process with respect information acquisition task. For instance, in a signal reconstruction task, the objective function can be defined as a loss function which measures the difference between the reconstructed signal and the original signal. The co- optimization problem of the sensing operator S and the information decoding network N can be formulated
Iapti O opt arg min (S 9)),
7,0
(2)
subject to ig. ................................ m are integers within interval [0,n- n0,n-1]
Given the entries of è are continuous variables, and the entries of are constrained to be integer variables within [0, n - 1]. the problem in (2) is a mixed discrete-continen
optimization problem
that can not directly solved with conventional gradient-descent based algorithms. This because the gradient of o with respect to I does not exist.
3.2 REFORMULATION BY CONTINUOUS INTERPOLATION AND DOMAIN EXTENSION
By applying the continuous interpolation on S with respect to and the extension on the domain of S. reformulate the problem in ) into two subproblems. For simplicity. adopt linear interpolation in our method. However, nonlinear interpolation methods can be also applied.
We define piece-wise linear function f(X,
((X, (X X(ra) - ia) + X(ra),
where i. (florr() 1, ia floor(i), To i, mod and ra is mod
In (3), i is realevalued scalar, floor() is the flooring function returning the closest integer that is less than or equal to the input, and mod the modulo operation. f(X,i) essentially interpolates the value of X over a continuous index in piece-wise linear fashion and extends the range of i to (-oc, oo). Given a X, f(X,t) is periodic over every n-length interval of At integer values of i, we have f(X,
X(i mod rı). which returns the original value of the (i mod n|-th element of X. Specifically, when is an integer in interval [0, n - 1], we have f(X, i) = X Due to the continuous interpolation and domain extension, f(X, i) is almost everywhere differentiable over i except for all the integer points. The choice of the gradient value at integer points turn out to be insensitive the algorithm performance. For simplicity, we define the derivatives of f(X, a) at integer values of as zero. As such, we have the gradient value of f with respect to i in the whole space which can be expressed as
0f [0,
if is integer, 0i (X|ral - X(ra)), otherwise.
(4)
4

---

Under review as a conference paper at ICLR 2021
Based on (4) define continuous selective sensing operator function S'
S'(X, = (o) 1m]-
(5)
Leveraging 3 and 5 we reformulate the mixed discrete-continuous optimization problem in (
into two subproblems defined as
IR. OR arg ((
(6)
1,0
and
Topt (round(1) mod n for each entry in In),
Optt arg min of (S O));
6
where round() an even rounding function that returns the closest integer value of the input, and the initial values of θ in 7 is On. Note that both the subproblems in and 7 are unconstrained and the gradient of o with respect
and θ can be calculated over the whole space in (6). Therefore,
can solve the subproblems in (6) and 7 sequentially using gradient-descent based algorithms.
For the brevity illustration, we refer to the process of solving the subproblem (6) and 7 as
initial- traiaittaaiing and the fine-tuning step, respectively. in the rest of this paper.
3.3 EXTENSION OF BACKPROPAGATION ALGORITHM
Generally, neural network models are trained over multiple training samples and the gradients of trainable parameters are calculated using the BP algorithm. We extend the BP algorithm and derive the gradient calculation (with respect to I) over a batch of training samples as follows.
Given a batch of samples XI.... Xs, of the signal X for training, the forward pass of the BP algorithm in the initial-training step can be derived as
S', 1). Z, N(Y, 9), o, o( Obotch
where i 1,... ., Z, the representation the information decoded by the network and Obatch the loss function that measures the average information loss throughout the selective sensing process. One can also choose to use the total information loss here.
In the backward pass of the BP algorithm, the gradient calculation with respect to θ is the same as in regular neural network training. The gradient calculation with respect to / can derived using the chain rule of derivative. Specifically, the gradient calculation of Obatch with respect to Y, can be derived as
Jobatch 0o; Y
for 1,..,,
Y,
Subsequently, the gradient calculation of Obatch with respect over batch of training samples can be derived as
Dobatch
1
dog
aY, 01
01
b
0Y,
1=1
(10)
do
0f [[]
1
0o,
0f(X,. [m 11)
1
0
0
b
aY,
01 [m 1]]
1=1
Leveraging the gradient calculations 9 and (10,, the subproblem in (6) can be therefore solved by using deettt-ddsseeetbeeeeeeeee algorithms. The outputs from the initial-training step include the opti- mized selection set İR and the corresponding reconstruction network parameters θ R.. As the entries interval (-oc, oc). one needs further convert In to integer selection
of In continuous set Topt as shown in 7p.. To compensate for the accuracy loss due to rounding, the reconstruction network shall be further fine tuned in the fine-tuning step while keeping Tpp fixed as shown in (7)
The entire algorithm of co-training a information decoding network and a selective sensing oper- ator S is summarized in Algorithm| [ the appendix.
5

---

Under review as a conference paper at ICLR 2021
3.4 IMAGE SELECTIVE SENSING AND RECONSTRUCTION
In the rest of the paper, we take image as the sensing modality and reconstruction as the informa- tion decoding task to demonstrate the first proof-of-concept of selective sensing. The prior work discussed in subsection 2.3 shows that neural network models can be trained to directly approx- imate the inverse mapping of compressive sensing perform the reconstruction. Therefore, we hypothesize that there exists a direct mapping from the selective sensed (nonuniformly subsampled) domain to the original image domain, and such mapping can be well approximated by neural network co-tra co-trained with the selective sensing operator. Furthermore, we hypothesize that the exist- ing image compressive sensing reconstruction networks can be also used for image selective sensing reconstruction.
Based on our hypotheses, we use the loss function 1(X, X) as the objective function in 2 where I a function that measures the distance between X and X, e.g. a mean rrean-square-eron function, and X is the output (Z in ( of the information decoding network N. As such, N is trained to directly reconstruct the original image from the selective sensing measurement as
X
X.
(1I)
Sensing
Reconstruction
We refer to the image selective sensing and reconstruction frameworks composed in such way as SS+Net.
To evaluate the performance of SS+Net, we compare against compressive sensing and uniform subsampling counterparts referred to as CS+Net and US+Net, respectively. CS+Net and US+Net use same reconstruction networks but replace the selective sensing operator in SS+Net with a Gaussian random sensing matrix] and uniform subsampling operator respectively. Additionally, we set all the hyper-parameters in SS+Net, CS+Net, and US+Net to be the same during the training for a fair comparison. The purpose of using CS+Net and US+Net as the reference methods is to reveal the true impact of selective sensing on compressive information acquisition comparison to the compressive sensing and uniform subsampling counterparts.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
We conduct experiments on two datasets with two different reconstruction networks at the measure- ment/subsampling rates ranging from 0.03125 0.25 (corresponding to the dimensionality reduc- tion ratios of 32-4x). The first dataset is CIFARIO 10 (Krizhevsky et al. 2009). The second dataset composed in the same way as illustrated in Xu et al. (2018), which has 228,688 training sam- ples and 867 testing samples. The testing samples are the non-overlapped image patches from Set5 (Bevilaquua et al. 2012) and Set14 (Zeyde et al.) 20100) All the samples are of size 64x64 with three color channels(RGB)) The sensing is performed channel-wise in all the experiments, i.e., for each framework of SS-Net, CS-Net and US-Net, each sample first reshaped to three 4096-dimensional vectors corresponding to the three color channels. Subsequently, three sensing operators correspond- ing to the three color channels are used to sense the three vectors of each sample, respectively, The sensed measurements from the three color channels grouped together and then fed into the construction network. Prior to the training, 5% of the training samples are randomly selected and separated the validation set.
The two reconstruction networks we experimented with are DCNet and ReconNet (Kulkarni et al. 2016). DCNet has the same network structure as the generator network of DCGAN (Radford al. 2015 We made some minor but necessary modifications to the structure of DCNet and ReconNet in order to perform image reconstruction and speed up the training. The modification details are summarized in subsection A.2 in the Appendix.
"The majority of existing drit-ddiven image compressive sensing reconstruction methods Kulkarni et al.
2016
Hora al. 2017 7 Metzler al. 2017 Xu [0188] [ /Zhang & Ghanem| 20188) Yao et al. 2019 Van Veen 2018) use a fixed Gaussian random sensing matrix to perform the sensing.
ct al.
Assume the sampling rate is m an input vector of length is equally divided into segments and the
central entry of each segment is sampled measurement

---

Under review as a conference paper at ICLR 2021
Network:
DCNel
ReconNNl
DCNel
ReconNel
45
40-
30
35
13. 94
14.05
4.45
Recorstructoon Accuracy
24-
25-
20
1.0
0.01
0.9
0.9-
0.17
0.8
0.8
0.16
WIS8
0.6
SS/Ours)
CS
US
0.06 0.12 0.18 0.24
0.06
0.12 0.18
0.24
0.06
0.12 0.18 0.24
0.08 0.12
0.18
Moasurement Rata
Set14
Figure 1: Comparison of information acquisition performance among selective sensing (SS), com- pressive sensing (CS), and uniform subsampling (US) measured in PSNR and SSIM. The average PSNR and SSIM improvements of SS over CS and US across all eight measurement rates are anno- tated on the figure.
In the beginning of the training process of SS+Net, the selection index I is randomly initialized with real values from the uniform distribution UO, n). We co-train the selective sensing operator and the reconstruction network for 300 iterations, which first 150 iterations are used for initial-training step and the rest are used for the fine-tuning step. We use two different optimizers to optimize different components of SS+Net: An Adam optimizer with learning rate 0.001 used to optimize reconstruction networks and a SGD optimizer with learning rate of 100000 is used to optimize sensing operators. Using a high learning rate for training the selective sensing operators is because the gradient values with respect the selection index 7 (calculated with (4) turns out be orders of magnitude smaller (because adjacent pixels in natural images mostly have very close pixel values) than the the rest of the gradient values and the learning rate of 100000 performs well in the experiments. For the training of the CS+Net and US+Net counterparts, except that there is no optimizer for sensing operators, all the other experiment setups remain the same with SS+Net.
4.2 EXPERIMENT RESULTS
The reconstruction accuracy is measured as average reconstruction PSNR and SSIM. The experi- ment results of PSNR and SSIM plotted in Figure I As shown in Figure [ selective sensing achieves up to 44.92dB/O. 995222 reconstruction PSNR/SSIM at the measurement rate of 0.25 (dimen- sionality reduction ratio of 4). Even the low measurement rate of 0.03125 (dimensionality reduc- reconstruction PSNR/SSIM.
tion ratio of 32), selective sensing still achieves at least 23.35dB/0.699 The experiment results validate our hypothesis that the direct mapping from the selective sensed do- main to the original image domain can be well approximated by existing reconstruction neural net- works co-trained with the selective sensing operator, and data-driven nonuniform subsampling can well preserve signal information under the presence of the co-trained information decoding neural network. Furthermore, the experiment results show that selective sensing consistently outperforms compressive sensing and uniform subsampling, especially at higher dimensionality reduction ratios. The average PSNR/SSIM improvement of selective sensing over compressive sensing and uniform subsampling across all the experiments is 3. 3.73dB/0.07 and 9.43dB/0.16 respectively. As the only difference between SS+Net, CS+Net, and US+Net is the sensing operator used, the experiment re- sults imply that selective sensing better preserves signal information than compressive sensing and

---

Under review as a conference paper at ICLR 2021
Dimensionality Reduction Ratio
16
Uniform Subsampling
Original
(20.82,0.56)
(19.16, 19..1)
(17 .8,0..9))
(16.33,0.23)
Selective Sensing(Ours)
(PSNR,SSIM)
(22.77,0.69)
(21 (,0.9)
(9.77,00.6))
(18.79,0.36)
Compressive Sensing
(20.49,0.52) (19.22,0.43)
(18.40,0.33) (17.7,0..88
Figure 2: Visual reconstruction quality comparison among selective sensing, compressive sensing and uniform subsampling. The reconstruction network used is ReconNet, and the example image is from the Set14. Across all the dimensionality reduction ratios, selective sensing produces sharper image with finer details presented, e.g. the beard and the skin textures.
uniform subsampling result of the co-optimization of the sensing and reconstruction stages. The detailed numerical PSNR and SSIM results are presented in Table Table []] Table and Table in the Appendix.
Compared with the images in CIFAR10, the images in Set5 and Set14 have more details, which makes the reconstruction inherently more difficult. We take one image from Set14 to illustrate the visual reconstruction quality comparison. As shown in Figure [ selective sensing reconstructs the image with finer and sharper details, such the beard and the textures of the skin, than compressive sensing and uniform subsampling across all dimensionality reduction ratios. More visual reconstruction quality comparison showing the same results are illustrated Figure Figure E and Figure 5 in the Appendix. These visual reconstruction quality comparisons show strong evidence that selective sensing better preserves signal information than compressive sensing and uniform subsampling.
example
4.3 COMPARISON AGAINST RANDOM INDICES
To understand the effectiveness of the trained selection indices selective sensing over randomly generated selection indices, design trivial random subsampling approach called RS+Net and compared it with SS+Net. The experimental setup for RS+Net is consistent with US+Net except that the selection indices of RS+Net are integers randomly generated from the uniform distribution UO. n). The experiments are performed on CIFAR1O at the dimensionality reduction ratios of 32 and (measurement rates 0.03125 and 0.125) with ReconNet. At each measurement rate, ran- domly generate eight different sets of selection indices, based on which we train eight reconstruction networks from scratch, respectively, for RS+Net.
We observe that the final reconstruction accuracy of RS+Net average is significantly lower than SS+Net counterparts. The average PSNR of RS+Net is 22.78 dB and 31.02 dB at measurement rates of 0.03125 and 125, which is 5.73 dB and 9,49 dB lower than the SS+Net counterparts, spectively. addition, we observe a large variance in the final reconstruction accuracy of RS+Net across the eight sets of selection indices. Specifically, the measurement rate of 0.03125 and 0.125, the gap between the maximum and the minimum PSNRs is 0.59 dB and 0,54dB, and the standard deviation of PSNRs is 0.19 dB and 0.20 dB, respectively. The detailed results summarized in Ta-
8

---

Under review as a conference paper at ICLR 2021
ble[5iin in the Appendix. Furthermore, illustrate the superiority of the co-optimized selection indices in selective sensing over random selection indices, we also train two reconstruction networks from scratch at the measurement rates of 0.03125 and 0.0125 with the corresponding pre-trained selection indices from SS+Net remaining fixed in the training process, respectively. The final reconstruction accuracy is 28.49 dB and 40.47 dB at the measurement of 0.03125 and 0.0125, which is 5.71 dB and 9.45 dB higher than the original RS+Net counterparts using random selection indices, respectively.
These results indicate that random selection indices are insufficient for retaining information of interest with respect to the subsequent reconstruction network. Differently, learned indices from co-training selective sensing operator with subsequent reconstruction network can significantly improve the selective sensing performance in terms of retaining the information of interest for high- accuracy reconstruction. The visualization and comparison of learned (co-trained) and random se- lection masks are shown in Figure 6 7 and g in subsection
A.3 of the Appendix.
4.4 DISCUSSION ON NONCONVEXITY AND INCOHERENCE
From the optimization perspective, both the original problem 2 and the relaxed problem 6 are non- convex problems. Thus, it is not guaranteed that the gradient-desseent-baeed algorithm can find the globally optimal solution of the problem [ Additionally, it is known that deep neural network models are typically non-convex with a great number of local minima (Goodfellow et al.) 2016). The gap between local and global minima remains an open topic of research. Similarly, our ap- proach leverages the gradient-descent algorithm and relaxed problem to find the local minimum of problem 2 that are empirically proven to have outstanding performance. Specifically, the experiment results show that the proposed algorithm | can consistently find a local optimum of the indices that has significantly better sensing performance than random indices and random Gaussian sensing matrices terms of the reconstruction accuracy across di different datasets and compression ratios.
It should be also noted that selective sensing, similar to many data-driven compressive sensing methods that co-train a sensing matrix with a reconstruction network (MouuvviMuuavvi et al., 2017 2018 Lohit et al. 2018), does not explicitly require the incoherence of sensing matrices nor sparsity ba- sis. Previous studies have empirically shown that these data-driven compressive sensing methods can achieve significantly higher reconstruction performance compared to conventional model-base methods(Li et al., 2013) Dong et al., (014) that have the incoherence and sparsity requirements. Similarly, experiment results various datasets, reconstruction networks, and measurement rates also show that selective sensing consistently outperforms compressive sensing and uniform subsampling counterparts. Furthermore, compared the data-driven compressive sensing method in Lohit et al. (2018), selective sensing can achieve 6.02 dB/0.031 and 0.72 dB/0.007 higher re- construction PSNR/SSIM on the CIFAR10 database with the same reconstruction network Kulkarni et al.) (2016) at the measurement rate of 0.125 125 and 0.25, respectively. These comparison results are the strong empirical evidence that selective sensing can well preserve signal information without the incoherence nor sparsity requirements.
CONCLUSION
In this paper, we propose a selective sensing framework that adopts the novel concept of data-driven nonuniform subsampling to perform on-sensor data dimensionality reduction in a computation free fashion. Selective sensing adopts co-optimization methodology co-train a selective sensing op- erator with a subsequent information decoding neural network. The co-training of selective sensing is first formulated as mixed-discrete- contrnnuoooo optimization problem. By applying continuous interpolation and domain extension to the sensing index domain with quantization and fine-tuning techniques, we reformulate the problem into two continuous optimization subproblems that can be solved by cradient-descent-beee lective sensing operator with the subsequent information decoding neural network. The experiments on CIFARIO, Set5, and Set14 datasets show that the proposed selective sensing framework can achieve an average reconstruction accuracy improvement in terms of PSNR/SSIM of 3.73dB//001 and 9.43dB/0.1 16 over compressive sensing and uniform subsampling counterparts across the dimen- sionality reduction ratios of 4-32x, respectively. The computation- free nature of selective sensing makes it highly suitable solution for performing compressive information acquisition on resource- constrained sensor devices or high-data-rate sensor devices dealing with hign-dimensional signals.
algorithms. This is the key to enabling the co-training of the se-

---

Under review as a conference paper at ICLR 2021
REFERENCES
Cagla Bahadir, Alan Wang, Adrian Dalca, and Mert Sabuncu. Deep-learning-hasee optimization of the under-sampling pattern in mri. IEEE Transactions on Computationnal Imaging, 2020.
Cagla Deniz Bahadir, Adrian V Dalca, and Mert Sabuncu. Learning based optimization of the sampligg pattern in mri. In International Conference on Information Processing in Medical Imaging, pp. 780-792. Springer, 2019.
Luca Baldassarre, Yen-Huan Li, Jonathan Scarlett, Baran Gözeü, Ilija Bogunovic, and Volkan Cevher. Learningbbssed compressive subsampling. IEEE Jourmal of Selected Topics in Signal Processing, 10(4):809-822) 822, 2016.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. 2012.
Saman Biookaghazadch, Ming Zhao, and Fengbo Ren. Are fpgas suitable for edge computing? In [USENIX} Workshop on Hot Topics in Edge Computing (HotEdge 18), 2018.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using genera- tive models. In Proceedings of the 34th International Conference Machine Learning-Volle VVllue 70, pp. 537-546. JMLR. org, 2017.
Centeye. Stonyman and hawksbill vision chips. current- Current-centeye- www.eeeeeeeeeeeeeeeeeeeeeeeeeeee vireeviiiion wineee-hhiiinn
http:://wwt /tte:://www.centeeee.e cotp://uucco///sssss//tc
2020. Accessed: 2020-09-27.
Sundeep Prabhakar Chepuri, Geert Leus, et al. Sparse sensing for statistical inference. Foundations and Trends@ in Signal Processing, (33-4))233-30 368, 2016.
Nathaniel Chodosh, Chaoyang Wang, and Simon Lucey. Deep convolutional compressed sensing for lidar depth completion. In Computer Vision ACCV 2018, pp. 499-513, 2019.
Hamid Dadkhahi and Marco Duarte. Masking schemes for image manifolds. In 2014 IEEE Workshop on Statistical Signal Processing (SSP), pp. 252-255. IEEF, 2014.
Hamza Djelouat, Abbes Amira, and Faycal Bensaali. Compressive sensing-based iot applications: A review. Journal of Sensor and Actuator Networks, 7(4), 2018
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions pattern analysis and machine intelligence, 38(2): 295-307, 2015.
Weisheng Dong, Guangming Shi, Xin Li, Yi Ma, and Feng Huang. Compressive sensing via nonlo- cal low-rank regularization. IEEE Transactions on Image Processing, 23(8):3618-3633 2014.
Marco F Duarte, Mark A Davenport, Dharmpal Takhar, Jason N Laska, Ting Sun, Kevin F Kelly, and Richard G Baraniuk. Single-pixel imaging via compressive sampling. IEEE signal processing magazine, 25(2):83-91. 2008.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7/ www. www.ddeppppeennie www..eeepleaningpoo..cc org
Baran Gözcü, Rabeeh Karimi Mahabadi, Yen-Huan Li, Efe Iheak, Tolga Çukur, Jonathan Scarlett, and Volkan Cevher. Learning baasedg-based compressive mri. JEEE transactions on medical imaging, 37 16):1394-1406, 2018.
Tao Hong, Xiao Li, Zhihui Zhu, and Qiuwei Li. Optimized structured sparse sensing matrices for compressive sensing. Signal Processing, 159:119-1222 2019.
Iris AM Huijben, Bastiaan S Veeling, and Ruud JG van Sloun. Deep probabilistic subsampling for task task-adaptive compressed sensing. In International Conference on Learning Representations, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2021
Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. Reconnet: Non- iterative reconstruction of images from compressively sensed measurements. In Proceedings of the IEEE Conference Computer Vision and Pattemn Recognition, pp. 449-458, 2016.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-46900 2017.
Chengbo Li, Wotao Yin, Hong Jiang, and Yin Zhang. An efficient augmented lagrangian method with applications to total variation minimization. Computational Optimization and Applications, 56(3):507-530, 2013.
Suhas Lohit, Kuldeep Kulkarni, Ronan Kerviche, Pavan Turaga, and Amit Ashok. Convolutional neural networks for noniterative reconstruction of compressively sensed images. IEEE Transac- fions on Computational Imaging, 4(3):326-340, 2018.
Alexander J Macfaden, George SD Gordon, and Timothy D Wilkinson. An optical fourier transform coprocessor with direct phase determination. Scientific reports, 7(1):1-8, 2017.
Addison Mayberry, Pan Hu, Benjamin Marlin, Christopher Salthouse, and Deepak Ganesan. ishadow: design of a wearable, real-time mobile gaze tracker. In Proceedings of the 12th an- nual international conference on Mobile systems, applications, and services, pp. 82-94, 2014.
Chris Metzler, Ali Mousavi, and Richard Baraniuk. Learned d-amp: Principled neural network based compressive image recovery. In Advances in Neural Information Processing Systems, pp. 1772-1783, 2017.
Ali Mousavi and Richard Baraniuk. Learning to invert: Signal recovery via deep convolutional networks. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2272-2276. IEEE, 2017.
Ali Mousavi, Gautam Dasarathy, and Richard G Baraniuk. Deepcodec: Adaptive sensing and re- covery via deep convolutional neural networks. arXiv preprint 1rXiv:1707.03386 2017.
Ali Mousavi, Gautam Dasarathy, and Richard G Baraniuk. A data-driven and distributed approach to sparse signal representation and recovery. 2018.
Duc Minh Nguyen, Evaggelia Tsiligianni, and Nikos Deligiannis. Deep learning sparse ternary projections for compressed sensing of images. In 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1125-1129. IEEE, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Ryan Robucci, Jordan D Gray. Leung Kin Chiu, Justin Romberg, and Paul Hasler. Compressive sensing cmos separable-transfore image sensor. Proceedings of the IEEE, 98(6); 2010.
Dave Van Veen, Ajil Jalal, Mahdi Soltanolkotabi, Eric Price, Sriram Vishwanath, and Alexandros G Dimakis. Compressed sensing with deep image prior and learned regularization. arXiv preprint arXiv:1806.06438, 2018.
Yuhao Wang, Xin Li, Kai Xu, Fengbo Ren, and Hao Yu. Data-driven sampling matrix boolean optimization for energy-efficient biomedical signal acquisition by compressive sensing. IEEE transactions on biomedical circuits and systems, 11(2):25--66 2016.
Tomer Weiss, Sanketh Vedula, Ortal Senouf, Alex Bronstein, Oleg Michailovich, and Michael Zibulevsky. Learning fast magnetic resonance imaging. arXiv preprint 19Xx::9905.09324 2019.
Shanshan Wu, Alexandros G Dimakis, Sujay Sanghavi, Felix X Yu, Daniel Holtmann-Rice Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar. Learning a compressed sensing measure- ment matrix via gradient unrolling. arXiv preprint arXiv:1806 10/75, 2018.

---

Under review as a conference paper at ICLR 2021
Kai Xu, Zhikang Zhang, and Fengbo Ren. Lapran: A scalable laplacian pyramid reconstructive ad- versarial network for flexible compressive sensing reconstruction. Proceedings of the European Conference Computer Vision (ECCV), pp. 485-500, 2018.
Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, Jing-Hao Xue, and Qingmin Liao. Deep learning for single image super-resolution: brief review. IEEE Transactions on Multimedia, 21 (12):3106-312,, 2019.
Hantao Yao, Feng Dai, Shiliang Zhang, Yongdong Zhang, Qi Tian, and Changsheng Xu. Dr2-net: Deep residual reconstruction network for image compressive sensing. Neurocompuiing, 359; 483-493, 2019.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- and surfaces, pp. 711-730 Springer, 2010.
representations. In International conference
Jian Zhang and Bernard Ghanem. Ista-net: Interpretable optimization-imspired deep network for image compressive sensing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1828-1837, 2018.
Wenfeng Zhao, Biao Sun, Tong Wu, and Zhi Yang. On-chip neural data compression based on compressed sensing with sparse sensing matrices. IEEE transactions biomedical circuits and systems, 12(1)2242-254, 2018.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 SUMMARIZED ALGORITHM
Algorithm Main algorithm
Input: training samples N XN, number of iterations mariters for initial-training step, batch size b
Initialize O
Initial-training
for iter = 1 to mariters do
for batch 1 to N. do
Forward pass
Execute 8
Backward pass
Using the BP algorithm to calculate the gradient with respect to θ
Execute 9
Execute 10 to calculate the gradient with respect to
Optimize 7 and & using the calculated gradients
end for end for
TR, On = 1,
Execute the rounding and modulo operations over entries of n as 7 to get Topt-
Fine-tuning
Initialize O R. further optimize & with the gradients calculated by the BP algorithm while
keeping Topt
End
Topts Opp 1,0
13

---

Under review as a conference paper at ICLR 2021
A.2 MORE EXPERIMENTS DETAILS
The code, datasets and pretrained models can be downloaded from:
https://Figgharee
The experiments are conducted in parallel on four RTX 2080 Ti GPU cards. One training pro- cess(300 iterations) runs one GPU card takes around 80 minutes.
We made following necessary modifications to the structure of DCNet and ReconNet in order to perform image reconstruction and speed up the training process. We modify the first layer of DCNet by increasing the number of input channels to be equal to the number of measurements in order to feed measurements into DCNet. As ReconNet is designed to reconstruct grayscale images, made some modifications to the network structure of ReconNet in order to reconstruct color images. Specifically, we replace the first layer of ReconNet, which fully-connected layer, with three fully- connected layers in parallel to generate three feature map corresponding to three color channels. In addition, we modify the third and the sixth convolutional layers to have three output channels in order maintain the dimensionality of the original image. The batch normalization layers are also added right behind each convolutional layer (except for the last convolutional layer which is output layer) to accelerate the training.
14

---

Under review as a conference paper at ICLR 2021
A.3 MORE EXPERIMENT RESULTS
Figure 6 Figure 7 and Figure 8 shows that sampling points in learned masks are more evenly dis- tributed the whole image and less clustered at specific locations across all measurement rates than random masks. Consequently, the measurements from selective sensing are less noisy but more in- formative than random sensing, For instance, the outlines of objects in the measurements sensed by selective sensing are more visually comprehensible than random sensing, which provides possible visual explanation on the effectiveness of selective sensing over random subsampling.
Table 1: Reconstruction performance comparison on CIFARIO using DCNet as the reconstruction network.
Metrics
PSNR (dB)
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
28.055576
23.678496
17.903165
4.3770796
10.152411
0.0625
33.76353
27.567673
20.432642
6.1958566
13.330888
0.09375
37,788893
30.687367
22.436043
7.1015266
15.352851
0.125
40.4781
33,434791
24.314229
7.043309
16.163872
0.15625
41.843324
35.391706
25.934857
6,4516183
15.908467
Measurement rate
0.1875
42.571765
37.023333
27.796602
5.5484323
14.775163
0.21875
43.227788
38.326256
29.525659
4.9015314
13.702128
0.25
43.928165
38.693752
31.793804
5.2344128
12.134361
Metrics
SSIM
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
0.8785
0.745
0.5759
0.1335
0.3026
0.0625
0.957
0.8542
0.6638
0.1028
0.2932
0.09375
0.9789
0.9132
0.732
0.0657
0.2469
0.125
0.987
0.9473
0.7899
0.0397
0.1971
0.15625
0.9899
0.9652
0.8334
0.0247
0.1565
Mcasurement rate
0.1875
0.9913
0.9747
0.8779
0.0166
0.1134
0.21875
0.9923
0.9795
0.9084
0.0128
0.0839
0.25
0.9932
0.9809
0.9381
0.0123
0.0551
15

---

Under review as a conference paper at ICLR 2021
Table 2: Reconstruction performance comparison on CIFARIO using ReconNet as the reconstruction network.
Metrics
PSNR (dB)
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
28.510575
24.162001
17,833156
4.3485742
10.677419
0.0625
33.835395
28.244744
20.37912
5.5906511
13.456275
0.09375
37.465897
31.460441
22.615043
6.005456
14.850854
0.125
40,505386
34.428422
24.507658
6.0769649
15.997729
0.15625
41.91067
37.112229
26.295812
4.7984413
15.614858
Mcasurement rate
0.1875
42.64523
39.830833
173977
2.8143966
14.471256
0.21875
44.613196
41.977892
30.068614 4
2.6353032
14.544581
0.25
44.922706
43.377697
32.
1.5450094
12.765289
Metrics
SSIM
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
0.8916
0.7629
0.5733
0.1287
0.3183
0.0625
0.9602
0.8683
0.6694
0.0919
0.2908
0.09375
0.9793
0.9225
0.7474
0.0568
0.2319
0.125
0.988
0.9549
0.8046
0.0331
0.1834
0.15625
0.9918
0.9731
0.8548
0.0187
0.137
Measurement rate
0.1875
0.9935
0.9849
0.889
0.0086
0.
0.21875
0.9952
0.9903
0.9199
0.0049
0.0753
0.25
0.9952
0.9932
0.9427
0.002
0.0525
16

---

Under review as a conference paper at ICLR 2021
Table Reconstruction performance comparison on Set5 and Set14 using DCNet as the reconstruc-
tion network.
Metrics
PSNR (dB)
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
23.356654
21.988134
19.158458
1.3685191
4.1981953
0.0625
25.020666
23.655472
21.360483
1.3651944
3.6601834
0.09375
25.821571
24.320524
22.822223
1,5010468
2.9993483
0.125
29.330996
24.941122
24.049497
4.3898736
5,2814991
0.15625
30.142491
26.041837
25.023534
4.1006538
5.1189574
Mcasurement rate
0.1875
30.798615
28.254278
25.973201
2.5443367
4.8254134
0.21875
31,402867
28.782112 12
26.572408
2.6207543
4.8304582
0.25
32.040132
29.214131
27.364247
2.8260015
4.6758851
Metrics
SSIM
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
0.6975
0.6131
0.539
0.0844
0.1585
0.0625
0.7696
0.6672
0.5995
0.1024
0.1701
0.09375
0.8079
0.7041
0.6448
0.1038
0.1631
0.125
0.8493
0.731
0.688
0.1183
0.1613
0.15625
0.8674
0.7559
0.7193
0.1115
0.1481
Measurement rate
0.1875
0.8811
0.7866
0.75
0.0945
0.1311
0.21875
0.8919
0.803
0.7719
0.0889
0.12
0.25
0.9005
0.8159
0.7969
0.0846
0.1036
17

---

Under review as a conference paper at ICLR 2021
Table 4: Reconstruction performance comparison on Set5 and Set14 using ReconNet as the recon-
struction network.
Metrics
PSNR (dB)
Sensing operator
SS
CS
US
SS over CS
SS US
0.03125
25.14523
23,492886
19.141701
1.652344
6.0035286
0.0625
27.002097
25.27621
21.385219
1.725887
5.6168772
0.09375
28.276923
26.41306
22.907161
1,8638628
5,3697623
0.125
29.357467
27.2674
24.135962
2.0900672
5.2215052
0.15625
30.240035
27.814269
25.232525
2.4257659
5.0075104
Measurement rate
0.1875
31.18125
28.651697
26.099135
2.5295576
5.0821148
0.21875
31.844453
29.195588
26.796003
2.6488653
5.0484498
0.25
32.397313
29.493561
27.412122
2.9037527
4.9851912
Metrics
SSIM
Sensing operator
SS
CS
US
SS over CS
SS over US
0.03125
0.7256
0.625
0.5444
0.1006
0.1812
0.0625
0.7924
0.6816
0.6019
0.1108
0.1905
0.09375
0.8314
0.7218
0.6547
0.1096
0.1767
0.125
0.8575
0.7519
0.698
0.1056
0.1595
0.155255
0.8783
0.7678
0.7354
0.1105
0.1429
Mcasurement rate
0.1875
0.8932
0.7891
0.7607
0.1041
0.1325
0.21875
0.9026
0.8106
0.7832
0.092
0.1194
0.25
0.9107
0.8194
0.8038
0.0913
0.1069
Table 5: Summary of reconstruction accuracy results of RS+Net on CIFARIO dataset, Metrics: PSNR(dBYSSSM
Measurement rate
Mean
Min
Max
Standard deviation
0.03125 0.125
22.78/0.74 31.02/0.93
22.57/0.73 30.71/0.93
23.16/0.75 31.25/0.94
0.19/0.006
0.20/0.0018
18

---

Under review as a conference paper at ICLR 2021
Original
(PSNR,SSIM)
Dimensionality Reduction Ratio
4
8
16
32
Selective Sensing(Ours)
(36.74,0.95)
(33.45,0.91)
(30.880..86)
(28.70,0.80)
Compressive Sensing
(33.04,0.89)
(30.63,0.83)
(28.52,0.77)
(26.710.72)
Uniform
Subsampling
(30.69,0.86) (26.36,0.75) (22.54,0.65) (19.43,0.59)
Figure 3: Visual reconstruction quality comparison among selective sensing, compressive sensing and uniform subsampling. The reconstruction network used is ReconNet, and the example image is from the Set5 dataset. Across all the compression ratios, selective sensing produces a sharper image with finer details presented, egg. the eyelashes and the textures of the hat.
19

---

Under review as a conference paper at ICLR 2021
Original
(PSNR,SSIM)
Dimensionality Reduction Ratio
4
8
16
32
Selective Sensing(Ours)
(30.89,0.95)
(26.64,0.89)
(22.7500.8
(19.48,0.69)
Compressive Sensing
(25.72,0.83)
(22.53,0.76)
19. .9..20,0663)
(16.3,00.49)
Uniform Subsampling
(22.19,0.82) (17. 16,0.62)
(14.63,0.46) (1..390.388)
Figure 4: Visual reconstruction quality comparison among selective sensing, compressive sensing and uniform subsampling. The reconstruction network used is ReconNet, and the example image from the Set5 dataset. Across all the compression ratios, selective sensing produces a sharper image with finer details presented, egg. the textures the wings.
20

---

Under review as a conference paper at ICLR 2021
Original
(PSNR,SSIM)
Dimensionality Reduction Ratio
4
8
32
Selective Sensing(Ours)
(33.10,0.95)
(28.64,0.88)
(24.66,0.76)
(2188,0.65)
Compressive Sensing
(26.73,0.81)
(22.97.0.67)
(19.98,0.54)
(17.90,0.45)
Uniform Subsampling
(23.34,0.77) (18.39,0.58)
(16.06,0.43)
(14.96,0.35)
Figure 5: Visual reconstruction quality comparison among selective sensing, compressive sensing and uniform subsampling. The reconstruction network used is ReconNet, and the example image is from the Set14 dataset. Across all the compression ratios, selective sensing produces a sharper image with finer details presented, e.g. the edges of the stripes.
21

---

Under review as a conference paper at ICLR 2021
Figure 6: Visual comparison between learned and random selection masks. Top to bottom: masks at the measurement rates from 0.03125 to 0.25. Left to right: learned selection masks of selective sensing, selective sensing measurements, random selection masks, and randomly selected measure- ments. The masks and measurements are plotted based on color channels.
22

---

Under review as a conference paper at ICLR 2021
Figure 7: Visual comparison between learned and random selection masks. Top to bottom: masks at the measurement rates from 0.03125 to 0.25. Left to right: learned selection masks of selective sensing, selective sensing measurements, random selection masks, and randomly selected measure- ments. The masks and measurements are plotted based on color channels.
23

---

Under review as a conference paper at ICLR 2021
Figure 8: Visual comparison between learned and random selection masks. Top to bottom: masks at the measurement rates from 0.03125 to 0.25. Left to right: learned selection masks of selective sensing, selective sensing measurements, random selection masks, and randomly selected measure- ments. The masks and measurements are plotted based on color channels.
24


Reviews:
[{"review_id": "mw6U9lMb6m5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to jointly optimize non-uniform subsampling pattern as well as the reconstruction network to perform image compressed sensing. It shows using learned selective sensing can significantly improve the reconstruction accuracy over the (random Gaussian) compressed sensing and uniform subsampling counterparts. \n\nHowever, the novelty of the proposed method is very limited. The idea of using continuous interpolation on the discrete indices/grid, i.e. the main technical contribution in this paper, has already been proposed for CS-MRI sensing pattern design [1]. Although it is conducted on the k-space, applying this technique on image pixel space is straightforward and trivial. Besides, the comparison to a closely related work [2] is missing -- the probabilistic subsampling could also be used for the problem this paper tackles. \nFinally, since the selective sensing pattern is learned from data, it's more fair to compare it with the learned compressed sensing counterpart, as exploited in the journal version of the ReconNet [3] used in this paper. \n\n[1] PILOT: Physics-Informed Learned Optimized Trajectories for Accelerated MRI  \n[2] Deep probabilistic subsampling for task-adaptive compressed sensing  \n[3] Convolutional Neural Networks for Noniterative Reconstruction of Compressively Sensed Images  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "jeHLZF1pzbM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a nonuniform sampling design scheme chosen using training data to reduce the computation of compressed sensing acquisition. The use of learning methods such as back propagation for the nonuniform sampling design problem is interesting. However, compressive sensing approaches in practice do not perform computation to obtain the measurement vector; instead, the sensors rely on custom hardware (such as the single pixel camera or the modulated wideband converter) to have the hardware act on the discretized signal according to the matrix design. Thus, considerations of the \"sensing complexity\" are moot in those cases, and the approach provided in this paper is only relevant in cases where custom sampling schemes can be designed (e.g., when the referred imaging sensors are used).\n\nIn these terms, the paper can be understood as the joint formulation of a nonuniform sampling and deep learning-based recovery of the sensed image. The authors provide numerical comparisons with random sampling and uniform subsampling, as well as with CS with random matrices. The use of uniform subsampling makes their approach comparable to existing approaches for image super-resolution based on deep learning, but this connection is not discussed.\n            \nThere are some other minor issues with the paper:\n\n* Claiming a complexity of O(1) for selective sensing ignores the fact that the number of measurements required for recovery is dependent on the signal length N, which is accounted for in the other complexities cited.\n* Like selective sensing, sparse coding also does not require any prior knowledge of the sparsity structure of the data.\n* Fourier (k-space) measurements can be obtained directly with optical hardware.\n* The function o(.) has different inputs in different instances; please check for consistency.\n* The relaxed formulation of the integer program does not specify the fact that the solution should be a subset of {0,...,M-1}; some of the entries of the set I might point to the same integer.\n* The description of fine-tuning should be more detailed - is it simply more iterations of GD over Theta?\n* The sentence \"We also train the reconstruction networks in RS+Net from scratch based on the pre-trained selection indices in SS+Net\" is not clear. Wouldn't this make the random samples not random?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WokzP87r6n0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a framework for jointly optimizing a selective sensing operator and a neural network for reconstruction. The motivation is to alleviate the quadratic (in dimensions) complexity associated with standard compressive sensing by using a dimension-free selective sensing approach, which can be jointly optimized with the decoder while guaranteeing that the relevant information is preserved. As this formulation yields a mixed discrete-continuous optimization, the authors propose a standard relaxation of the discrete constraints using interpolation. \n\nStrengths: The paper is generally well-written and organized and easy to follow. The idea of jointly optimizing the sensing and the decoder is interesting, although not really that novel.  \n\nWeaknesses: While the work has some merit, the development of ideas is lacking in many respects. Here are few major comments I have: \n- The paper incorrectly overstressed the limitation of compressive sensing due to the computational complexity of applying a matrix, say from a Gaussian random ensemble, to the signal of interest. In many applications, applying such matrices follows from the structure of the sensing device. For example, applying a Fourier matrix, which come with provable guarantees, can be done using a lens! In other words, physics does it for you for free. In other cases, a mask amounts to applying a Hadamard matrix to your image. \n- At a fundamental level, I am questioning the generality of the proposed approach which uses selective sampling. More specifically, it is well recognized that compressive reconstruction from a small number of measurements requires the incoherence of the sensing matrix and the sparsifying basis. This is known not to hold with the sparse matrices associated with selective sampling. The paper does not even explain any of these issues and limitations, and does not investigate it. \n- The main motivation is to reduce the computational complexity of sensing. However, the paper does not provide any comparisons to the state-of-the-art concerning computational gains. While the joint optimization of the selector and the decoder is done during the training phase, in practice they will need to be re-trained for different datasets and scenarios, which is not necessarily going to save on computation.\n- There is a large body of work on non-uniform compressive sampling that is not discussed in this work. \n- Autoencoders can also be trained to obtain compressed representations that preserve information. This is a well-studied area that the paper fails to discuss. \n\nGiven the above, I am unable to recommend this paper for publication. For me, it is a clear reject. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DmxXopoAnQu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides a data-driven approach to the selection of sampling points for compressive sensing. This is not a new problem, and has been studied for a while now. There is a large literature on it. For example, see below, references within and papers citing those:\n- L. Baldassarre, Y. Li, J. Scarlett, B. Gözcü, I. Bogunovic and V. Cevher, \"Learning-Based Compressive Subsampling,\" in IEEE Journal of Selected Topics in Signal Processing, vol. 10, no. 4, pp. 809-822, June 2016, doi: 10.1109/JSTSP.2016.2548442.\n- Gözcü, Baran, et al. \"Learning-based compressive MRI.\" IEEE transactions on medical imaging 37.6 (2018): 1394-1406.\n- Bahadir, Cagla Deniz, Adrian V. Dalca, and Mert R. Sabuncu. \"Learning-based optimization of the under-sampling pattern in MRI.\" International Conference on Information Processing in Medical Imaging. Springer, Cham, 2019.\n\nThe authors propose a relatively ad hoc relaxation of the problem to a continuous set instead of an index and then perform the selection there. However, there is no discussion on how this relaxed problem is in fact quite non-convex and how this choice of relaxation introduces a significant number of local minima. The comparison the authors offer is with very simplistic algorithms, rather than the more sophisticated ones in the literature.\n\nOverall, this is an interesting approach that I think deserves to be eventually published. However, the paper needs significant work and comparison with the state of the art in the literature.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the problems studied in this paper are interesting, and the solutions provided are reasonable.  However qualitative and quantitative comparisons to state of the art methods are missing, and the sensing model assumed by the paper needs to be more well motivated.

Decision: Reject