Compressed Sensing Supplemental Material for Graphical Models and Inference

Compressed sensing (CS) is an area of signal processing and statistics that emerged in the late 1990’s and then exploded in the mid-2000’s [1, 2, 3, 4, 5]. The main idea is that many signals can be reconstructed and/or processed using many fewer measurements than predicted by a naive application of Nyquist’s theorem. CS originated with the observation that many signal processing systems first sample a large amount of data, then perform a linear (e.g., wavelet) transform, and finally throw all the small coefficients away. If the locations of the large-magnitude transform coefficients were known in advance, then one could just sample those values directly and reduce the complexity enormously. CS is a research area that emerged when people realized that a random transform could be used (with a small penalty) to achieve the same result without prior knowledge about the locations of important coefficients. CS can be broken into two phases: sampling and reconstruction. In the sampling phase, the signal of interest (SOI) is sampled by computing its dot product with a set of sampling kernels. The reconstruction phase estimates the SOI from the observed samples. In many cases, the number of samples required for a good estimate is much smaller than other methods (e.g., Nyquist sampling at twice the maximum frequency). The recent excitement about CS is largely motivated by a series of practical signal reconstruction algorithms: basis pursuit [1, 4, 5], LASSO [6, 7], and approximate message-passing (AMP) [8, 9]. The original basis-pursuit algorithm was actually developed to solve the (essentially equivalent) best-basis selection problem, whose goal is to find a good sparse approximation for a vector in terms of an overcomplete dictionary of vectors. Best-basis selection is a classic problem in signal processing and statistics and basis-pursuit defines a linear program (LP) which efficiently approximates the answer. The LASSO uses a linearly constrained quadratic program to solve the same problem when there is measurement noise. Finally, AMP provides a very efficient message-passing solution for the LASSO problem. Other natural applications of these algorithms include compression, denoising, and super-resolution estimation. To motivate CS, assume that some linear transform of the SOI is either (i) exactly sparse or (ii) approximately sparse. For example, a 1D piecewise constant signal with a small number of jumps is exactly sparse after first-order differencing because the number of non-zero elements in the difference sequence is equal to the number of jumps. This result is easily generalized to piecewise polynomial functions by considering higher order differences. Likewise, many 2D images associated real applications are approximately sparse in some wavelet transform domain. In fact, most 2D signals associated with physical phenomena (e.g., topographic maps) “nice” enough to be approximately sparse in some natural transform domain. In general, compressed sensing can be applied to signals which are approximately sparse in some known transform domain. One can compare sampling and reconstruction algorithms for CS in terms of their efficiency and complexity. The efficiency of a CS system increases when the number of samples required for “good” reconstruction decreases. A system has near-optimum efficiency if the number of samples required is only slightly larger than the theoretical minimum. For exactly sparse signals, it is now known that the theoretical minimum of one measurement per non-zero entry in the vector is asymptotically achievable [10, 11]. The complexity of a CS system is the computational complexity of signal reconstruction algorithm relative to the problem size. Implementation of the basis-pursuit algorithm based on a direct application of LP has roughly cubic complexity in the problem size [1], but this can reduced significantly. In particular, the message-passing reconstruction techniques are linear in the signal dimension [12, 13, 8].