# Patent

## Title
Computer-implemented method for correcting transmission errors using linear programming

## Abstract

A computer-implemented method for correcting transmission errors. According to the method, a transmitted vector corrupted by error can be recovered solving a linear program. The method has applications in the transmission of Internet media, Internet telephony, and speech transmission. In addition, error correction is embedded as a key building block in numerous algorithms, and data-structures, where corruption is possible; corruption of digital data stored on a hard-drive, CD, DVD or similar media is a good example. In short, progress in error correction has potential to impact several storage and communication systems.

## Background
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a U.S. Divisional application of Ser. No. 11/344,951, filed on Jan. 31, 2006, which, in turn, claims the benefit of U.S. Provisional Application No. 60/652,992, filed on Feb. 14, 2005, the contents of all of which are incorporated herein by reference in their entirety.


FEDERAL SUPPORT
This invention was made with U.S. Government support under grant No. DMS 01-40698 (FRG) awarded by the National Science Foundation. The U.S. Government has certain rights in this invention.


BACKGROUND
1. Field
The present disclosure relates to a method for correcting errors. In particular, it relates to a computer-implemented method for correcting transmission errors in a physical system.
2. Related Art
Finding sparse solutions to underdetermined systems of linear equations—a problem of great importance in signal processing, and design of robust communication channels—is in general NP-hard [9,21]. For example, the sparsest solution is given by




  
    
      (
      
        P
        0
      
      )
    
    ⁢
    
      
        min
        
          d
          ∈
          
            R
            m
          
        
      
      ⁢
      
        
          
            
            d
            
          
          
            l
            0
          
        
        ⁢
        
          
        
        ⁢
        subject
        ⁢
        
          
        
        ⁢
        to
        ⁢
        
          
        
        ⁢
        Fd
      
    
  
  =
  
    
      y
      ~
    
    ⁡
    
      (
      
        =
        Fe
      
      )
    
  




and solving this problem essentially requires exhaustive searches over all subsets of columns of F, a procedure which clearly is combinatorial in nature and has exponential complexity.

This computational intractability has recently led researchers to develop alternatives to (P0), and a frequently discussed approach considers a similar program in the l1-norm which goes by the name of Basis Pursuit [6]:




  
    
      (
      
        P
        1
      
      )
    
    ⁢
    
      
        min
        
          d
          ∈
          
            R
            m
          
        
      
      ⁢
      
        
          
            
            d
            
          
          
            l
            1
          
        
        ⁢
        
          
        
        ⁢
        subject
        ⁢
        
          
        
        ⁢
        to
        ⁢
        
          
        
        ⁢
        Fd
      
    
  
  =
  
    y
    ~
  




where we recall that ∥d∥l1=Σi=1m|di|. Unlike the l0-norm which enumerates the nonzero coordinates, the l1-norm is convex. It is also well-known [7] that (P1) can be recast as a linear program (LP).

Motivated by the problem of finding sparse decompositions of special signals in the field of mathematical signal processing and following upon the groundbreaking work of Donoho and Huo [11], a series of beautiful articles [16, 12, 13, 14, 24] showed exact equivalence between the two programs (P0) and (P1). In a nutshell, it has been shown that for m/2 by m matrices F obtained by concatenation of two orthonormal bases, the solution to both (P0) and (P1) are unique and identical provided that in the most favorable case, the vector e has at most 0.914√{square root over (m/2)} nonzero entries. This is of little practical use here since we are interested in procedures that might recover a signal when a constant fraction of the output is unreliable.
Using very different ideas and together with Romberg [3], the applicants proved that the equivalence holds with overwhelming probability for various types of random matrices provided that the number of nonzero entries in the vector e be of the order of m/log m [4, 5]. In the special case where F is an m/2 by m random matrix with independent standard normal entries, [9] proved that the number of nonzero entries may be as large as ρ·m, where ρ>0 is some very small and unspecified positive constant independent of m.
SUMMARY
The present application considers the model problem of recovering an input vector fεRn from corrupted measurements y=Af+e. Here, A is an m by n matrix (we will assume throughout that m>n), and e is an arbitrary and unknown vector of errors. The problem considered by the applicants is whether it is possible to recover f exactly from the data y. And if so, how?
In its abstract form, this problem is of course equivalent to the classical error correcting problem which arises in coding theory as A may be thought of as a linear code; a linear code is a given collection of codewords which are vectors a1, . . . , an εRm—the columns of the matrix A. Given a vector fεRn (the plaintext), a vector Af in Rm (the ciphertext) can be generated; we refer to the transformation from f to Af as the encoding scheme. If A has full rank, then one can clearly recover the plaintext f from the ciphertext Af by standard linear algebra methods. But now we suppose that the ciphertext Af is corrupted by an arbitrary vector eεRm giving rise to the corrupted ciphertext Af+e. The question is then: given the coding matrix A and Af+e, can one recover f exactly?
As is well-known, if the fraction of the corrupted entries is too large, then of course there is no hope of reconstructing f from Af+e; for instance, assume that m=2n and consider two distinct plaintexts f, f′ and form a vector gεRm by setting half of its m coefficients equal to those of Af and half of those equal to those of Af′. Then g=Af+e=Af′+e′ where both e and e′ are supported on sets of size at most n=m/2. This simple example shows that accurate decoding is impossible when the size of the support of the error vector is greater or equal to a half of that of the output Af. Therefore, a common assumption in the literature is to assume that only a small fraction of the entries are actually damaged:

∥e∥l0:=|{i: ei≠0}|≦ρ·m. 

For which values of ρ can we hope to reconstruct e with practical algorithms? That is, with algorithms whose complexity is at most polynomial in the length m of the code A? We refer to such algorithms as decoding schemes.
To decode f, note that it is obviously sufficient to reconstruct the vector e since knowledge of Af+e together with e gives Af, and consequently f since A has full rank. The decoding scheme of the present application is then as follows. The applicants construct a matrix which annihilates the m by n matrix A on the left, i.e. such that FA=0. This can be done by taking a matrix F whose kernel is the range of A in Rm, which is an n-dimensional subspace (e.g. F could be the orthogonal projection onto the cokernel of A). Then apply F to the output y=Af+e to obtain

{tilde over (y)}=F(Af+e)=Fe 

since FA=0. Therefore, the decoding problem is reduced to that of reconstructing a sparse vector e from the observations Fe (by sparse, the applicants mean that only a fraction of the entries of e are nonzero).

Therefore, the present application shows that a novel method that enables decoding of a transmitted vector of data in the presence of corruption of the original values.
According to a first aspect, a computer-implemented method for correcting transmission errors is provided, comprising the steps of: transmitting a vector using a linear code, said vector becoming corrupted by error upon transmission; and recovering the vector by solving a linear program in form of a convex optimization problem




  
    min
    
      x
      ∈
      
        R
        n
      
    
  
  ⁢
  
    
      
      
        y
        -
        Ax
      
      
    
    
      l
      1
      ′
    
  




wherein y is the corrupted vector, A is a coding matrix expressing the linear code, and l1 is a norm such that





  
    
      
      x
      
    
    
      l
      1
    
  
  :=
  
    
      ∑
      i
    
    ⁢
    
      
        
        
          x
          i
        
        
      
      .
    
  




According to a second aspect, a computer system for simulating a physical process is disclosed, comprising: means for storing in memory a received vector y, said received vector being corrupted by error; means for storing in memory a linear code A used to transmit said received vector; means for storing in memory a solution vector x; means for performing operations on A, x and y, the operations comprising performing




  
    min
    
      x
      ∈
      
        R
        n
      
    
  
  ⁢
  
    
      
      
        y
        -
        Ax
      
      
    
    
      l
      1
    
  




wherein l1 represents a norm such that





  
    
      
      x
      
    
    
      l
      1
    
  
  :=
  
    
      ∑
      i
      
        
      
    
    ⁢
    
      
    
    ⁢
    
      
      
        x
        i
      
      
    
  




According to a third aspect, a method for transmitting packets over a communication network is disclosed, comprising: encoding the packets using a coding matrix; transmitting the encoded packets; and decoding the transmitted encoded packets by solving a linear program




  
    min
    
      x
      ∈
      
        R
        n
      
    
  
  ⁢
  
    
      
      
        y
        -
        Ax
      
      
    
    
      l
      1
      
        ′
        
          
        
      
    
  




wherein A is the coding matrix, y is a vector expressing the transmitted encoded packets and l1 represents a norm such that





  
    
      
      x
      
    
    
      l
      1
    
  
  :=
  
    
      ∑
      i
      
        
      
    
    ⁢
    
      
    
    ⁢
    
      
        
        
          x
          i
        
        
      
      .
    
  




The present application is relevant in the design of practical communication networks (especially those of a digital nature) over noisy channels, as for instance arises in Internet, telephony, or electromagnetic communications. For instance, if a data vector is encoded using a random matrix A (thus expanding the length of the data, say by a factor of 4) and then each component of the ciphertext thus obtained is sent through the Internet as a separate packet, one can tolerate a fixed fraction (say 30%) of the packets being lost, corrupted, or delayed, and still be able to reconstruct the original data exactly. Of course, the matrix A has to be known for data recovery to work, but A can be generated by both transmitter and receiver through a standard random number generator with some previously agreed upon seed. It should be remarked that even if an adversary has the ability to choose which packets to destroy or corrupt, and what data to replace them with, this does not affect the ability to reconstruct the data so long as the adversary is only able to affect a fixed fraction of the packets. Also, as the protocol is non-adaptive, there is no need to query the transmitter at any stage in order to perform recovery; in particular, the data can be collected passively and decoded at some convenient later date. Note that because one is likely to recover the data in perfect condition, one can safely pre-process the data (e.g. by compressing it) even if such pre-processing is very sensitive to the change of even a single bit of the data.
The present application also has relevance to robust data storage and recovery protocols. Traditional error-correcting storage protocols, such as those employed on modern CDs and DVDs, allow one to tolerate small amounts of damage (such as scratches), and even large damage will corrupt only a fraction of the data. The algorithm presented here can achieve a similar performance (by dividing the data into blocks and applying the algorithm on each block separately), but can also scale to increase the robustness of the data, at the cost of speed of recovery. In the most extreme case, when the entire contents of a DVD are encoded as a single unit, (thus increasing reducing the capacity of the data by a factor of two to four, though this can be compensated for somewhat by first compressing the data), the data can then be recovered perfectly as long as no more than 10%-30% of the disk is damaged, though recovery in this case may take a significant amount of time (e.g. months) due the need to perform an immense amount of linear algebra. Again, the location of the damage is not important; deleting a consecutive sector of 30% of the disk can be handled just as easily as damage to 30% of the disk that is distributed randomly. Also, as before, one can safely pre-process the data in any desired manner.
Both the encoding and decoding aspects of the algorithm in accordance with the present disclosure can be easily and efficiently implemented in any computer architecture. The encoding scheme, being the application of a single linear matrix fAf, is extremely fast and can even be performed by very simple, low-power devices at high speeds. One can for instance imagine a “smart dust” application consisting of a large number of cheap devices measuring some data (represented mathematically as a vector, f), with each device recording and then transmitting a single component of Af, thus the sole purpose of each device is to perform and then transmit a single random linear measurement (together with some identifying code to signify which component of Af is being transmitted). Even if a significant fraction of these devices fail, are jammed, or are compromised to transmit false data, the algorithm will still be able to recover f perfectly. The decoding scheme, requiring the solution of a linear program, does require more computational power, but standard library packages can easily implement this program in an efficient manner, and for data of small size (e.g. vectors of at most 1024 entries in length) the decoding is essentially instantaneous. It is conceivable that in some cases where the data is of large size (e.g. several million entries), one could take advantage of parallel processing techniques to speed up the decoding.
The democratic nature of LP codes makes them particularly well-suited for distributing data to multiple recipients. Suppose, for example, that we wish to distribute a large piece of data from a satellite to many different users. The LP code for this data can be divided into small packets, which are in turn broadcast to the users. The quality of the channel between the satellite and each user could be very different, and could change over time, meaning that each user will successfully receive a different subset of the packets. Their ability to decode the data, however, depends only on the *number* of packets received, and not at all on *which* packets are received. Successful transmission is possible even though there is no feedback from any of the users about which packets are missing. Another possible application is robust distributed storage. Suppose a large file is LP coded, divided into packets, and then distributed over an array. If some of the disks fail, the data can still be retrieved from the subset of packets still available.



BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1, used to illustrate Lemma 8, shows the behaviour of the upper bound ρp/m(r) for the three values of the ratio p/m, namely p/m=3⁄4, 2⁄3, 1⁄2
FIGS. 2(a) and 2(b) show l1 recovery of an input signal from y=Af+e with A an m by n matrix with independent Gaussian entries. In this experiment, we ‘oversample’ the input signal by a factor of 2 so that m=2n. (a) Success rate of (P1) for m=512. (b) Success rate of (P1) for m=1024. Observe the similar pattern and cut-off point. In these experiments, exact recovery occurs as long as about 17% or less of the entries are corrupted.
FIG. 3 shows l1 recover of an input signal from y=Af+e with A an m by n matrix with independent Gaussian entries. In this experiment, the applicants ‘oversample’ the input signal by a factor 4, thus m=4n. In these experiments, exact recovery occurs as long as about 34% or less of the entries are corrupted.
FIG. 4 describes abstractly how the encoding and decoding scheme could be implemented in practice, starting with arbitrary data or data packets f (10) and then obtaining decoded data or data packets f# (20) which will equal the original data f if the fraction of transmitted data that has been corrupted is below a certain threshold. For instance, the transmission could be done over the Internet (30) (with errors coming from packet loss or malicious corruption of data), with one computer (40) performing the encoding operation fAf and another computer (50) performing the decoding operation (P1) or (P′1). Alternatively, the transmission could be via physical data storage (30) (to a hard disk, CD or DVD, with the error now arising from physical damage to the storage device), with encoding and decoding again performed by computer. A third possibility is with the encoding performed by a large number of cheap (and not necessarily reliable) low-power electronic measurement devices, the transmission conducted over open air using some electromagnetic frequency (and thus subject to interference or attenuation), and the data then collected by some receiver and decoded by a computer.
FIG. 5 shows a possible example of a computer (100) embodying the method in accordance with the present disclosure. The computer (100) comprises a memory (110) and a processor (120). The memory (110) can store a received vector corrupted by error a liner code used to transmit the received vector, and a solution vector. The processor (120) can perform operations on the stored variables by way of the method described herein.



DETAILED DESCRIPTION
The present application introduces the concept of a restrictedly almost orthonormal system—a collection of vectors which behaves like an almost orthonormal system but only for sparse linear combinations. Thinking about these vectors as the columns of the matrix F, the applicants show that this condition allows for the exact reconstruction of sparse linear combination of these vectors, i.e. e. The applicants' results are significantly different than those mentioned above as they are deterministic and do not involve any kind of randomization, although they can of course be specialized to random matrices. For instance, it shall be shown that a Gaussian matrix with independent entries sampled from the standard normal distribution is restrictedly almost orthonormal with overwhelming probability, and that minimizing the l1-norm recovers sparse decompositions with a number of nonzero entries of size ρ0·m; the applicants shall actually give numerical values for ρ0.
The applicants presented the connection with sparse solutions to underdetermined systems of linear equations merely for pedagogical reasons. There is a more direct approach. To decode f from corrupted data y=Af+e, we consider solving the following l1-minimization problem




  
    (
    
      
        P
        ⁢
        
          
        
      
      1
      ′
    
    )
  
  ⁢
  
    
      min
      
        g
        ∈
        
          R
          n
        
      
    
    ⁢
    
      
        
          
          
            y
            -
            Ag
          
          
        
        
          l
          1
        
      
      .
    
  




Now f is the unique solution of (P1′) if and only if e is the unique solution of (P1). In other words, (P1) and (P1′) are equivalent programs. To see why these is true, observe on the one hand that since y=Af+e, we may decompose g as g=f+h so that





  
    (
    
      
        P
        ⁢
        
          
        
      
      1
      ′
    
    )
  
  ⇔
  
    
      min
      
        h
        ∈
        
          R
          n
        
      
    
    ⁢
    
      
        
          
          
            e
            -
            Ah
          
          
        
        
          l
          1
        
      
      .
    
  




On the other hand, the constraint Fx=Fe means that x=e−Ah for some hεRn and, therefore,




  
    
      (
      
        
          P
          ⁢
          
            
          
        
        1
        ′
      
      )
    
    ⇔
    
      
        min
        
          h
          ∈
          
            R
            n
          
        
      
      ⁢
      
        
          
          x
          
        
        
          l
          1
        
      
    
  
  ,
  
    

  
  ⁢
  
    x
    =
    
      
        e
        -
        Ah
      
      ⇔
      
        
          min
          
            h
            ∈
            
              R
              n
            
          
        
        ⁢
        
          
            
            
              e
              -
              Ah
            
            
          
          
            l
            1
            
              ′
              
                
              
            
          
        
      
    
  




which proves the claim.

The decoding procedure (P1′) may also be re-expressed as a Linear Program (LP)—hence the title of the present application. Indeed, the l1-minimization problem is equivalent to

min1Tt, −t≦y−Ag≦t,   (1)

where the optimization variables are tεRm and gεRn (as is standard, the generalized vector inequality x≦y means that xi≦yi for all i). As a result, the decoding procedure (P1′) is a LP with inequality constraints and can be solved efficiently using standard optimization algorithms, see [2].

Restricted Isometries

In the remainder of the present application, it will be convenient to use some linear algebra notations. We denote by (vj)jεJεRp the columns of the matrix F and by H the Hilbert space spanned by these vectors. Further, for any TJ, we let FT be the submatrix with column indices jεT so that




  
    
      F
      r
    
    ⁢
    c
  
  =
  
    
      
        ∑
        
          j
          ∈
          T
        
        
          
        
      
      ⁢
      
        
      
      ⁢
      
        
          c
          j
        
        ⁢
        
          v
          j
        
      
    
    ∈
    
      H
      .
    
  




To introduce the notion of an almost orthonormal system, the applicants first observe that if the columns of F are sufficiently “degenerate”, the recovery problem cannot be solved. In particular, if there exists a non-trivial sparse linear combination




  
    
      ∑
      
        j
        ∈
        T
      
      
        
      
    
    ⁢
    
      
        c
        j
      
      ⁢
      
        v
        j
      
    
  
  =
  
    
  
  ⁢
  0




of the vj which sums to zero, and T=T1∪T2 is any partition of T into two disjoint sets, then the vector





  y
  :=
  
    
      
        ∑
        
          j
          ∈
          
            T
            1
          
        
        
          
        
      
      ⁢
      
        
      
      ⁢
      
        
          c
          j
        
        ⁢
        
          v
          j
        
      
    
    =
    
      
        ∑
        
          j
          ∈
          
            T
            2
          
        
        
          
        
      
      ⁢
      
        
          (
          
            -
            
              c
              j
            
          
          )
        
        ⁢
        
          v
          j
        
      
    
  




has two distinct sparse representations. On the other hand, linear dependencies





  
    
      ∑
      
        j
        ∈
        J
      
      
        
      
    
    ⁢
    
      
        c
        j
      
      ⁢
      
        v
        j
      
    
  
  =
  
    
  
  ⁢
  0




which involve a large number of nonzero coefficients cj, as opposed to a sparse set of coefficients, do not present an obvious obstruction to sparse recovery. At the other extreme, if the (vj)jεJ are an orthonormal system, then the recovery problem is easily solved by setting cj:=f, vj

In present application, if a “restricted orthonormality hypothesis”, which is far weaker than assuming orthonormality, is imposed, then (P1) solves the recovery problem, even if the (vj)jεJ are highly linearly dependent (for instance, it is possible for m:=|J| to be much larger than the dimension of the span of the vj's). To make this quantitative the following definition is introduced.

    Definition. (Restricted isometry constants) Let F be the matrix with the finite collection of vectors (vj)jεJ εRp as columns. For every integer 1≦S≦|J|, we define the S-restricted isometry constants δs to be the smallest quantity such that FT obeys

(1−δs)∥c∥2≦∥FTc∥2≦(1+δS)∥c∥2   (2)

for all subsets T⊂J of cardinality at most S, and all real coefficients (cj)jεT. Similarly, we define the (S,S′)-restricted orthogonality constants θS,S′ for S+S′≦|J| to be the smallest quantity such that

|FTc, FT′c′|≦θS,S′·∥c∥∥c′∥  (3)

holds for all disjoint sets T, T′J of cardinalities |T|≦S and |T′|≦S′.



The numbers δS and θS,S′ measure how close the vectors vi are to behaving like an orthonormal system, but only when restricting attention to sparse linear combinations involving no more than S vectors. These numbers are clearly non-decreasing in S, S′. For S=1, the value δ1 only conveys magnitude information about the vectors vj; indeed δ1 is the best constant such that

1−δ1≦∥vj∥H2≦1+δ1 for all jεJ. 

In particular, δ1=0 if and only if all of the vj have unit length. We will later show that the higher δS control the orthogonality numbers θS,S′:

    Lemma 1. We have θS,S′≦δS+S′≦θS,S,′+max(δS, δS′) for all S, S′.


To see the relevance of the restricted isometry numbers δS to the sparse recovery problem, consider the following simple observation:

    Lemma 2. Suppose that S≧1 is such that δ2S<1, and let T⊂J be such that |T|≦S. Let f:=FTc for some arbitrary |T|-dimensional vector c. Then the set T and the coefficients (cj)jεT can be reconstructed uniquely from knowledge of the vector f and the vj's.
    Proof. We prove that there is a unique c with ∥c∥l0≦S obeying






  f
  =
  
    
      ∑
      j
      
        
      
    
    ⁢
    
      
    
    ⁢
    
      
        c
        j
      
      ⁢
      
        
          v
          j
        
        .
      
    
  




Suppose for contradiction that f had two distinct sparse representations f=FTc=FT′c′ where |T|, |T′|≦S. Then

FT∪T′d=0, dj:=cj1jεT−c′j′1jεT′.

Taking norms of both sides and applying (2) and the hypothesis δ2S<1 we conclude that ∥d∥2=0, contradicting the hypothesis that the two representations were distinct.

Main Results

The previous lemma is an abstract existence argument which shows what might theoretically be possible, but does not supply any efficient algorithm to recover T and cj from f and the vj other than by brute force search—as discussed earlier. In contrast, the main result is that, by imposing slightly stronger conditions on the restricted isometry constants, the l1-minimization program (P1) decodes f exactly.

    Theorem 3. Suppose that S≧1 is such that

δS+θS+θS,2S<1,   (4)

and let c be a real vector supported on a set T⊂J obeying |T|≦S. Put f:=Fc. Then c is the unique minimizer to

(P1)min∥d∥l1; Fd=f. 



Note from Lemma 1 that (4) implies that δ2S<1, and is in turn implied by δS+δ2S+δ3S<1⁄4. Thus the condition (4) is roughly “three times as strict” as the condition required for Lemma 2.
Theorem 3 is inspired by the applicants' previous work [4], see also [3, 5], but unlike those earlier results, the applicants' results here are deterministic, and thus do not have a non-zero probability of failure, provided of course one can ensure that the vj verify the condition (4). By virtue of the previous discussion, we have the companion result:

    Theorem 4. Suppose F is such that FA=0 and let S≧1 be a number obeying the hypothesis of Theorem 3. Set y=Af+e, where e is a real vector supported on a set of size at most S. Then f is the unique minimizer to






  
    (
    
      P
      1
      ′
    
    )
  
  ⁢
  
    
      min
      
        g
        ∈
        
          R
          n
        
      
    
    ⁢
    
      
        
          
          
            y
            -
            Ag
          
          
        
        
          l
          1
        
      
      .
    
  




Gaussian Random Matrices

An important question is to find matrices with good restricted isometry constants, i.e. such that (4) holds for large values of S. Indeed, such matrices will tolerate a larger fraction of output in error while still allowing exact recovery of the original input by linear programming. How to construct such matrices might be delicate. In a later section, however, the applicants will argue that generic matrices, namely samples from the Gaussian unitary ensemble obey (4) for relatively large values of S.

    Theorem 5. Assume p≦m and let F be a p by m matrix whose entries are i.i.d. Gaussian with mean zero and variance 1/p. Then the condition of Theorem 3 holds with overwhelming probability provided that r=S/m is small enough so that

r<r*(p,m)

where r*(p,m) is given in (12). (By “with overwhelming probability”, we mean with probability decaying exponentially in m.) In the limit of large samples, r*(p,m) only depends upon the ratio, and numerical evaluations show that the condition holds for r≦3.6×10−4 in the case where p/m=3⁄4, r≦3.2×10−4 when p/m=2⁄3, and r≦2.3×10−4 when p/m=1⁄2.



In other words, Gaussian matrices are a class of matrices for which one can solve an underdetermined systems of linear equations by minimizing l1 provided, of course, the input vector has fewer than ρ·m nonzero entries with ρ>0. The applicants mentioned earlier that this result is similar to [9]. What is new and non-obvious here is that by using a very different machinery, one can obtain explicit numerical values which were not available before.
In the context of error correcting, the consequence is that a fraction of the output may be corrupted by arbitrary errors and yet, solving a convex problem would still recover f exactly—a rather unexpected feat.

    Corollary. Suppose A is an n by m Gaussian matrix and set p=m−n. Under the hypotheses of Theorem 5, the solution to (P1′) is unique and equal to f.


This is an immediate consequence of Theorem 5. The only thing we need to argue is why we may think of the annihilator F (such that FA=0) as a matrix with independent Gaussian entries. Observe that the range of A is a random space of dimension n embedded in Rm so that the data {tilde over (y)}=Fe is the projection of e on a random space of dimension p. The range of a p by m matrix with independent Gaussian entries precisely is a random subspace of dimension p, which justifies the claim.
The applicants would like to point out that the theoretical bounds derived in the present application are overly conservative; numerical experiments have shown that the error tolerance that the algorithm can handle in fact exceeds the theoretical bounds by one or two orders of magnitude.
Proof of Main Results
The main result, namely, Theorem 3 is proved by duality. As it will be seen in a later section, c is the unique minimizer if the matrix FT has full rank and if one can find a vector w with the two properties
1. w, vj=sgn(cj) for all jεT,
2. and w, vj<1 for all j∉T,
where sgn(cj) is the sign of cj (it equals zero when cj=0). The two conditions 1, 2 above say that a specific dual program is feasible and is called the exact reconstruction property in [4]}, see also [3]. For |T|≦S with S obeying the hypothesis of Theorem 3, FT has full rank since δS<1 and thus, the proof simply consists in constructing a dual vector w; this is the object of the next section.

Exact Reconstruction Property

The applicants now examine the sparse reconstruction property and begin with coefficients w, vj for j∉T being only small in an l2 sense.

    Lemma 6. (Dual sparse reconstruction property, l2 version)

Let S, S′≧1 be such that δS<1, and c be a real vector supported on T⊂J such that |T|≦S. Then there exists a vector wεH such that w, vj=cj for all jεT. Furthermore, there is an “exceptional set” EJ which is disjoint from T, of size at most |E|≦S′, and with the properties






  
    
      
        
        
          〈
          
            w
            ,
            
              v
              j
            
          
          〉
        
        
      
      ≤
      
        
          
            
              θ
              
                S
                ,
                
                  S
                  ′
                
              
            
            
              
                (
                
                  1
                  -
                  
                    δ
                    S
                  
                
                )
              
              ⁢
              
                
                  S
                  ′
                
              
            
          
          .
          
            
            c
            
          
        
        ⁢
        
          
            
          
          ⁢
          
            
          
        
        ⁢
        for
        ⁢
        
          
        
        ⁢
        all
        ⁢
        
          
        
        ⁢
        j
      
    
    ∉
    
      T
      ⋃
      E
    
  




  
    
      
        and
        ⁢
        
          
            
          
          ⁢
          
            
          
        
        (
        
          
            ∑
            
              j
              ∈
              E
            
            
              
            
          
          ⁢
          
            
          
          ⁢
          
            
              
              
                〈
                
                  w
                  ,
                  
                    v
                    j
                  
                
                〉
              
              
            
            2
          
        
        )
      
      
        1
        /
        2
      
    
    ≤
    
      
        
          θ
          S
        
        
          1
          -
          
            δ
            S
          
        
      
      ·
      
        
          
          c
          
        
        .
      
    
  



In addition, ∥w∥H≦K·∥c∥ for some constant K>0 only depending upon δS.

    Proof. Recall that FT:l2(T)→H is the linear transformation






  
    
      F
      T
    
    ⁢
    
      c
      T
    
  
  :=
  
    
      ∑
      
        j
        ∈
        T
      
      
        
      
    
    ⁢
    
      
    
    ⁢
    
      
        c
        j
      
      ⁢
      
        v
        j
      
    
  




where cT:=(cj)jεT (we use the subscript T in cT to emphasize that the input is a |T|-dimensional vector), and let F*T be the adjoint transformation

F*Tw:=(w, vj)jεT.

Property 1 gives

1−δS≦λmin(F*TFT)≦λmax(F*TFT)≦1+δS,

where λmin and λmax are the minimum and maximum eigenvalues of the positive-definite operator F*TFT. In particular, since δ|T|<1, we see that F*TFT is invertible with





  
    
      
        
          
          
            
              (
              
                
                  F
                  T
                  *
                
                ⁢
                
                  F
                  T
                
              
              )
            
            
              -
              1
            
          
          
        
        ≤
        
          
            1
            
              1
              -
              
                δ
                s
              
            
          
          .
        
      
    
    
      
        (
        5
        )
      
    
  




Also note that





  
    
    
      
        
          F
          T
        
        ⁡
        
          (
          
            
              F
              T
              *
            
            ⁢
            
              F
              T
            
          
          )
        
      
      
        -
        1
      
    
    
  
  ≤
  
    
      
        1
        +
        
          δ
          s
        
      
    
    
      1
      -
      
        δ
        s
      
    
  




and set wεH to be the vector

w:=FT(F*TFT)−1cT;

it is then clear that F*Tw=cT, i.e. w, vj=cj for all jεT. In addition, ∥w∥≦K·∥cT∥ with





  K
  :=
  
    
      
        
          1
          +
          
            δ
            s
          
        
      
      
        1
        -
        
          δ
          s
        
      
    
    .
  




Finally, if T′ is any set in J disjoint from T with |T′|≦S′ and dT′=(dj)jεT′ is any sequence of real numbers, then (3) and (5) give





  
    
      
      
        
          〈
          
            
              
                F
                
                  T
                  ′
                
                *
              
              ⁢
              w
            
            ,
            
              d
              
                T
                ′
              
            
          
          〉
        
        
          
            l
            2
          
          ⁡
          
            (
            
              T
              ′
            
            )
          
        
      
      
    
    =
    
      
        
        
          
            〈
            
              w
              ,
              
                
                  F
                  
                    T
                    ′
                  
                  *
                
                ⁢
                
                  d
                  
                    T
                    ′
                  
                
              
            
            〉
          
          
            
              l
              2
            
            ⁡
            
              (
              
                T
                ′
              
              )
            
          
        
        
      
      =
      
        
          
          
            
              〈
              
                
                  
                    ∑
                    
                      j
                      ∈
                      T
                    
                  
                  ⁢
                  
                    
                  
                  ⁢
                  
                    
                      
                        (
                        
                          
                            
                              (
                              
                                
                                  F
                                  T
                                  *
                                
                                ⁢
                                
                                  F
                                  T
                                
                              
                              )
                            
                            
                              -
                              1
                            
                          
                          ⁢
                          
                            c
                            T
                          
                        
                        )
                      
                      j
                    
                    ⁢
                    
                      v
                      j
                    
                  
                
                ,
                
                  
                    ∑
                    
                      j
                      ∈
                      
                        T
                        ′
                      
                    
                  
                  ⁢
                  
                    
                      d
                      j
                    
                    ⁢
                    
                      v
                      j
                    
                  
                
              
              〉
            
            H
          
          
        
        ≤
        
          
            θ
            
              s
              ,
              
                s
                ′
              
            
          
          ·
          
            
            
              
                
                  (
                  
                    
                      F
                      T
                      *
                    
                    ⁢
                    
                      F
                      T
                    
                  
                  )
                
                
                  -
                  1
                
              
              ⁢
              
                c
                T
              
            
            
          
          ·
          
            
            
              d
              
                T
                ′
              
            
            
          
        
        ≤
        
          
            
              θ
              
                s
                ,
                
                  s
                  ′
                
              
            
            
              1
              -
              
                δ
                s
              
            
          
          ⁢
          
            
              
              
                c
                T
              
              
            
            ·
            
              
              
                d
                
                  T
                  ′
                
              
              
            
          
        
      
    
  
  ;




since dT′ was arbitrary, we thus see from duality that





  
    
      
      
        
          F
          
            T
            ′
          
          *
        
        ⁢
        w
      
      
    
    
      
        l
        2
      
      ⁡
      
        (
        
          T
          ′
        
        )
      
    
  
  ≤
  
    
      
        θ
        
          s
          ,
          
            s
            ′
          
        
      
      
        1
        -
        
          δ
          s
        
      
    
    ⁢
    
      
        
        
          c
          T
        
        
      
      .
    
  




In other words,





  
    
      
        
          
            
              (
              
                
                  ∑
                  
                    j
                    ∈
                    
                      T
                      ′
                    
                  
                
                ⁢
                
                  
                    
                    
                      〈
                      
                        w
                        ,
                        
                          v
                          j
                        
                      
                      〉
                    
                    
                  
                  2
                
              
              )
            
            
              1
              /
              2
            
          
          ≤
          
            
              
                θ
                
                  s
                  ,
                  
                    s
                    ′
                  
                
              
              
                1
                -
                
                  δ
                  s
                
              
            
            ⁢
            
              
              
                c
                T
              
              
            
            ⁢
            
              
            
            ⁢
            whenever
          
        
        ⁢
        
          

        
        ⁢
        
          
            T
            ′
          
          ⊆
          
            
              
                J
                ∖
                T
              
              ⁢
              
                
              
              ⁢
              and
              ⁢
              
                
              
              ⁢
              
                
                
                  T
                  ′
                
                
              
            
            ≤
            
              
                S
                ′
              
              .
            
          
        
      
    
    
      
        (
        6
        )
      
    
  




If in particular if we set





  
    E
    :=
    
      {
      
        j
        ∈
        
          J
          ∖
          
            T
            :
            
              
                
                
                  〈
                  
                    w
                    ,
                    
                      v
                      j
                    
                  
                  〉
                
                
              
              >
              
                
                  
                    θ
                    
                      s
                      ,
                      
                        s
                        ′
                      
                    
                  
                  
                    
                      (
                      
                        1
                        -
                        
                          δ
                          s
                        
                      
                      )
                    
                    ⁢
                    
                      
                        S
                        ′
                      
                    
                  
                
                ·
                
                  
                  
                    c
                    T
                  
                  
                
              
            
          
        
      
      }
    
  
  ,




then |E| must obey |E|≦S′, since otherwise we could contradict (6) by taking a subset T′ of E of cardinality S′. The claims now follow.

The applicants now derive a solution with better control on the sup norm of |w, vj| outside of T, by iterating away the exceptional set E (while keeping the values on T fixed).

    Lemma 7. (Dual sparse reconstruction property, l∞ version)

Let S≧1 be such that δS+θS,2S<1, and c be a real vector supported on T⊂J obeying |T|≦S Then there exists a vector wεH such that w, vj=cj for all jεT. Furthermore, w obeys







  
    
      
        
          
            
            
              
                〈
                
                  w
                  ,
                  
                    v
                    j
                  
                
                〉
              
              H
            
            
          
          ≤
          
            
              
                
                  θ
                  s
                
                
                  
                    (
                    
                      1
                      -
                      
                        δ
                        s
                      
                      -
                      
                        θ
                        
                          s
                          ,
                          
                            2
                            ⁢
                            
                              
                            
                            ⁢
                            s
                          
                        
                      
                    
                    )
                  
                  ⁢
                  
                    S
                  
                
              
              ·
              
                
                c
                
              
            
            ⁢
            
              
            
            ⁢
            for
            ⁢
            
              
            
            ⁢
            all
            ⁢
            
              
            
            ⁢
            j
          
        
        ∉
        T
      
    
    
      
        (
        7
        )
      
    
  




    Proof. We may normalize






  
    
      ∑
      
        j
        ∈
        T
      
    
    ⁢
    
      
        
        
          c
          j
        
        
      
      2
    
  
  =
  
    
      S
    
    .
  




Write T0:=T. Using Lemma 6, we can find a vector w1εH and a set T1J such that




  
    
      
        T
        0
      
      ⋂
      
        T
        1
      
    
    =
    Ø
  




  
    
      
      
        T
        1
      
      
    
    ≤
    S
  




  
    
      〈
      
        
          w
          1
        
        ,
        
          v
          j
        
      
      〉
    
    =
    
      
        
          c
          j
        
        ⁢
        
          
        
        ⁢
        for
        ⁢
        
          
        
        ⁢
        all
        ⁢
        
          
        
        ⁢
        j
      
      ∈
      
        T
        0
      
    
  




  
    
      
        
        
          
            〈
            
              
                w
                1
              
              ,
              
                v
                j
              
            
            〉
          
          H
        
        
      
      ≤
      
        
          
            θ
            
              s
              ,
              
                s
                ′
              
            
          
          
            1
            -
            
              δ
              s
            
          
        
        ⁢
        
          
        
        ⁢
        for
        ⁢
        
          
        
        ⁢
        all
        ⁢
        
          
        
        ⁢
        j
      
    
    ∉
    
      
        
          T
          0
        
        ⋃
        
          
            
              T
              1
            
            ⁢
            
              

            
            (
            
              
                ∑
                
                  j
                  ∈
                  
                    T
                    1
                  
                
              
              ⁢
              
                
                  
                  
                    
                      〈
                      
                        
                          w
                          1
                        
                        ,
                        
                          v
                          j
                        
                      
                      〉
                    
                    H
                  
                  
                
                2
              
            
            )
          
          
            1
            /
            2
          
        
      
      ≤
      
        
          
            θ
            s
          
          
            1
            -
            
              δ
              s
            
          
        
        ⁢
        
          S
        
      
    
  




  
    
      
        
        
          w
          1
        
        
      
      H
    
    ≤
    
      K
      .
    
  



Applying Lemma 6 iteratively gives a sequence of vectors wn+1εH and sets Tn+1J for all n≧1 with the properties




  
    
      
        T
        n
      
      ⋂
      
        (
        
          
            T
            0
          
          ⋃
          
            T
            
              n
              +
              1
            
          
        
        )
      
    
    =
    Ø
  




  
    
      
      
        T
        
          n
          +
          1
        
      
      
    
    ≤
    S
  




  
    
      
        〈
        
          
            w
            
              n
              +
              1
            
          
          ,
          
            v
            j
          
        
        〉
      
      H
    
    =
    
      
        
          
            〈
            
              
                w
                n
              
              ,
              
                v
                j
              
            
            〉
          
          H
        
        ⁢
        
          
        
        ⁢
        for
        ⁢
        
          
        
        ⁢
        all
        ⁢
        
          
        
        ⁢
        j
      
      ∈
      
        T
        n
      
    
  




  
    
      
        〈
        
          
            w
            
              n
              +
              1
            
          
          ,
          
            v
            j
          
        
        〉
      
      H
    
    =
    
      
        0
        ⁢
        
          
        
        ⁢
        for
        ⁢
        
          
        
        ⁢
        all
        ⁢
        
          
        
        ⁢
        j
      
      ∈
      
        T
        0
      
    
  




  
    
      
        
        
          
            〈
            
              
                w
                
                  n
                  +
                  1
                
              
              ,
              
                v
                j
              
            
            〉
          
          H
        
        
      
      ≤
      
        
          
            θ
            s
          
          
            1
            -
            
              δ
              s
            
          
        
        ⁢
        
          
        
        ⁢
        
          
            (
            
              
                θ
                
                  s
                  ,
                  
                    2
                    ⁢
                    
                      
                    
                    ⁢
                    s
                  
                
              
              
                1
                -
                
                  δ
                  s
                
              
            
            )
          
          n
        
        ⁢
        
          
        
        ⁢
        for
        ⁢
        
          
        
        ⁢
        all
        ⁢
        
          
        
        ⁢
        j
      
    
    ∉
    
      
        
          T
          0
        
        ⋃
        
          T
          n
        
        ⋃
        
          
            
              T
              
                n
                +
                1
              
            
            ⁢
            
              

            
            (
            
              
                ∑
                
                  j
                  ∈
                  
                    T
                    
                      n
                      +
                      1
                    
                  
                
              
              ⁢
              
                
                  
                  
                    
                      〈
                      
                        
                          w
                          
                            n
                            +
                            1
                          
                        
                        ,
                        
                          v
                          j
                        
                      
                      〉
                    
                    H
                  
                  
                
                2
              
            
            )
          
          
            1
            /
            2
          
        
      
      ≤
      
        
          
            θ
            s
          
          
            1
            -
            
              δ
              s
            
          
        
        ⁢
        
          
            (
            
              
                θ
                
                  s
                  ,
                  
                    2
                    ⁢
                    
                      
                    
                    ⁢
                    s
                  
                
              
              
                1
                -
                
                  δ
                  s
                
              
            
            )
          
          n
        
        ⁢
        
          S
        
      
    
  




  
    
      
        
        
          w
          
            n
            +
            1
          
        
        
      
      H
    
    ≤
    
      
        
          (
          
            
              θ
              s
            
            
              1
              -
              
                δ
                s
              
            
          
          )
        
        
          n
          -
          1
        
      
      ⁢
      
        K
        .
      
    
  



By hypothesis, we have





  
    
      θ
      
        s
        ,
        
          2
          ⁢
          
            
          
          ⁢
          s
        
      
    
    
      1
      -
      
        δ
        s
      
    
  
  ≤
  1.




Thus if we set





  w
  :=
  
    
      ∑
      
        n
        =
        1
      
      ∞
    
    ⁢
    
      
    
    ⁢
    
      
        
          (
          
            -
            1
          
          )
        
        
          n
          -
          1
        
      
      ⁢
      
        w
        n
      
    
  




then the series is absolutely convergent and, therefore, w is a well-defined vector in H. We now study the coefficients





  
    
      
        
          
            〈
            
              w
              ,
              
                v
                j
              
            
            〉
          
          H
        
        =
        
          
            ∑
            
              n
              =
              1
            
            ∞
          
          ⁢
          
            
              
                (
                
                  -
                  1
                
                )
              
              
                n
                -
                1
              
            
            ⁢
            
              
                〈
                
                  
                    w
                    n
                  
                  ,
                  
                    v
                    j
                  
                
                〉
              
              H
            
          
        
      
    
    
      
        (
        8
        )
      
    
  




for jεJ.

Consider first jεT0, it follows from the construction that w1, vj=cj and wn, vj=0 for all n>1, and hence

w, vj=cj for all jεT0.

Second, fix j with j∉T0 and let Ij:={n≧1:jεTn}. Since Tn and Tn+1 are disjoint, we see that the integers in the set Ij are spaced at least two apart. Now if nεIj, then by definition jεTn and, therefore,

wn+1, vj=wn, vj

In other words, the n and n+1 terms in (8) cancel each other out. Thus we have





  
    
      〈
      
        w
        ,
        
          v
          j
        
      
      〉
    
    H
  
  =
  
    
      ∑
      
        
          n
          ≥
          1
        
        
          n
          ,
          
            
              n
              -
              1
            
            ∉
            
              I
              j
            
          
        
      
      ∞
    
    ⁢
    
      
        
          (
          
            -
            1
          
          )
        
        
          n
          -
          1
        
      
      ⁢
      
        
          
            〈
            
              
                w
                n
              
              ,
              
                v
                j
              
            
            〉
          
          H
        
        .
      
    
  




On the other hand, if n,n−1∉Ij and n≠0, then j∉Tn∩Tn−1 and





  
    
    
      〈
      
        
          w
          n
        
        ,
        
          v
          j
        
      
      〉
    
    
  
  ≤
  
    
      
        θ
        S
      
      
        1
        -
        
          δ
          S
        
      
    
    ⁢
    
      
        (
        
          
            θ
            
              S
              ,
              
                2
                ⁢
                S
              
            
          
          
            1
            -
            
              δ
              S
            
          
        
        )
      
      
        n
        -
        1
      
    
  




which by the triangle inequality and the geometric series formula gives





  
    
    
      
        ∑
        
          
            n
            ≥
            1
          
          
            n
            ,
            
              
                n
                -
                1
              
              ∉
              
                I
                j
              
            
          
        
      
      ⁢
      
        
          
            (
            
              -
              1
            
            )
          
          
            n
            -
            1
          
        
        ⁢
        
          
            〈
            
              
                w
                n
              
              ,
              
                v
                j
              
            
            〉
          
          H
        
      
    
    
  
  ≤
  
    
      
        θ
        S
      
      
        1
        -
        
          δ
          S
        
        -
        
          θ
          
            S
            ,
            
              2
              ⁢
              S
            
          
        
      
    
    .
  




In conclusion,





  
    
      
      
        
          
            〈
            
              w
              ,
              
                v
                j
              
            
            〉
          
          H
        
        -
        
          
            1
            
              0
              ∈
              
                I
                j
              
            
          
          ⁢
          
            
              〈
              
                
                  w
                  0
                
                ,
                
                  v
                  j
                
              
              〉
            
            H
          
        
      
      
    
    ≤
    
      
        θ
        S
      
      
        1
        -
        
          δ
          S
        
        -
        
          θ
          
            S
            ,
            
              2
              ⁢
              S
            
          
        
      
    
  
  ,




and since |T|≦S, the claim follows.

Lemma 7 actually solves the dual recovery problem. Indeed, the applicants' result states that one can find a vector wεH obeying both properties 1 and 2 stated at the beginning of the section. To see why 2 holds, observe that ∥sgn(c)∥=√{square root over (|T|)}≦√{square root over (S)} and, therefore, (7) gives for all j∉T




  
    
    
      
        〈
        
          w
          ,
          
            v
            j
          
        
        〉
      
      H
    
    
  
  ≤
  
    
      
        θ
        S
      
      
        1
        -
        
          δ
          S
        
        -
        
          θ
          
            S
            ,
            
              2
              ⁢
              S
            
          
        
      
    
    ·
    
      
        
          
          T
          
        
        S
      
    
  
  ≤
  
    
      θ
      S
    
    
      1
      -
      
        δ
        S
      
      -
      
        θ
        
          S
          ,
          
            2
            ⁢
            S
          
        
      
    
  
  <
  1




provided that (4) holds.

Proof of Theorem 3

Observe first that standard convex arguments give that there exists at least one minimizer d=(dj)jεJ to the problem (P1). We need to prove that d=c. Since c obeys the constraints of this problem, d obeys




  
    
      
        
          
            
              
              d
              
            
            
              l
              1
            
          
          ≤
          
            
              
              c
              
            
            
              l
              1
            
          
        
        =
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
              
              
                c
                j
              
              
            
            .
          
        
      
    
    
      
        (
        9
        )
      
    
  




Now take a w obeying properties 1 and 2 (see the remark following Lemma 7). Using the fact that the inner product w, vj is equal to the sign of c on T and has absolute value strictly less than one on the complement, we then compute




  
    
      
      d
      
    
    
      l
      1
    
  
  =
  
    
      
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
            
              
                c
                j
              
              +
              
                (
                
                  
                    d
                    j
                  
                  -
                  
                    c
                    j
                  
                
                )
              
            
            
          
        
        +
        
          
            ∑
            
              j
              ∉
              T
            
          
          ⁢
          
            
            
              d
              j
            
            
          
        
      
      ≥
      
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
              sgn
              ⁡
              
                (
                
                  c
                  j
                
                )
              
            
            ⁢
            
              (
              
                
                  c
                  j
                
                +
                
                  (
                  
                    
                      d
                      j
                    
                    -
                    
                      c
                      j
                    
                  
                  )
                
              
              )
            
          
        
        +
        
          
            ∑
            
              j
              ∉
              T
            
          
          ⁢
          
            
              d
              j
            
            ⁢
            
              
                〈
                
                  w
                  ,
                  
                    v
                    j
                  
                
                〉
              
              H
            
          
        
      
    
    =
    
      
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
            
              c
              j
            
            
          
        
        +
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
              (
              
                
                  d
                  j
                
                -
                
                  c
                  j
                
              
              )
            
            ⁢
            
              
                〈
                
                  w
                  ,
                  
                    v
                    j
                  
                
                〉
              
              H
            
          
        
        +
        
          
            ∑
            
              j
              ∉
              T
            
          
          ⁢
          
            
              d
              j
            
            ⁢
            
              
                〈
                
                  w
                  ,
                  
                    v
                    j
                  
                
                〉
              
              H
            
          
        
      
      =
      
        
          
            
              ∑
              
                j
                ∈
                T
              
            
            ⁢
            
              
              
                c
                j
              
              
            
          
          +
          
            
              〈
              
                w
                ,
                
                  
                    
                      ∑
                      
                        j
                        ∈
                        J
                      
                    
                    ⁢
                    
                      
                        d
                        j
                      
                      ⁢
                      
                        v
                        j
                      
                    
                  
                  -
                  
                    
                      ∑
                      
                        j
                        ∈
                        T
                      
                    
                    ⁢
                    
                      
                        c
                        j
                      
                      ⁢
                      
                        v
                        j
                      
                    
                  
                
              
              〉
            
            H
          
        
        =
        
          
            
              
                ∑
                
                  j
                  ∈
                  T
                
              
              ⁢
              
                
                
                  c
                  j
                
                
              
            
            +
            
              
                〈
                
                  w
                  ,
                  
                    f
                    -
                    f
                  
                
                〉
              
              H
            
          
          =
          
            
              ∑
              
                j
                ∈
                T
              
            
            ⁢
            
              
                
                
                  c
                  j
                
                
              
              .
            
          
        
      
    
  




Comparing this with (9) we see that all the inequalities in the above computation must in fact be equality. Since w, vj was strictly less than 1 for all j∉T, this in particular forces dj=0 for all j∉T. Thus




  
    
      ∑
      
        j
        ∈
        T
      
    
    ⁢
    
      
        (
        
          
            d
            j
          
          -
          
            c
            j
          
        
        )
      
      ⁢
      
        v
        j
      
    
  
  =
  
    
      f
      -
      f
    
    =
    0.
  




Applying (2) (and noting from hypothesis that δS<1) we conclude that dj=cj for all jεT. Thus d=c as claimed. This concludes the proof of the applicants' theorem.

It is likely that one may push the condition (4) a little further. The key idea is as follows. Each vector wn in the iteration scheme used to prove Lemma 7 was designed to annihilate the influence of wn−1 on the exceptional set Tn−1. But annihilation is too strong of a goal. It would be just as suitable to design wn to moderate the influence of wn−1 enough so that the inner product with elements in Tn−1 is small rather than zero.
Approximate Orthogonality
Lemma 1 gives control of the size of the principal angle between subspaces of dimension S and S′ respectively. This is useful because it allows one to guarantee exact reconstruction from the knowledge of the δ numbers only.

    Proof of Lemma 1. The applicants first show that θS,S′≦δS+S′. By homogeneity it will suffice to show that






  
    
    
      
        〈
        
          
            
              ∑
              
                j
                ∈
                T
              
            
            ⁢
            
              
                c
                j
              
              ⁢
              
                v
                j
              
            
          
          ,
          
            
              ∑
              
                
                  j
                  ′
                
                ∈
                
                  T
                  ′
                
              
            
            ⁢
            
              
                c
                
                  j
                  ′
                
                ′
              
              ⁢
              
                v
                
                  j
                  ′
                
              
            
          
        
        〉
      
      H
    
    
  
  ≤
  
    δ
    
      S
      +
      
        S
        ′
      
    
  




whenever |T|≦S, |T′|≦S′, T, T′ are disjoint, and





  
    
      ∑
      
        j
        ∈
        T
      
    
    ⁢
    
      
        
        
          c
          j
        
        
      
      2
    
  
  =
  
    
      
        ∑
        
          
            j
            ′
          
          ∈
          
            T
            ′
          
        
      
      ⁢
      
        
          
          
            c
            
              j
              ′
            
            ′
          
          
        
        2
      
    
    =
    1.
  




Now (2) gives





  
    2
    ⁢
    
      (
      
        1
        -
        
          δ
          
            S
            +
            
              S
              ′
            
          
        
      
      )
    
  
  ≤
  
    
      
      
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
              c
              j
            
            ⁢
            
              v
              j
            
          
        
        +
        
          
            ∑
            
              
                j
                ′
              
              ∈
              
                T
                ′
              
            
          
          ⁢
          
            
              c
              
                j
                ′
              
              ′
            
            ⁢
            
              v
              
                j
                ′
              
            
          
        
      
      
    
    H
    2
  
  ≤
  
    2
    ⁢
    
      (
      
        1
        +
        
          δ
          
            S
            +
            
              S
              ′
            
          
        
      
      )
    
  




together with





  
    2
    ⁢
    
      (
      
        1
        -
        
          δ
          
            S
            +
            
              S
              ′
            
          
        
      
      )
    
  
  ≤
  
    
      
      
        
          
            ∑
            
              j
              ∈
              T
            
          
          ⁢
          
            
              c
              j
            
            ⁢
            
              v
              j
            
          
        
        -
        
          
            ∑
            
              
                j
                ′
              
              ∈
              
                T
                ′
              
            
          
          ⁢
          
            
              c
              
                j
                ′
              
              ′
            
            ⁢
            
              v
              
                j
                ′
              
            
          
        
      
      
    
    H
    2
  
  ≤
  
    2
    ⁢
    
      (
      
        1
        +
        
          δ
          
            S
            +
            
              S
              ′
            
          
        
      
      )
    
  




and the claim now follows from the parallelogram identity.

It remains to show that δS+S′≦θS,S′+max(δS, δS′). Again by homogeneity, it suffices to establish that




  
    
    
      
        
          〈
          
            
              
                ∑
                
                  j
                  ∈
                  
                    T
                    ~
                  
                
              
              ⁢
              
                
                  c
                  j
                
                ⁢
                
                  v
                  j
                
              
            
            ,
            
              
                ∑
                
                  
                    j
                    ′
                  
                  ∈
                  
                    T
                    ~
                  
                
              
              ⁢
              
                
                  c
                  
                    j
                    ′
                  
                
                ⁢
                
                  v
                  
                    j
                    ′
                  
                
              
            
          
          〉
        
        H
      
      -
      1
    
    
  
  ≤
  
    
      max
      ⁡
      
        (
        
          
            δ
            S
          
          ,
          
            δ
            
              S
              ′
            
          
        
        )
      
    
    +
    
      θ
      
        S
        ,
        
          S
          ′
        
      
    
  




whenever |{tilde over (T)}|≦S+S′ and





  
    
      ∑
      
        j
        ∈
        
          T
          ~
        
      
    
    ⁢
    
      
        
        
          c
          j
        
        
      
      2
    
  
  =
  1.




To prove this property, we partition {tilde over (T)} as {tilde over (T)}=T∪T′ where |T|≦S and |T′|≦S′ and write





  
    
      ∑
      
        j
        ∈
        T
      
    
    ⁢
    
      
        
        
          c
          j
        
        
      
      2
    
  
  =
  
    
      α
      ⁢
      
        
      
      ⁢
      and
      ⁢
      
        
      
      ⁢
      
        
          ∑
          
            j
            ∈
            
              T
              ′
            
          
        
        ⁢
        
          
            
            
              c
              j
            
            
          
          2
        
      
    
    =
    
      1
      -
      
        α
        .
      
    
  




(2) together with (3) give




  
    
      
    
    ⁢
    
      
        
          (
          
            1
            -
            
              δ
              S
            
          
          )
        
        ⁢
        α
      
      ≤
      
        
          〈
          
            
              
                ∑
                
                  j
                  ∈
                  T
                
              
              ⁢
              
                
                  c
                  j
                
                ⁢
                
                  v
                  j
                
              
            
            ,
            
              
                ∑
                
                  
                    j
                    ′
                  
                  ∈
                  T
                
              
              ⁢
              
                
                  c
                  
                    j
                    ′
                  
                
                ⁢
                
                  v
                  
                    j
                    ′
                  
                
              
            
          
          〉
        
        H
      
      ≤
      
        
          (
          
            1
            +
            
              δ
              S
            
          
          )
        
        ⁢
        
          α
          ⁢
          
            

          
          ⁢
          
            
          
          (
          
            1
            -
            
              δ
              
                S
                ′
              
            
          
          )
        
        ⁢
        
          (
          
            1
            -
            α
          
          )
        
      
      ≤
      
        
          〈
          
            
              
                ∑
                
                  j
                  ∈
                  
                    T
                    ′
                  
                
              
              ⁢
              
                
                  c
                  j
                
                ⁢
                
                  v
                  j
                
              
            
            ,
            
              
                ∑
                
                  
                    j
                    ′
                  
                  ∈
                  
                    T
                    ′
                  
                
              
              ⁢
              
                
                  c
                  
                    j
                    ′
                  
                
                ⁢
                
                  v
                  
                    j
                    ′
                  
                
              
            
          
          〉
        
        H
      
      ≤
      
        
          (
          
            1
            +
            
              δ
              
                S
                ′
              
            
          
          )
        
        ⁢
        
          (
          
            1
            -
            α
          
          )
        
      
    
  




  
    
      
    
    ⁢
    
      
        
        
          
            〈
            
              
                
                  ∑
                  
                    j
                    ∈
                    T
                  
                
                ⁢
                
                  
                    c
                    j
                  
                  ⁢
                  
                    v
                    j
                  
                
              
              ,
              
                
                  ∑
                  
                    
                      j
                      ′
                    
                    ∈
                    
                      T
                      ′
                    
                  
                
                ⁢
                
                  
                    c
                    
                      j
                      ′
                    
                  
                  ⁢
                  
                    v
                    
                      j
                      ′
                    
                  
                
              
            
            〉
          
          H
        
        
      
      ≤
      
        
          θ
          
            S
            ,
            
              S
              ′
            
          
        
        ⁢
        
          
            
              
                α
                
                  1
                  /
                  2
                
              
              ⁡
              
                (
                
                  1
                  -
                  α
                
                )
              
            
            
              1
              /
              2
            
          
          .
          
            

          
          ⁢
          
            
          
          ⁢
          Hence
        
      
    
  




  
    
      
      
        
          
            〈
            
              
                
                  ∑
                  
                    j
                    ∈
                    
                      T
                      ~
                    
                  
                
                ⁢
                
                  
                    c
                    j
                  
                  ⁢
                  
                    v
                    j
                  
                
              
              ,
              
                
                  ∑
                  
                    
                      j
                      ′
                    
                    ∈
                    
                      T
                      ~
                    
                  
                
                ⁢
                
                  
                    c
                    
                      j
                      ′
                    
                  
                  ⁢
                  
                    v
                    
                      j
                      ′
                    
                  
                
              
            
            〉
          
          H
        
        -
        1
      
      
    
    ≤
    
      
        
          δ
          S
        
        ⁢
        α
      
      +
      
        
          δ
          
            S
            ′
          
        
        ⁡
        
          (
          
            1
            -
            α
          
          )
        
      
      +
      
        2
        ⁢
        
          θ
          
            S
            ,
            
              S
              ′
            
          
        
        ⁢
        
          
            
              α
              
                1
                /
                2
              
            
            ⁡
            
              (
              
                1
                -
                α
              
              )
            
          
          
            1
            /
            2
          
        
      
    
    ≤
    
      
        max
        ⁡
        
          (
          
            
              δ
              S
            
            ,
            
              δ
              
                S
                ′
              
            
          
          )
        
      
      +
      
        θ
        
          S
          ,
          
            S
            ′
          
        
      
    
  



as claimed.

Gaussian Random Matrices

In this section, the applicants argue that with overwhelming probability, Gaussian random matrices have “good” isometry constants. Consider a p by m matrix F whose entries are i.i.d. Gaussian with mean zero and variance 1/p and let T be a subset of the columns. We wish to study the extremal eigenvalues of F*TFT. Following upon the work of Marchenko and Pastur [18], Geman [15] and Silverstein [22] (see also [1]) proved that

λmin(F*TFT)→(1−√{square root over (γ)})2a.s.

λmax(F*TFT)→(1+√{square root over (γ)})2a.s.

in the limit where p and |T|→∞ with |T|/p→γ≦1. In other words, this says that loosely speaking and in the limit of large p, the restricted isometry constant δ(FT) for a fixed T behaves like

1−δ(FT)≦λmin(F*TFT)≦λmax(F*TFT)≦1+δ(FT), δ(FT)≈2√{square root over (|T|/p)}+|T|/p. 

Restricted isometry constants must hold for all sets T of cardinality less or equal to S, and the applicants shall make use of concentration inequalities to develop such a uniform bound. Note that for T′⊂T, one obviously has

λmin(F*TFT)≦λmin(F*T′FT′)≦λmax(F*T′FT′)≦λmax(F*TFT)

and, therefore, attention may be restricted to matrices of size S. Now, there are large deviation results about the singular values of FT [23]. For example, letting σmax(FT):=λmax(F*TFT)1/2 be the largest singular value of FT, and defining σmin(FT) similarly, Ledoux [18] applies the concentration inequality for Gaussian measures, and for a fixed t>0, obtains the deviation bounds

P(σmax(FT)>1+√{square root over (|T|/p)}+o(1)+t)≦e−pt2/2   (10)

P(σmin(FT)<1−√{square root over (|T|/p)}−o(1)−t)≦e−pt2/2;   (11)

here, o(2) is a small term tending to zero as p→∞ and which can be calculated explicitly, see [14]. For example, this last reference shows that one can select o(2) in (10) as





  
    1
    
      2
      ⁢
      
        p
        
          1
          /
          3
        
      
    
  
  ·
  
    
      
        
          γ
          
            1
            /
            6
          
        
        ⁡
        
          (
          
            1
            +
            
              γ
            
          
          )
        
      
      
        2
        /
        3
      
    
    .
  




Lemma 8. Put r:=S/m and set

f(r):=√{square root over (m/p)}·(√{square root over (r)}+√{square root over (2H(r))}),

where H is the entropy function H(q):=−q log q−(1−q)log(1−q) defined for 0<q<1. For each ε>0 the restricted isometry constant δS of a p by m Gaussian matrix F obeys

P(1+δS>[1+(1+ε)f(r)]2)≦2·e−mH(r)·ε/2. 

    Proof. As discussed above, we may restrict the applicants' attention to sets |T|such that |T|=S. Denote by ηp the o(2)-term appearing in either (10) or (11). Put λmax:λmax(F*TFT) for short, and observe that






  
    P
    (
    
      
        
          sup
          
            
              T
              :
              
                
                T
                
              
            
            =
            S
          
        
        ⁢
        
          λ
          
            ma
            ⁢
            
              
            
            ⁢
            x
          
        
      
      >
      
        
          (
          
            1
            +
            
              
                S
                /
                p
              
            
            +
            
              η
              p
            
            +
            t
          
          )
        
        2
      
    
    )
  
  ≤
  
    
      
      
        {
        
          
            T
            ⁢
            
              :
            
            ⁢
            
              
              T
              
            
          
          =
          S
        
        }
      
      
    
    ⁢
    
      P
      ⁡
      
        (
        
          
            λ
            
              ma
              ⁢
              
                
              
              ⁢
              x
            
          
          >
          
            
              (
              
                1
                +
                
                  
                    S
                    /
                    p
                  
                
                +
                
                  η
                  p
                
                +
                t
              
              )
            
            2
          
        
        )
      
    
  
  ≤
  
    
      (
      
        
          
            m
          
        
        
          
            S
          
        
      
      )
    
    ⁢
    
      
        e
        
          
            -
            p
          
          ⁢
          
            
          
          ⁢
          
            
              t
              2
            
            /
            2
          
        
      
      .
    
  




From Stirling's approximation log m!=m log m−m+O(log m) we have the well-known formula





  
    log
    ⁡
    
      (
      
        
          
            m
          
        
        
          
            S
          
        
      
      )
    
  
  =
  
    
      m
      ⁢
      
        
      
      ⁢
      
        H
        ⁡
        
          (
          r
          )
        
      
    
    +
    
      O
      ⁡
      
        (
        
          log
          ⁢
          
            
          
          ⁢
          m
        
        )
      
    
  




which gives





  
    P
    (
    
      
        
          sup
          
            
              T
              :
              
                
                T
                
              
            
            =
            S
          
        
        ⁢
        
          λ
          
            ma
            ⁢
            
              
            
            ⁢
            x
          
        
      
      >
      
        
          (
          
            1
            +
            
              
                S
                /
                p
              
            
            +
            
              η
              p
            
            +
            t
          
          )
        
        2
      
    
    )
  
  ≤
  
    
      e
      
        m
        ⁢
        
          
        
        ⁢
        
          H
          ⁡
          
            (
            r
            )
          
        
      
    
    ·
    
      e
      
        O
        ⁡
        
          (
          
            lo
            ⁢
            
              
            
            ⁢
            g
            ⁢
            
              
            
            ⁢
            m
          
          )
        
      
    
    ·
    
      
        e
        
          
            -
            p
          
          ⁢
          
            
          
          ⁢
          
            
              t
              2
            
            /
            2
          
        
      
      .
    
  




The exact same argument applied to the smallest eigenvalues yields





  
    P
    (
    
      
        
          inf
          
            
              T
              :
              
                
                T
                
              
            
            =
            S
          
        
        ⁢
        
          λ
          
            m
            ⁢
            
              
            
            ⁢
            i
            ⁢
            
              
            
            ⁢
            n
          
        
      
      <
      
        
          (
          
            1
            -
            
              
                S
                /
                p
              
            
            -
            
              η
              p
            
            -
            t
          
          )
        
        2
      
    
    )
  
  ≤
  
    
      e
      
        m
        ⁢
        
          
        
        ⁢
        
          H
          ⁡
          
            (
            r
            )
          
        
      
    
    ·
    
      e
      
        O
        ⁡
        
          (
          
            lo
            ⁢
            
              
            
            ⁢
            gm
          
          )
        
      
    
    ·
    
      
        e
        
          
            -
            p
          
          ⁢
          
            
          
          ⁢
          
            
              t
              2
            
            /
            2
          
        
      
      .
    
  




Fix ηp+t=(1+ε)·√{square root over (m/p)}·√{square root over (2H(r))}. Assume now that m and p are large enough so that





  
    η
    p
  
  ≤
  
    
      ɛ
      2
    
    ·
    
      
        m
        /
        p
      
    
    ·
    
      
        
          2
          ⁢
          
            H
            ⁡
            
              (
              r
              )
            
          
        
      
      .
    
  




Then





  
    P
    (
    
      
        
          sup
          
            T
            :
            
              
                
                T
                
              
              +
              S
            
          
        
        ⁢
        
          λ
          
            m
            ⁢
            
              
            
            ⁢
            a
            ⁢
            
              
            
            ⁢
            x
          
        
      
      >
      
        
          (
          
            1
            +
            
              
                S
                /
                p
              
            
            +
            
              ɛ
              ·
              
                
                  m
                  /
                  p
                
              
              ·
              
                
                  2
                  ⁢
                  
                    H
                    ⁡
                    
                      (
                      r
                      )
                    
                  
                
              
            
          
          )
        
        2
      
    
    )
  
  ≤
  
    e
    
      
        -
        m
      
      ⁢
      
        
      
      ⁢
      
        
          H
          ⁡
          
            (
            r
            )
          
        
        ·
        
          ɛ
          /
          2
        
      
    
  




where we used the fact that the term O(log m) is less than mεH(r)/2 for sufficiently large m. The same bound holds for the minimum eigenvalues and the claim follows.

Ignoring the ε's, Lemma 8 states that with overwhelming probability

δS<−1+[1+f(r)]2.

A similar conclusion holds for δ2S and δ3S and, therefore, we established that





  
    
      
        
          
            
              δ
              S
            
            +
            
              δ
              
                2
                ⁢
                S
              
            
            +
            
              δ
              
                3
                ⁢
                S
              
            
          
          <
          
            
              ρ
              
                p
                /
                m
              
            
            ⁡
            
              (
              r
              )
            
          
        
        ,
        
          
            
              ρ
              
                p
                /
                m
              
            
            ⁡
            
              (
              r
              )
            
          
          =
          
            
              
                ∑
                
                  j
                  =
                  1
                
                3
              
              ⁢
              
                -
                1
              
            
            +
            
              
                
                  [
                  
                    1
                    +
                    
                      f
                      ⁡
                      
                        (
                        
                          j
                          ⁢
                          
                            
                          
                          ⁢
                          r
                        
                        )
                      
                    
                  
                  ]
                
                2
              
              .
            
          
        
      
    
    
      
        (
        12
        )
      
    
  




with very high probability. In conclusion, Lemma 1 shows that the hypothesis of the applicants' main theorem holds provided that the ratio r=S/m be so small that ρp/m(r)<1$. In other words, in the limit of large samples, S/m may be taken as any value obeying ρp/m(S/m)<1 which we used to give numerical values in Theorem 5. FIG. 1 graphs the function ρp/m(r) for several values of the ratio p/m.

Numerical Experiments

This section investigates the practical ability of l1 to decode an object fεRn from corrupted encoded data y=Af+e, yεRm (or equivalently, to recover the sparse vector of errors eεRm from the underdetermined system of equations Fe=zεRm−n). The goal here is to evaluate empirically the location of the breakpoint as to get an accurate sense of the performance one might expect in practice. In order to do this, the applicants performed a series of experiments designed as follows:

    
    
        1. select n (the size of the input signal) and m so that with the same notations as before, A is an n by m matrix; sample A with independent Gaussian entries;
        2. select S as a percentage of m;
        3. select a support set T of size |T|=S uniformly at random, and sample a vector e on T with independent and identically distributed Gaussian entries. (Just as in [5], the results presented here do not seem to depend on the actual distribution used to sample the errors.);
        4. perform the noisy encoding {tilde over (y)}=Ax+e of the data x (the choice of x does not matter as is clear from the discussion and here, x is also selected at random), apply the decoding procedure (P1′) and obtain x*;
        5. compare x to x*;
        6. repeat 100 times for each S and A;
        7. repeat for various sizes of n and m.
    
    


The results are presented in FIG. 2 and FIG. 3. FIG. 2 examines the situation in which the length of the code is twice that of the input vector m=2n, for m=512 and m=1024. The applicants' experiments show that one recovers the input vector all the time as long as the fraction of the corrupted entries is below 17%. This holds for m=512 (FIG. 2(a)) and m=1024 (FIG. 2(b)). In FIG. 3, the applicants investigate how these results change as the length of the codewords increases compared to the length of the input, and examine the situation in which m=4n, with m=512. The applicants' experiments show that one decodes the input vector all the time as long as the fraction of the corrupted entries is below 34%.
Optimal Signal Recovery
The applicants' recent work [4] developed a set of ideas showing that it is surprisingly possible to reconstruct interesting classes of signals accurately from highly incomplete measurements. The results in the present application are inspired and improve upon this earlier work and the applicants now elaborate on this connection. Suppose one wishes to reconstruct an object αεRm from the K linear measurements

yk=α, φk k=1, . . . K, i.e.y=Fα,   (13)

with φk equal to the kth row of the matrix F. Of special interest is the vastly underdetermined case, K<<N, where there are many more unknowns than observations. The applicants choose to formulate the problem abstractly but for concreteness, one might think of α as the coefficients α:Ψ*f of a digital signal or image f in some nice orthobasis, e.g. a wavelet basis so that the information about the signal is of the form y=Fα=FΨ*f.

Suppose now that the object of interest is compressible in the sense that the reordered entries of α decay like a power-law; concretely, suppose that the entries of α, rearranged in decreasing order of magnitude, |α|(1)≧|α|(2)≧ . . . ≧|α|(m), obey

|α|(k)≦B·k−s   (14)

for some s≧1. The applicants will denote by Fs(B) the class of all signals αεRm obeying (14). The claim is that it is possible to reconstruct compressible signals from only a small number of random measurements.

    Theorem 9. Let F be the measurement matrix as in (13) and consider the solution α# to






  
    
      min
      
        
          α
          ~
        
        ∈
        
          R
          m
        
      
    
    ⁢
    
      
        
          
          
            α
            ~
          
          
        
        
          l
          1
        
      
      ⁢
      
        
      
      ⁢
      subject
      ⁢
      
        
      
      ⁢
      to
      ⁢
      
        
      
      ⁢
      F
      ⁢
      
        
      
      ⁢
      
        α
        ~
      
    
  
  =
  
    y
    .
  




Let S≦K such that δS+2θS+θS,2S<1 and set λ:K/S. Then α# obeys





  
    
      
        
          
            sup
            
              α
              ⁢
              
                
              
              ∈
              
                
                  F
                  s
                
                ⁢
                
                  
                
                ⁢
                
                  (
                  B
                  )
                
              
            
          
          ⁢
          
            
            
              α
              -
              
                α
                #
              
            
            
          
        
        ≤
        
          C
          ·
          
            
              
                (
                
                  K
                  /
                  λ
                
                )
              
              
                -
                
                  (
                  
                    s
                    -
                    
                      1
                      /
                      2
                    
                  
                  )
                
              
            
            .
          
        
      
    
    
      
        (
        15
        )
      
    
  




To appreciate the content of the theorem, suppose one would have available an oracle letting us know which coefficients αk,1≦k≦m, are large (e.g. in the scenario we considered earlier, the oracle would tell us which wavelet coefficients of f are large). Then we would acquire information about the K largest coefficients and obtain a truncated version αK of α obeying

∥α−αK∥≧c·K−(s−1/2),

for generic elements taken from Fs(B). Now (15) says that not knowing anything about the location of the largest coefficients, one can essentially obtain the same approximation error by non-adaptive sampling, provided the number of measurements be increased by a factor λ. The larger S, the smaller the oversampling factor, and hence the connection with the decoding problem. Such considerations make clear that Theorem 9 supplies a very concrete methodology for recovering a compressible object from limited measurements and as such, it may have a significant bearing on many fields of science and technology. The applicants refer the reader to [4] and [10] for a discussion of its implications.

Suppose for example that F is a Gaussian random matrix as discussed in an earlier section. We will assume the same special normalization so that the variance of each individual entry is equal to 1/K. Calculations identical to those from earlier sections give that with overwhelming probability, F obeys the hypothesis of the theorem provided that

S≦K/λ, λ=ρ·log(m/K),

for some positive constant ρ>0. Now consider the statement of the theorem; there is a way to invoke linear programming and obtain a reconstruction based upon O(K log m/K) measurements only, which is at least as good as that one would achieve by knowing all the information about f and selecting the K largest coefficients. In fact, this is an optimal statement as (15) correctly identifies the minimum number of measurements needed to obtain a given precision. In short, it is impossible to obtain a precision of about K−(s−1/2) using only O(K log m/K) measurements, see [4, 10].

Theorem 9 is stronger than the applicants' former result, namely, Theorem 1.4 in [4]. To see why this is true, recall the former claim: [4] introduced two conditions, the uniform uncertainty principle (UUP) and the exact reconstruction principle (ERP). In a nutshell, a random matrix F obeys the UUP with oversampling factor λ if F obeys

δS≦1/2, S=ρ·K/λ,   (16)

with probability at least 1−O(Nγ/ρ) for some fixed positive constant γ>0. Second, a measurement matrix F obeys the ERP with oversampling factor λ if for each fixed subset T of size |T|≦S (16) and each “sign” vector c defined on T, there exists with the same overwhelmingly large probability a vector wεH with the following properties:

1. w, vj=cj, for all jεT.
2. and w, vj<1⁄2 for all j not in T.
Note that these are the conditions 1, 2 studied in previous sections, except for the 1⁄2 factor on the complement of T. Fix αεFs(B). In [4] it was argued that if a random matrix obeyed the UUP and the ERP both with oversampling factor λ, then

∥α−α#∥≦C·(K/λ)−s−1/2),

with inequality holding with the same probability as before. Against this background, several comments are now in order:

    
    
        First, the new statement is more general as it applies to all matrices, not just random matrices.
        Second, whereas the applicants' previous statement argued that for each αεRm, one would be able—with high probability—to reconstruct a accurately, it did not say anything about the worst case error for a fixed measurement matrix F. This is an instance where the order of the quantifiers plays a role. Do we need different F's for different objects? Theorem 9 answers this question unambiguously; the same F will provide an optimal reconstruction for all the objects in the class.
        Third, Theorem 9 says that the ERP condition is redundant, and hence the hypothesis may be easier to check in practice. In addition, eliminating the ERP isolates the real reason for success as it ties everything down to the UUP. In short, the ability to recover an object from limited measurements depends on how close F is to an orthonormal system, but only when restricting attention to sparse linear combinations of columns.
    
    


The applicants will not prove this theorem as this is a minor modification of that of Theorem 1.4 in the aforementioned reference. The key point is to observe that if F obeys the hypothesis of the applicants' theorem, then by definition F obeys the UUP with probability one, but F also obeys the ERP, again with probability one, as this is the content of Lemma 7. Hence both the UUP and ERP hold and therefore, the conclusion of Theorem 9 follows. (The fact that the ERP actually holds for all sign vectors of size less than S is the reason why (15) holds uniformly over all elements taken from Fs(B), see [4].)
Discussion
Improvements
More elaborate arguments can yield versions of Theorem 5 with tighter bounds. Immediately following the proof of Lemma 7, we already remarked that one might slightly improve the condition (4) at the expense of considerable complications. The applicants will now discuss improvements in this direction using tools from Random Matrix Theory.
The applicants' main hypothesis in (4) reads δS+θS+θS,2S<1, but in order to reduce the problem to the study of those δ numbers (and use known results), the applicants' analysis actually relied upon the more stringent condition δS+δ2S+δ3S<1 instead, since

δS+θS+θS,2S≦δS+δ2S+δ3S.

This introduces a gap. Consider a fixed set T of size |T|=S. Using the notation of previous sections, the applicants argued that

δ(FT)≈2√{square root over (S/p)}+S/p,

and developed a large deviation bound to quantify the departure from the right hand-side. Now let T and T′ be two disjoint sets of respective sizes S and S′ and consider θ(FT, FT′), which is the cosine of the principal angle between the two random subspaces spanned by the columns of FT and FT′ respectively; formally

θ(FT, FT′):=supu, u′ uεspan(FT), u′εspan(FT′), ∥u∥=∥u′∥=1.

The applicants remark that this quantity plays an important analysis in statistical analysis because of its use to test the significance of correlations between two sets of measurements, compare the literature on Canonical Correlation Analysis [20]. Among other things, it is known [25] that

θ(FT, FT′)→√{square root over (γ(1−γ′))}+√{square root over (γ′(1−γ))} a.s.

as p→∞ with S/p→γ and S′/p→γ′. In other words, whereas we used the limiting behaviors

δ(F2T)→2√{square root over (2γ)}+2γ, δ(F3T)→2√{square root over (3γ)}+3γ,

there is a chance one might employ instead

θ(FT, FT′)→2√{square root over (γ(1−γ))}, θ(FT, FT′)→√{square root over (γ(1−2γ))}+√{square root over (2γ(1−γ))}

for |T|=|T′|=S and |T′|=2|T|=2S respectively, which is better. Just as in previous sections, one might then look for concentration inequalities transforming this limiting behavior into corresponding large deviation inequalities. Recent work of Johnstone and his colleagues [17] might be of help here.

Finally, tighter large deviation bounds might exist together with more clever strategies to derive uniform bounds (valid for all T of size less than S) from individual bounds (valid for a single T). With this in mind, it is interesting to note that the applicants' approach hits a limit as




  
    
      
        
          
            
              
                lim
                
                  
                    S
                    ->
                    ∞
                  
                  
                    
                      S
                      /
                      m
                    
                    ->
                    r
                  
                
              
              ⁢
              
                inf
                ⁢
                
                  
                
                ⁢
                
                  δ
                  S
                
              
            
            +
            
              θ
              S
            
            +
            
              θ
              
                S
                ,
                
                  2
                  ⁢
                  S
                
              
            
          
          ≥
          
            J
            ⁡
            
              (
              
                
                  m
                  p
                
                ·
                r
              
              )
            
          
        
        ,
      
    
    
      
        (
        17
        )
      
    
  




where J(r):=2√{square root over (r)}+r+(2+√{square root over (2)})√{square root over (r(1−r))}+√{square root over (r(1−2r))}. Since J(r) is greater than 1 if and only if r>2.36, one would certainly need new ideas to improve Theorem 5 beyond cut-off point in the range of about 2%. The lower limit (17) is probably not sharp since it does not explicitly take into account the ratio between m and p; at best, it might serve as an indication of the limiting behavior when the ratio p/m is not too small.

Other Coding Matrices

The present application introduced general results stating that it is possible to correct for errors by l1-minimization. We then explained how the results specialize in the case where the coding matrix A is sampled from the Gaussian ensemble. It is clear, however, that an expert in these methods could use other matrices and still obtain similar results; namely, that (P1′) recovers f exactly provided that the number of corrupted entries does not exceed ρ·m. In fact, the applicants' previous work suggests that partial Fourier matrices would enjoy similar properties [5, 4]. Other candidates might be the so-called noiselets of Coifman, Geshwind and Meyer [7]. These alternative might be of great practical interest because they would come with fast algorithms for applying A or A* to an arbitrary vector g and, hence, speed up the computations to find the l1-minimizer.
Conclusions
The present disclosure is about the classical error correcting problem which is frequently discussed in communication and coding theory, The problem is as follows: a vector fεRn (i.e. a sequence f=(f1, f2, . . . , fn) of n real numbers) is encoded and then transmitted using a linear code, i.e. a collection of codewords which are vectors a1, a2, . . . , anεRm—the columns of a matrix A. Given a vector fεRn (the “plaintext”) we can then generate a vector Af in Rm (the “ciphertext”). A recurrent problem with real communication or storage devices is that some of the entries of the ciphertext Af may become corrupted; when this is the case, the corrupted entries are unreliable and may have nothing to do with the original values. In general, it is not known which entries are affected, nor is it known how they are affected. The applicants' disclosure shows that no matter what the corruption is like, it is provably possible to decode the plaintext from the corrupted ciphertext by solving a convenient linear program, with success guaranteed as long as that the fraction of corrupted entries in not excessively large. The algorithm proceeds by solving the convex optimization program




  
    min
    
      x
      ∈
      
        R
        n
      
    
  
  ⁢
  
    
      
      
        y
        -
        Ax
      
      
    
    
      l
      1
    
  




where y is the vector of corrupted encoded data and





  
    
      
      x
      
    
    
      l
      1
    
  
  :=
  
    
      ∑
      i
    
    ⁢
    
      
        
        
          x
          i
        
        
      
      .
    
  




The following are two key features of the decoding algorithm presented in this disclosure:

    
    
        1. Tractability. The optimization program is equivalent a linear program (1). This observation is of practical significance since it enables large scale decoding problems to be solved rapidly and efficiently using standard off-the-shelf optimization software.
        2. Performance. The applicants performed numerical experiments which show that in practice, the decoding algorithm of the present disclosure works extremely well when the coding matrix is generated at random. Suppose for example, that the number of codewords m is 4 times the length of the plaintext (m=4n). The applicants' numerical experiments have shown that the plaintext is recovered exactly all the time as long as the fraction of the corrupted entries is below about 34% of the size of the ciphertext. (In contrast, it is known that accurate decoding is impossible if the corruption rate is 50% or higher).
    
    


This invention has many potential applications as efficient error-correction codes are in widespread use in many fields of technology. Examples include the transmission of Internet media, Internet telephony, and speech transmission. In addition, error correction is embedded as a key building block in numerous algorithms, and data-structures, where corruption is possible; corruption of digital data stored on a hard-drive, CD, DVD or similar media is a good example. In short, progress in error correction has potential to impact several storage and communication systems.
Where several illustrative embodiments of the invention have been shown and described above, numerous variations and alternative embodiments will occur to experts. Such variations and alternative embodiments are contemplated and can be made without departing from the scope of the invention as defined in the appended claims.
References


    [1] Z. D. Bai and Y. Q. Yin. Limit of the smallest eigenvalue of a large-dimensional sample covariance matrix. Ann. Probab. 21 (1993), 1275-1294.
    [2] S. Boyd, and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
    [3] E. J. Candès, J. Romberg, and T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. To appear, IEEE Transactions on Information Theory. Available at http://arxiv.org/abs/math.NA/0409186
    [4] E. J. Candès, and T. Tao, Near optimal signal recovery from random projections: universal encoding strategies? Submitted to IEEE Transactions on Information Theory, October 2004. Available at http://arxiv.org/abs/math.CA/0410542
    [5] E. J. Candès, and J. Romberg, Quantitative Robust Uncertainty Principles and Optimally Sparse Decompositions. To appear, Foundations of Computational Mathematics, November 2004. Available at http://arxiv.org/abs/math.CA/0411273
    [7] S. S. Chen. Basis Pursuit. Stanford Ph. ̃D. ̃Thesis, 1995.
    [6] S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM J. Scientific Computing 20 (1999), 33-61.
    [7] R. Coifman, F. Geshwind, and Y. Meyer. Noiselets. Appl. Comput. Harmon. Anal. 10 (2001), 27-44.
    [8] K. R. Davidson and S. J. Szarek. Local operator theory, random matrices and Banach spaces. In: Handbook in Banach Spaces, Vol I, ed. W. B. Johnson, J. Lindenstrauss, Elsevier (2001), 317-366.
    [9] D. L. Donoho. For Most Large Underdetermined Systems of Linear Equations the Minimal l1-norm Solution is also the Sparsest Solution. Manuscript, September 2004. Available at http://www-stat.stanford.edu/ ̃donoho/reports.html
    [10] D. L. Donoho, Compressed Sensing, Manuscript, September 2004. Available at http://www-stat.stanford.edu/ ̃donoho/reports.html
    [11] D. L. Donoho and X. Huo. Uncertainty principles and ideal atomic decomposition. IEEE Transactions on Information Theory, 47 (2001), 2845-2862.
    [12] D. L. Donoho and M. Elad. Optimally sparse representation in general (nonorthogonal) dictionaries via l1 minimization, Proc. Natl. Acad. Sci. USA, 100 (2003), 2197-2202.
    [13] M. Elad and A. M. Bruckstein. A generalized uncertainty principle and sparse representation in pairs of RN bases. IEEE Transactions on Information Theory, 48 (2002), 2558-2567.
    [14] N. El Karoui. New Results about Random Covariance Matrices and Statistical Applications. Stanford Ph.D. Thesis, August 2004.
    [15] S. Geman. A limit theorem for the norm of random matrices. Ann. Probab. 8 (1980), 252-261.
    [16] R. Gribonval and M. Nielsen. Sparse representations in unions of bases. IEEE Trans. Inform. Theory 49 (2003), 3320-3325.
    [17] I. M. Johnstone. Large covariance matrices. Third 2004 Wald Lecture, 6th World Congress of the Bernoulli Society and 67th Annual Meeting of the IMS, Barcelona, July 2004.
    [18] M. Ledoux. The concentration of measure phenomenon. Mathematical Surveys and Monographs 89, American Mathematical Society, Providence, R.I., 2001.
    [19] V. A. Marchenko and L. A. Pastur. Distribution of eigenvalues in certain sets of random matrices. Mat. Sb. (N.S.)} 72 (1967), 407-535 (in Russian).
    [20] R. J. Muirhead. Aspects of multivariate statistical theory. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, Inc., New York, 1982.
    [21] B. K. Natarajan. Sparse approximate solutions to linear systems. SIAM J. Comput. 24 (1995), 227-234.
    [22] J. W. Silverstein. The smallest eigenvalue of a large dimensional Wishart matrix. Ann. Probab. 13 (1985), 1364-1368.
    [23] S. J. Szarek. Condition numbers of random matrices. J. Complexity 7 (1991), 131-149.
    [24] J. A. Tropp. Greed is good: Algorithmic results for sparse approximation. Technical Report, the University of Texas at Austin, 2003.
    [25] K. W. Wachter. The limiting empirical measure of multiple discriminant ratios. Ann. Statist. 8 (1980), 937-957.

## Claims

1. A system comprising:
a first computer configured as an encoding device for transmitting data via a transmission medium, the transmitting comprising:
encoding the data using a linear code, the encoded data represented by an original vector y; and
providing the original vector y to the transmission medium; and

a second computer configured as a decoding device for receiving and recovering the original vector y from data received via the transmission medium, the received data comprising a corrupted vector y′ corrupted by one or more errors introduced in the transmission medium, the recovering comprising:
correcting the corrupted vector y′ by solving the convex optimization problem






  
    min
    
      x
      ∈
      
        R
        n
      
    
  
  ⁢
  
    
      
      
        y
        -
        Ax
      
      
    
    
      l
      1
    
  




 using a linear program wherein A is a coding matrix expressing the linear code, and l1 represents a norm such that






  
    
      
      x
      
    
    
      l
      1
    
  
  :=
  
    
      ∑
      i
    
    ⁢
    
      
        
        
          x
          i
        
        
      
      .

2. The system of claim 1, wherein the coding matrix A is a random matrix.

3. The system of claim 1, wherein the coding matrix A is sampled from a Gaussian ensemble.

4. The system of claim 1, wherein the original vector comprises data relating to a field selected from Internet media, Internet telephony and speech transmission.

5. The system of claim 1, wherein the coding matrix A used to encode said original vector is known by both the first and the second computers.

6. The system of claim 1, wherein the coding matrix A is generated through a standard random number generator with an agreed upon seed.

7. The system of claim 1, wherein the transmission medium is a communication network, and wherein providing the original vector y to the transmission medium comprises transmitting the original vector y into the communication network.

8. The system of claim 7, wherein the communication network is the Internet.

9. The system of claim 8, wherein the one or more errors is introduced by noise in the communication network.

10. The system of claim 1, wherein the transmission medium comprises a storage device, and wherein providing the original vector y to the transmission medium comprises providing the original vector y in the storage device.

11. The system of claim 10, wherein the storage device is one of a hard disk, a CD, or a DVD.

12. The system of claim 11, wherein the one or more errors is introduced by at least one of: a) damage to the CD, or b) damage to the DVD.

13. The system of claim 1, wherein the first computer is the same as the second computer, and the communication medium comprises a storage device.

14. The system of claim 13, wherein the storage device is one of a hard disk, a CD, or a DVD.