Short and Sparse Deconvolution --- A Geometric Approach

1 INTRODUCTION . Many signals arising in science and engineering can be modeled as superpositions of basic , recurring motifs , which encode critical information about a physical process of interest . Signals of this type can be modeled as the convolution of a zero-padded short kernel a0 P Rp0 ( the motif ) with a longer sparse signal x0 P Rm ( m `` p0 ) which encodes the locations of the motifs in the sample1 : y “ ιa0 f x0 . ( 1 ) We term this a short-and-sparse ( SaS ) model . Since often only y is observed , short-and-sparse deconvolution ( SaSD ) is the problem of recovering both a0 and x0 from y. Variants of SaSD arise in areas such as microscopy ( Cheung et al. , 2018 ) , astronomy ( Briers et al. , 2013 ) , and neuroscience ( Song et al. , 2018 ) . SaSD is a challenging inverse problem in both theory and practice . Natural formulations are nonconvex , and very little algorithmic theory was available . Moreover , practical instances are often ill-conditioned , due to the spectral decay of the kernel a0 ( Cheung et al. , 2018 ) . This paper is motivated by recent theoretical advances in nonconvex optimization and , in particular , on the geometry of SaSD . Zhang et al . ( 2017 ) and Kuo et al . ( 2019 ) study particular optimization ˚YL and QQ contributed equally to this work . The full version of this work can be found at https : //arxiv.org/abs/1908.10959 . 1For simplicity , ( 1 ) uses cyclic convolution ; algorithms are results also apply to linear convolution with minor modifications . Here ι denotes the zero padding operator . formulations for SaSD and show that the landscape is largely driven by the problem symmetries of SaSD . They derive provable methods for idealized problem instances , which exactly recover pa0 , x0q up to trivial ambiguities . While inspiring , these methods are not practical and perform poorly on real problem instances . Where the emphasis of Zhang et al . ( 2017 ) and Kuo et al . ( 2019 ) is on theoretical guarantees , here we focus on practical computation . We show how to combine ideas from this theory with heuristics that better address the properties of practical deconvolution problems , to build a novel method that performs well on data arising in a range of application areas . A critical issue in moving from theory to practice is the poor conditioning of naturally-occurring deconvolution problems : we show how to address this with a combination of ideas from sparse optimization , such as momentum , continuation , and reweighting . The end result is a general purpose method , which we demonstrate on data from neural spike sorting , calcium imaging and fluorescence microscopy . Notation . The zero-padding operator is denoted by ι : Rp Ñ Rm . Projection of a vector v P Rp onto the sphere is denoted by PSp´1pvq . “ v { } v } 2 , and Pzpvq . “ v ´ xv , zy z denotes projection onto the tangent space of z P Sp´1 . The Riemannian gradient of a function f : Sp´1 Ñ R at point z on the sphere is given by grad fpzq . “ Pzp∇fpzqq . Reproducible research . The code for implementations of our algorithms can be found online : https : //github.com/qingqu06/sparse_deconvolution . For more details of our work on SaSD , we refer interested readers to our project website https : //deconvlab.github.io/ . 2 SYMMETRY AND GEOMETRY IN SASD . In this section , we begin by describing two intrinsic properties for SaSD . Later , we show how these play an important role in the geometry of optimization and the design of efficient methods . An important observation of the SaSD problem is that it admits multiple equivalent solutions . This is purely due to the cyclic convolution between a0 and x0 , which exhibits the trivial ambiguity2 y “ ιa0 f x0 “ pαs ` rιa0sq f ` 1 αs´ ` rx0s ˘ , for any nonzero scalar α and cyclic shift s ` r¨s . These scale and shift symmetries create several acceptable candidates for a0 and x0 , and in the absence of further information we only expect to recover a0 and x0 up to symmetry . Furthermore , they largely drive the behavior of certain nonconvex optimization problems formulated for SaSD . Since the success of SaSD requires distinguishing between overlapping copies of a0 , its difficulty also depends highly on the “ similarity ” of the a0 to its shifts . Here we capture this notion using the shift-coherence of a0 , µpa0q . “ max ` “ 0 |xιa0 , s ` rιa0sy| P r0 , 1s . ( 2 ) Intuitively , the shifts of a0 become closer together as µpa0q increases ( Figure 10 ) , making objective landscapes for optimization less favorable for recovering any specific shift of a0 . 2.1 LANDSCAPE GEOMETRY UNDER SHIFT-INCOHERENCE . A natural approach to solving SaSD is to formulate it as a suitable optimization problem . In this paper we will focus on the Bilinear Lasso ( BL ) problem , which minimizes the squared error between the observation y and its reconstruction af x , plus a ` 1-norm sparsity penalty on x , min aPSp´1 , xPRm ” ΨBLpa , xq . “ 12 } y ´ ιaf x } 2 2 ` λ } x } 1 ı . ( 3 ) Later in this section , we will see that the kernel length p should be set slightly larger than p0 . The Bilinear Lasso is a nonconvex optimization problem , as the shift symmetries of SaSD create discrete local minimizers in the objective landscape . The regularization created by problem symmetries 2We therefore assume w.l.o.g . that } a0 } 2 “ 1 in this paper . in nonconvex inverse problems are a fairly general phenomenon ( Sun et al. , 2015 ) and , as Kuo et al . ( 2019 ) shows , its influence in SaSD extends beyond the neighborhoods of these local minimizers . Kuo et al . analyzed an Approximate Bilinear Lasso ( ABL ) objective3 ΨABL , which satisfies ΨABLpa , xq » ΨBLpa , xq , when µpaq » 0 . This non-practical objective serves as a valid simplification of the Bilinear Lasso for analysis when the true kernel is itself incoherent , i.e . µpa0q » 0 ( Figures 1d and 1e ) . Under its marginalization4 ϕABLpaq . “ minxPRm ΨABLpa , xq , ( 4 ) certain crucial properties regarding its curvature can be characterized for generic choices of x . The reason we choose to partial minimize x instead of a is because ( i ) the problem ( 4 ) is convex w.r.t . x , and ( ii ) the dimension of the subspace of a is significantly smaller than that of x ( i.e. , p ! m ) , which is the place that the measure concentrates . Curvature in the span of a few shifts . Suppose we set p ą p0 , which ensures that we can find an a » α1s ` 1ra0s ` α2s ` 2ra0s P Sp´1 that lies near the span of two shifts of a0 . If α1 » ˘1 ( or α2 » 0 ) then , under suitable conditions on a0 and x0 , Kuo et al . ( 2019 ) asserts that a lies in a strongly convex region of ϕABL , containing a single minimizer near s ` 1ra0s ( Figure 1a ) ; the converse is also true . A saddle-point exists nearby when α1 » α2 is balanced , characterized by large negative curvature along the two shifts and positive curvature in orthogonal directions ( Figure 1b ) . Interpolating between these two cases , large negative gradients point towards individual shifts . The behavior of ϕABL between two shifts of a0 — strong convexity near single shifts , and saddlepoints near balanced points — extends to regions of the sphere spanned by several shifts ( Figure 1c ) ; we elaborate on this further in Appendix A.1 . This regional landscape guarantees that a0 can be efficiently recovered up to a signed shift using methods for first and second-order descent , as soon as a can be brought sufficiently close to the span of a few shifts . Optimization over the sphere . For both the Bilinear Lasso and ABL , a unit-norm constraint on a is enforced to break the scaling symmetry between a0 and x0 . Choosing the ` 2-norm , however , has surprisingly strong implications for optimization . The ABL objective , for example , is piecewise concave whenever a is sufficiently far away from any shift of a0 , but the sphere induces positive curvature near individual shifts to create strong convexity . These two properties combine to ensure recoverability of a0 . In contrast , enforcing ` 1-norm constraints often leads to spurious minimizers for deconvolution problems ( Levin et al. , 2011 ; Benichoux et al. , 2013 ; Zhang et al. , 2017 ) . Initializing near a few shifts . The landscape of ϕABL makes single shifts of a0 easy to locate if a is initialized near a span of a few shifts . Fortunately , this is a relatively simple matter in SaSD , as y is 3As the intention here is apply some key intuition from the ABL objective towards the Bilinear Lasso itself , we intentionally omit the concrete form of ΨABLpaq . Readers may refer to Appendix A for more details . 4Minimizing ϕABL , this is equivalent to minimizing ΨABL as x can be recovered via convex optimization . itself a sparse superposition of shifts . Therefore , one initialization strategy is to randomly choose a length-p0 window ryi . “ ryi yi ` 1 . . . yi ` p0´1s T from the observation and set ap0q . “ PSp´1 ` r 0p0´1 ; ryi ; 0p0´1 s ˘ . ( 5 ) This brings ap0q suitably close to the sum of a few shifts of a0 ( Appendix A.2 ) ; any truncation effects are absorbed by padding the ends of ryi , which also sets the length for a to be p “ 3p0 ´ 2 . Implications for practical computation . The ( regionally ) benign optimization landscape of ϕABL guarantees that efficient recovery is possible for SaSD when a0 is incoherent . Applications of sparse deconvolution , however , are often motivated by sharpening or resolution tasks ( Huang et al. , 2009 ; Candès & Fernandez-Granda , 2014 ; Campisi & Egiazarian , 2016 ) where the motif a0 is smooth and coherent ( i.e . µpa0q is large ) . The ABL objective is a poor approximation of the Bilinear Lasso in such cases and fails to yield practical algorithms , so we should optimize the Bilinear Lasso directly . From Figures 1d and 1e , we can see that low-dimensional subspheres spanned by shifts of a0 are empirically similar when a0 is incoherent . Although this breaks down in the coherent case , as we illustrate in Appendix A.3 , the symmetry breaking properties of ϕBL remain present . This allows us to apply the geometric intuition discussed here to create an optimization method that , with the help of a number of computational heuristics , performs well in for SaSD even in general problem instances . Algorithm 1 Inertial Alternating Descent Method ( iADM ) Input : Initializations ap0q P Sp´1 , x P Rm ; observation y P Rm ; penalty λ ě 0 ; momentum α P r0 , 1q . Output : papkq , xpkqq , a local minimizer of ΨBL . Initialize ap1q “ ap0q , xp1q “ xp0q . for k “ 1 , 2 , . . . until converged do Update x with accelerated proximal gradient step : wpkq Ð xpkq ` α ¨ ` xpkq ´ xpk´1q ˘ xpk ` 1q Ð softλtk “ wpkq ´ tk ¨∇xψλ ` apkq , wpkq ˘‰ , where softλpvq . “ signpvq dmaxp|v ´ λ| ,0q denotes the soft-thresholding operator . Update a with accelerated Riemannian gradient step : zpkq Ð PSp´1 ` apkq ` αxapkq , apk´1qy ¨ Papk´1q ` apkq ˘˘ apk ` 1q Ð PSp´1 ` zpkq ´ τk ¨ grada ψλ ` zpkq , xpk ` 1q ˘˘ . end for

Summary: The work considers sparse and short blind deconvolution problem, which is to inverse a convolution of a sparse source (such as spikes at cell locations in microscopy) with a short (of limited spatial size) kernel or point spread function, not known in advance. This is posed as a bilinear lasso optimization problem. The work applies a non-linear optimization method with some practical improvements (such as data-driven initialization, momentum, homotopy continuation) and applies it to 3 real imagine problems.