Beyond the Signs: Nonparametric Tensor Completion via Sign Series

1 Introduction . Higher-order tensors have recently received much attention in enormous fields including social networks [ 3 ] , neuroscience [ 38 ] , and genomics [ 26 ] . Tensor methods provide effective representation of the hidden structure in multiway data . In this paper we consider the signal plus noise model , Y = Θ + E , ( 1 ) where Y ∈ Rd1×···×dK is an order-K data tensor , Θ is an unknown signal tensor of interest , and E is a noise tensor . Our goal is to accurately estimate Θ from the incomplete , noisy observation of Y . In particular , we focus on the following two problems : Q1 [ Nonparametric tensor estimation ] . How to flexibly estimate Θ under a wide range of structures , including both low-rankness and high-rankness ? Q2 [ Complexity of tensor completion ] . How many observed tensor entries do we need to consistently estimate the signal Θ ? Inadequacies of low-rank models . The signal plus noise model ( 2 ) is popular in tensor literature . Existing methods estimate the signal tensor based on low-rankness of Θ [ 28 , 34 ] . Common low-rank models include Canonical Polyadic ( CP ) tensors [ 24 ] , Tucker tensors [ 11 ] , and block tensors [ 40 ] . While these methods have shown great success in theory , tensors in applications often violate the low-rankness . Here we provide two examples to illustrate the limitation of classical models . The first example reveals the sensitivity of tensor rank to order-preserving transformations . Let Z ∈ R30×30×30 be an order-3 tensor with tensor rank ( Z ) = 3 ( formal definition is deferred to the end of this section ) . Suppose a monotonic transformation f ( z ) = ( 1 + exp ( −cz ) ) −1 is applied to Z 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . entrywise , and we let the signal Θ in model ( 1 ) be the tensor after transformation . Figure 1a plots the numerical rank ( see Appendix B.1 ) of Θ versus c. As we see , the rank increases rapidly with c , rending traditional low-rank tensor methods ineffective in the presence of mild order-preserving nonlinearities . Similar observations apply to both CP and Tucker models , and more generally , to lowrank models with scale-sensitive rank measures . In digital processing [ 17 ] and genomics analysis [ 26 ] , the tensor of interest often undergoes unknown transformation prior to measurements . The sensitivity to transformation makes the low-rank model less desirable in practice . The second example demonstrates the inadequacy of classical low-rankness in representing special structures . Here we consider the signal tensor of the form Θ = log ( 1 +Z ) , where Z ∈ Rd×d×d is an order-3 tensor with entries Z ( i , j , k ) = d−1 max ( i , j , k ) for i , j , k ∈ { 1 , . . . , d } . The matrix analogy of Θ was studied in graphon analysis [ 8 ] . In this case neither Θ nor Z is low-rank ; in fact , the rank is no smaller than the dimension d as illustrated in Figure 1b . Again , classical low-rank models fail to address this type of tensor structure . In the above and many other examples , the signal tensors Θ of interest have high rank . Classical low-rank models will miss these important structures . The observations have motivated us to develop more flexible tensor modeling . Our contributions . We develop a new model called sign representable tensors to address the aforementioned challenges . Figure 2 illustrates our main idea . Our approach is built on the sign series representation of the signal tensor , and we propose to estimate the sign tensors through a series of weighted classifications . In contrast to existing methods , our method is guaranteed to recover a wide range of low- and high-rank signals . We highlight two main contributions that set our work apart from earlier literature . Statistically , the problem of high-rank tensor estimation is challenging . Existing estimation theory [ 3 , 34 , 7 ] exclusively focuses on the regime of fixed r and growing d. However , such premise fails in high-rank tensors , where the rank may grow with , or even exceed , the dimension . A proper notion of nonparametric complexity is crucial . We show that , somewhat surprisingly , the sign tensor series not only preserves all information in the original signals , but also brings the benefits of flexibility and accuracy over classical low-rank models . The results fill the gap between parametric ( low-rank ) and nonparametric ( high-rank ) tensors , thereby greatly enriching the tensor model literature . Computationally , a number of polynomial-time algorithms are readily available under moderate-tohigh signal-to-noise ratio for 1-bit tensor estimation [ 39 , 20 , 17 ] . These algorithms enjoy computational efficiency while being restricted to binary inputs . Our work is orthogonal to these algorithm development , and we show that the high-rank tensor estimate is provably reducible to a series of binary tensor problems with carefully-designed weights . This reduction provides a generic engine to empower existing algorithms for a wider range of structured tensor problems . We use a divideand-concur approach to combine efficient base algorithms , thereby achieving computational accuracy without the need to reinvent the wheel . The flexibility to import and adapt existing tensor algorithms is one advantage of our method . We also highlight the challenges associated with tensors compared to matrices . High-rank matrix estimation has been studied under graphon models [ 41 , 44 , 8 ] , nonlinear models [ 15 ] , and subspace clustering [ 35 , 13 ] . In particular , the recent work [ 30 ] proposes a general nonparametric framework to address a variety of matrix problems including regression and completion . However , high-rank tensor problems is more challenging , because the tensor rank often exceeds the dimension when order K greater than two [ 4 ] . This is in sharp contrast to matrices ( K = 2 ) . We show that , applying matrix methods to higher-order tensors results in suboptimal estimates . A full exploitation of the higher-order structure is needed ; this is another challenge we address in this paper . Notation . We use [ n ] = { 1 , . . . , n } for n-set with n ∈ N+ , an . bn if limn→∞ an/bn ≤ c for some constant c > 0 , and an bn if c1 ≤ limn→∞ an/bn ≤ c2 for some constants c1 , c2 > 0 . We use O ( · ) to denote the big-O notation , Õ ( · ) the variant hiding logarithmic factors . Let Θ ∈ Rd1×···×dK denote an order-K ( d1 , . . . , dK ) -dimensional tensor , and Θ ( ω ) ∈ R denote the tensor entry indexed by ω ∈ [ d1 ] × · · · × [ dK ] . An event A is said to occur “ with very high probability ” if P ( A ) tends to 1 faster than any polynomial of tensor dimension d : = mink dk →∞ . The tensor rank [ 24 ] is defined by rank ( Θ ) = min { r ∈ N : Θ = ∑r s=1 a ( 1 ) s ⊗ · · · ⊗ a ( K ) s } , where a ( k ) s ∈ Rdk are vectors for k ∈ [ K ] , s ∈ [ r ] , and ⊗ denotes the outer product of vectors . We use sgn ( · ) : R→ { −1 , 1 } to denote the sign function , where sgn ( y ) = 1 if y ≥ 0 and −1 otherwise . We allow univariate functions , such as sgn ( · ) and general f : R→ R , to be applied to tensors in an element-wise manner . 2 Model and proposal overview . Let Y be an order-K ( d1 , . . . , dK ) -dimensional tensor generated from the model Y = Θ + E , ( 2 ) where Θ ∈ Rd1×···×dK is an unknown signal tensor , and E is a noise tensor consisting of zero-mean , independent but not necessarily identically distributed entries . We allow heterogenous noise , in that the marginal distribution of noise entry E ( ω ) may depend on ω . For a cleaner exposition , we assume the noise is bounded and the range of observation is in [ −1 , 1 ] ; the extension to unbounded observations with sub-Gaussian noise is provided in Appendix B.3 . Our observation is an incomplete data tensor from ( 2 ) , denoted YΩ , where Ω ⊂ [ d1 ] × · · · × [ dK ] is the index set of observed entries . We consider a general model on Ω that allows both uniform and non-uniform samplings . Specifically , let Π = { pω } be an arbitrarily predefined probability distribution over the full index set with∑ ω∈ [ d1 ] ×···× [ dK ] pω = 1 . We use ω ∼ Π to denote the sampling rule , meaning ω in Ω are i.i.d . draws with replacement from distribution Π . The goal is to estimate Θ from YΩ . Note that Θ is not necessarily low-rank . Proposal intuition . Before describing our main results , we provide the intuition behind our method . In the two examples in Section 1 , the high-rankness in the signal Θ makes the estimation challenging . Now let us examine the sign of the π-shifted signal sgn ( Θ − π ) for any given π ∈ [ −1 , 1 ] . It turns out that , these sign tensors share the same sign patterns as low-rank tensors . Indeed , the signal tensor in the first example has the same sign pattern as a rank-4 tensor , since sgn ( Θ− π ) = sgn ( Z − f−1 ( π ) ) . The signal tensor in the second example has the same sign pattern as a rank-2 tensor , since sgn ( Θ ( i , j , k ) −π ) = sgn ( max ( i , j , k ) −d ( eπ−1 ) ) ( see Example 5 in Section 3 ) . The above observation suggests a general framework to estimate both low- and high-rank signal tensors . Figure 2 illustrates the main crux of our method . We propose to estimate the signal tensor Θ by taking the average over structured sign tensors Θ̂ = 1 2H + 1 ∑ π∈H sgn ( Ẑπ ) , where Ẑπ = arg min low rank tensor Z Weighted-Loss ( sgn ( Z ) , sgn ( YΩ−π ) ) . ( 3 ) Here sgn ( Ẑπ ) ∈ { −1 , 1 } d1×···×dK is the sign tensor estimated at a series of π ∈ H = { −1 , . . . , −1/H , 0 , 1/H , . . . , 1 } , and Weighted-Loss ( · , · ) denotes a classification objective function with an entry-specific weight to each tensor entry ; its specific form will be described in Section 3.2 . To obtain sgn ( Ẑπ ) for a given π , we propose to dichotomize the data tensor into a sign tensor sgn ( YΩ− π ) and estimate the de-noised sign by performing weighted classification . Our approach is built on the nonparametric sign representation of signal tensors . We show that a careful aggregation of dichotomized data preserves all information in the original signals and brings benefits of accuracy and flexibility over classical low-rank models . Unlike traditional methods , the sign representation is guaranteed to recover both low- and high-rank signals . In addition , a total of H = poly ( d ) dichotomized problems suffice to recover Θ under the considered model . The method therefore enjoys both statistical effectiveness and computational efficiency .

Summary: This work studies the problem of tensor estimation from noisy observations with possibly missing entries. A nonparametric approach to sign tensor completion is developed. which can deal with both low-rank and high-rank tensor decomposition. Theoretical analysis is provided in terms of risk bound, estimation error, and sample complexity.