Title: Refining Dual Spectral Sparsity in Transformed Tensor Singular Values
Venue: NeurIPS 2025
Accepted: no
Keywords: tensor recovery

tensor nuclear norm

low-rank decomposition

t-SVD
Abstract: The Tensor Nuclear Norm (TNN), derived from the tensor Singular Value Decomposition, is a central low-rank modeling tool that enforces *element-wise sparsity* on frequency-domain singular values and has been widely used in multi-way data recovery for machine learning and computer vision. However, as a direct extension of the matrix nuclear norm, it inherits the assumption of *single-level spectral sparsity*, which strictly limits its ability to capture the *multi-level spectral structures* inherent in real-world data—particularly the coexistence of low-rankness within and sparsity across frequency components. To address this, we propose the tensor $\ell_p$-Schatten-$q$ quasi-norm ($p, q \in (0,1]$), a new metric that enables *dual spectral sparsity control* by jointly regularizing both types of structure. While this formulation generalizes TNN and unifies existing methods such as the tensor Schatten-$p$ norm and tensor average rank, it differs fundamentally in modeling principle by coupling global frequency sparsity with local spectral low-rankness. This coupling introduces significant theoretical and algorithmic challenges.  To tackle these challenges, we provide a theoretical characterization by establishing the first minimax error bounds under dual spectral sparsity, and an algorithmic solution by designing an efficient reweighted optimization scheme tailored to the resulting nonconvex structure. Numerical experiments demonstrate the effectiveness of our method in modeling complex multi-way data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the $\\ell_p$-Schatten-$q$ quasi-norm for tensors to jointly characterize the slice-wise low-rankness and the frequency sparsity under the t-SVD framework using frequency-domain transformation. When $p=1$ the framework recovers the Schatten-$q$ TNN. The authors give the worst-case minimax bounds for the parameter spaces induced by the dual assumptions. Experiments are conducted on noisy tensor completion to show the performance of the new tensor norm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The motivation is clear. Characterizing the sparsity of frequency components jointly under the TNN framework is indeed helpful for better tensor structure modeling. The framework jointly considers two types of priors (namely low-rankness inside a frequency and the global frequency sparsity) using a single regularizer, which is concise and efficient.\n\n2. The presentation is clear and the manuscript is well-structured, although may not be friendly for readers not doing t-SVD research.\n\n3. The authors did a rigorous and comprehensive job on establishing the theoretical minimax bounds of the parameter spaces induced by dual assumptions. \n\nWeaknesses:\n\n1. The first major weakness I found is that this framework needs an exhaustive grid search to determine the optimal value for $p$ and $q$, which is not needed by conventional TNN frameworks. Although the optimal parameters $(0.8961,0.8966)$ are found in the experiments at hand, I was curious about whether the parameter combination is generalizable to different tensor datasets. For instance, one may find different optimal parameters for different types of data, which brings a significant hyperparameter optimization challenge.\n\n2. While the authors presented a Proximal Operator Formulation and a relaxed singular value thresholding (SVT) algorithm for the $\\ell_p$-Schatten-$q$ quasi-norm, it is unclear how the computational complexity compares with standard SVT used in TNN. Moreover, the algorithm employs a reweighted $\\ell_{1/2}$-surrogate on the norm, which is completely independent of $p,q$, and $p,q$ are only used in the weights $w$. This actually recovers to the weighted nuclear norm-type methods. Could the authors give more elaborations on the discrepancy between the actual $\\ell_p$-Schatten-$q$ quasi-norm and the introduced surrogate norm (e.g. error analysis)?\n\n3. While the authors mainly focus on the DCT as the frequency transform, it was not sufficient to show the consistent performance gain of the new tensor norm. Have other transforms been tested? (e.g. DFT, framelets, or any invertible linear transforms?) Could the proposed framework gain improvements by leveraging different transforms?\n\n4. Finally, the performance gain seems limited by comparing TNN-DCT and the proposed method (in some cases less than 1 dB), given that the proposed method is based on careful parameter control of $p,q$, and furthermore, TNN-DCT is not the state-of-the-art in TNN methods. Are such improvements robust w.r.t. $p,q$? How does the proposed weighting scheme in Eq. (11) compare with conventional weighted-nuclear norm-type methods?\n\nOverall, my main assessment is based on the limitations, including exhaustive grid search of hyperparameters, the discrepancy between the introduced $\\ell_p$-Schatten-$q$ quasi-norm and the actual implementation, and the limited performance gain and comparison.\n\nQUESTIONS:\n1. Could you further discuss the robustness and selection of $p,q$ in a more comprehensive manner?\n\n2. Could you add the computational complexity of the proximal step?\n\n3. Could you show that the proposed tensor norm could gain improvements over different transformations? (DFT, framelets, or any invertible linear transform)\n\n4. Could you add comparisons with more state-of-the-art TNN models? (e.g., framelet TNN [R1] and adaptive transform TNN [R2], by combining the new tensor norm)\n\n[R1] Framelet Representation of Tensor Nuclear Norm for Third-Order Tensor Completion\n[R2] Tensor Q-rank: new data-dependent definition of tensor rank\n\n5. Finally, the noisy tensor completion is not a primary task considered in previous TNN works. Could you assess the performance of the proposed method on standard tensor completion without noise?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-01"}

comments: SUMMARY:
This paper considers the Gaussian location model (GLM).

The contribution includes:

(1) Within the t-svd framework, the paper rigorously formalize the GLM, where inter-frequency sparsity coexists with intra-frequency low-rankness.  The tensor  $\ell_q$-Schatten-$q$ quasi-norm ($p, q \in  (0, 1]$) is proposed, which  enables dual spectral  sparsity control by jointly regularizing both types of structure. 

(2) Under three dual spectral sparsity assumptions, theorem 4.2 establishes min-max bounds, which quantify the worst-case squared Frobenius norm errors that any  estimator must incur when recovering a structured tensor from noisy observations. 

(3) An algorithm is proposed to solve the proposed optimization problem, and numerical experiments show the merits of the methods.

STRENGTHS AND WEAKNESSES:
Strength: The main contribution of this paper is in theory — theorem 4.2: the min-max bound which quantifies the errors that an estimator much incur when recovering the tensor from noisy data.

Weakness: The algorithm requires tsvd of matrices, which is expensive for large tensors. So, it is not appropriate to call it “scalable”. Perhaps, re-parameterize the frontal slices of the tensor in a low rank form is a solution, just as in the matrix completion.

QUESTIONS:
1 SVD is used in the numerical method, which is not suitable for large problems. Thus, the method is not scalable as declared.

2 Convergence of the numerical method is not given.

3 Is the numerical method able to achieve the established theoretical bounds?

4 Currently, the transformation matrix $M$ is fixed/prescribed. Is it possible to make it learnable/optimized?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors propose a novel regularizer for order-3 tensors, building on the t-SVD (tensor singular value decomposition) framework, a tensor decomposition which generalizes the SVD to order-3 tensors in a semantically meaningful yet computationally tractable manner. This regularizer is a mixed $\\ell_p$-Schatten-$q$ quasi-norm with $p, q \\in (0,1]$ and generally non-convex, generalizing $\\ell_{p,q}$ quasi-norms that have been studied for group sparsity problems. This quasi-norm is designed to better and more flexibly capture multi-level spectral structures present in real-world third-order data tensors.\nThe authors provide an interpretation of the parameters $p$ and $q$ in the context of the data tensors ($p$ governs sparsity among different frequency components and $q$ controls the low-rankness within each frequency component). The manuscript then continues to establish a theory for a denoising problem an estimator based on the $\\ell_p$-Schatten-$q$ quasi-norm given noisy observations, for which minimax bounds that match certain lower bounds are established (Theorem 4.2, proofs are not contained in the main paper). While the use of the regularizer comes with intrinsic computational challenges due to its non-convexity for $p,q < 1$, the authors propose a proximal algorithm based on reweighted optimization that is able to somewhat handle the minimization of the regularizer. In Section 6, the authors use the $\\ell_p$-Schatten-$q$ quasi-norm and proposed optimization scheme to solve tensor reconstruction problems from incomplete noisy observations based on real-data tensors arising in hyperspectral and thermal imaging. If used in conjunction with a discrete cosine transform, this quasi-norm leads to improved peak signal-to-noise ratio compared to other low-rank tensor regularizers, including the tensor nuclear norm, which coincides with the $\\ell_p$-Schatten-$q$ in the special case of $p=q=1$. \n\nIn the supplementary material, the minimax rates of Theorem 4.2 are established using advanced tools from information theory and statistics.\n\nSTRENGTHS AND WEAKNESSES:\nA main concern in the assessment of this submission is that it is questionable if the scope of the paper fits into a machine learning conference. From my perspective, the submission would much better fit into an image processing, applied mathematics or data science journal. Reasons for this are that the tasks on which the methodology is evaluated are rather artificial (noisy tensor completion) as it assumed that the ground truth is known to tune the problem parameters, and as the majority of related works (works considered t-SVD and related regularization notions) is also located within venues from the above areas, while this submission not making a particular connection to machine learning. Furthermore, the majority of the scientific value of this paper hinges on the correctness of the proofs and proof techniques used for establishing the minimax bounds of Theorem 4.2, which are all contained in the supplementary material. In a NeurIPS review, the time that a reviewer can allocate to checking the correctness of the proofs is limited, which does not do justice to the potentially deep results established in this submission.\n\nI will nevertheless point out some strong and weak points I noticed in this submission below.\n\n**Strengths:**\n- The proposed quasi-norm for third-order tensors is novel and addresses a relevant challenge in tensor modelling of data - how to design a low-complexity notion and respective computational representation that best captures internal dependencies of real-world date. The $\\ell_p$-Schatten-$q$ quasi-norm for $3$-tensors can be seen as a flexible framework in this direction.\n- The minimax bounds of Theorem 4.2 (which also extend to combinatorial variants of the proposed regularization) appear to be sharp and, if correct, could capture the potential information theoretical benefits of the regularization rather well (despite the setup considered being different than what is expected in application scenarios, where a completion/recovery setup might be more powerful than a denoising setup).\n- While the scope of the paper does not fit quite into the 9 pages and many important proofs and discussions of related concepts and related works are postpone into the supplementary material, the writing, related work discussion and proof structure is quite clear.\n\n**Weaknesses**:\n- While the $\\ell_p$-Schatten-$q$ quasi-norm is advertised as a flexible regularization framework that can capture flexibly intra-frequency low-rankness and inter-freuqency sparsity dependent on particularities of the data, when the method is used in Section 6, a global tuning of $p$ and $q$ is performed that leads to a value of $p,q \\approx 0.896(7)$. These values, are, on the one hand, rather close to $p=q=1$, as in the case of the well-known tensor nuclear norm, but they also seem to be _not_ dataset dependent at all. This suggests that in practice, it would be hard to make use of the flexibility that the framework provides.\n- In the comparison of Table 1 / Section 6, TNN-DCT and the proposed method perform rather similarly, with an advantage of $\\ell_p(S_q)$ mostly in the PSNR, but not exhibiting in the SSIM metric. The large performance drop of the other regularization approaches could be potentially explained by the lack of usage of the DCT transformation, as TNN-DCT and $\\ell_q(S_q)$(proposed) are the only methods that use this preprocessing transformation. Thus, the provided experiments might be too premature to be convincing that $\\ell_p(S_q)$ provides a significant benefit for tensor completion problems. \n- The authors are rather transparant about this, but it is unclear if the proposed optimization scheme (ADMM intertwined with reweighted optimization and $\\ell_{1/2}$-soft-thresholding) is able to find good minimizers of the objective in practice. An empirical study of this question (or comparison with other solvers) would have provided better insights on this question.\n- Maybe I misunderstand what is being done, but the tuning methodology of the parameters $p$ and $q$ described in lines 1275-1287 is rather concerning, as it seems to explicitly assume access to / knowledge of the ground truth, which will not be available in ML applications, limiting the practicality of the proposed regularizer.\n\nQUESTIONS:\n1. Was the tuning of $p$ and $q$ not performed in a dataset-dependent manner? If a feasible way of doing that would be available, the flexibility of the framework could better be harnessed.\n2. How do other methods of Table 1 perform if preprocessed with a DCT transformation? Having this data would provide a better apple-to-apple comparison, as the effect of the regularizers would better be isolated.\n\nIn my ratings, I am taking into account that the paper is somewhat out of scope for the conference and that checking the proofs of the paper is beyond the scope of this review.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This work proposes the tensor $l_p$-Schatten-$q$ quasi-norm as a new metric to capture the multi-level spectral structure inherent in real-world data—particularly the coexistence of low-rankness within and sparsity across frequency components. Both rigorous theoretical analyses and extensive numerical experiments are conducted to show the merits of the proposed framework in handling complex multi-way data.

STRENGTHS AND WEAKNESSES:
1. Both rigorous theoretical analyses and extensive numerical results are provided.

2. A detailed discussion of the advancements of the proposed  $l_p$-Schatten-$q$ quasi-norm over TNN and $l_\mu(l_q)$ norms are provided. This highlights the technical novelty of the proposed procedure. 

3. The process of selecting good hyperparameters $(p, q)$ seems to be too complicated, which limits the proposed procedure for practical use.

4. The converge behave of the ADMM based solving algorithm seems to be always guaranteed.

QUESTIONS:
1. In the process of parameter tuning, if the ratio of the observed entries is relatively high (e.g., 40%), the coarse grid search procedure seems to be overfitting?

2. How about the noise tensor beyond Gaussian distribution?

LIMITATIONS:
See Weaknesses and Questions parts.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-09