Title: On Traceability in $\ell_p$ Stochastic Convex Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: generalization theory

differential privacy

learning theory
Abstract: In this paper, we investigate the necessity of traceability for accurate learning in stochastic convex optimization (SCO) under $\ell_p$ geometries. Informally, we say a learning algorithm is \emph{$m$-traceable} if, by analyzing its output, it is possible to identify at least $m$ of its training samples. Our main results uncover a fundamental tradeoff between traceability and excess risk in SCO. For every $p\in [1,\infty)$, we establish the existence of an excess risk threshold below which every sample-efficient learner is traceable with the number of samples which is a \emph{constant fraction} of its training sample. For $p\in [1,2]$, this threshold coincides with the best excess risk of differentially private (DP) algorithms, i.e., above this threshold, there exist algorithms that are not traceable, which corresponds to a sharp phase transition. For $p \in (2,\infty)$, this threshold instead gives novel lower bounds for DP learning, partially closing an open problem in this setup. En route to establishing these results, we prove a sparse variant of the fingerprinting lemma, which is of independent interest to the community.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the complexity and tractability of $\\ell_p$-approximation problems in the context of information-based complexity. Specifically, it addresses traceability in the setting of deterministic linear approximation schemes for compact linear operators between Banach spaces, focusing on the minimal worst-case error achievable by algorithms using $n$ pieces of information. The core of the paper lies in characterizing traceability via the asymptotic behavior of approximation numbers $a_n(T)$, with particular attention paid to the cases $1 \\leq p < \\infty$.\n\nThe authors introduce and analyze a notion of $(\\xi, m)$-weak tractability in the $\\ell_p$-approximation context and relate it to polynomial and quasi-polynomial tractability. They extend existing results from the Hilbert space setting to more general Banach spaces, yielding both necessary and sufficient conditions for various types of tractability.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThe paper demonstrates a strong command of empirical process, approximation theory, and the machinery of information-based complexity. The results are technically non-trivial and reflect a significant extension of previous work. The main contribution are as follows:\n\n* For $1 < p \\le 2$: any learner that achieves excess-risk $\\alpha$ below the best differentially-private risk $\\alpha_{\\mathrm{DP}}$ must leak at least $\\Omega(1/\\alpha^{2})$ training samples.  \n* For $p=1$ the bound sharpens to $\\Omega\\!\\bigl(\\log d \\,/\\,\\alpha^{2}\\bigr)$, closing the $\\log d$ gap left by Dwork et al.\\ (2015).\n* For $p>2$: the lower bound becomes $\\Omega(1/\\alpha^{p})$, matching sample-complexity in the over-parameterised regime $d\\gtrsim n$.\n* Introduces a **sparse fingerprinting lemma** that extends classical hyper-cube fingerprinting to sparse inputs and drives the new bounds.\n* Defines a **sub-Gaussian trace value** that unifies (i) traceability lower bounds, (ii) DP lower bounds, and (iii) non-private sample-complexity bounds.\n* Improves the best known high-dimensional DP lower bounds for $\\ell_p$-SCO with $p>2$.\n\n### Weaknesses\n* Exposition is dense; more intuition and concrete examples would help non-experts.\n* Many constants and logarithmic factors are hidden—explicit rates would clarify sharpness.\n\nQUESTIONS:\nI'm not an expert in this area and do not have any technical questions.\n\nLIMITATIONS:\nNo experiments involved.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of traceability in stochastic convex optimization under $l_p$ geometries. Intuitively, traceability measures how the excess risk threshold determines the fraction of training samples that can be identified by an adversary. The traceability of an algorithm depends critically on the value of $p$, which defines the $l_p$ diameter of the feasible region.\n\n- More specifically, for $p \\in [1, \\infty)$, this paper establishes that there exists a risk threshold such that any algorithm achieving excess risk below this level must effectively memorize a constant fraction of its training data.\n\n- In addition, the paper investigates how this excess risk threshold relates to the number of training samples that can be traced, as a function of the best achievable excess risk of differentially private (DP) algorithms in comparable settings.\n\n- The paper also develops a sparse fingerprinting lemma, showing that over sparse domains, the correlation between a learner’s output and its training samples is inversely proportional to the sparsity parameter.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- This is an important problem, as it relates to whether the learner is simply memorizing the training data or learning the underlying structure from the samples.\n- This paper presents many results, and the proofs are very thorough.\n- The introduction of this paper is well written and presents the main results clearly.\n\nWeakness: \n- There is no contribution section, but for a theory paper, this is acceptable. \n- There are too many notations in this paper, and some of them are used inconsistently, which makes it very difficult for readers who want to fully understand the technical contributions. For example, the definition of $A_n$ in line 125 does not align with the one given in Equation (2).\n\nOther Comments:\n- It might be helpful to add columns in Table 1 for comparison with existing results.\n\nQUESTIONS:\nHow do you assess the significance of your results compared to existing work, and why?\n\nLIMITATIONS:\nIn the checklist, the authors claim that they have discussed this in the related work section, but it is unclear to the reviewer what the limitations actually are.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers an interesting framework on finding the fundamental tradeoff between the learnability of model parameters and the traceability of the dataset. The authors focused on a general class of loss functions that are characterized by $\\ell_p$ conditions and showed that when the excess error belongs to certain intervals, it is guaranteed that a certain fraction of the training dataset can be recovered from the learned parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \nThe problem considered in this work is well-motivated and built on sound principles from PAC. If the proposed results are valid, they would address timely and relevant challenges on how to maintain data security guarantees throughout the learning process.\n\nWeakness: \nThe critical concern of this work is about the validity (or specifically, inapplicability) of the main theorems. For instance, the main theoretical guarantee in Theorem 1 is only applicable when the learning error achieved by the algorithm falls into a regime of $n=O(d)$, which is already quite restricted. However, it appears that this entire regime could not be met at all except for a few corner cases, since the reviewer suspect the minimum sample complexity in the authors' formulation for p=1 is $d/\\alpha^2$ instead of $log(d)/\\alpha^2$, which is larger than the upper bound in Thm 1 and 2, rendering the claimed safety guarantee vacuous (see the question section for details). This issue is rooted from the fact that the benchmark learning error the authors used for the optimal sample complexity for small p is is fact the complexity for the $p=\\infty$ case ($log(d)/\\alpha^2$ as mentioned on page 2) instead of the correct rate for p=1.\n\nQUESTIONS:\n1. The authors are recommended to check or refer to rigorous proofs for the optimal sample complexity rates for p=1. If we follow the definition 2.2, then a valid cost function would be $f(z,\\theta)=\\frac{1}{2}||z-\\theta||_1$, where z is chosen to be uniformly random on the standard hypercube. In this case the sample complexity of the mean estimator would result in an error that scales linearly w.r.t. d instead of log d reported in table 1.  \n\n2 The authors frame the problem as SCO, which often instead describe a framework where the data points are obtained iteratively. The authors are recommended to refer the formulation with an appropriate terminology.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the question of traceability in stochastic convex optimization with respect to the $L_p$-norm. The setting involves assuming the existence of a parameter space $\\Theta$ that is convex and has bounded $L_p$-diameter, and a data space $\\mathcal{Z}$. Given samples from an unknown distribution $\\mathcal{D}$ over $\\mathcal{Z}$, the goal is to identify a $\\widehat{\\theta} \\in \\Theta$ that approximately minimizes $\\underset{Z \\sim \\mathcal{D}}{\\mathbb{E}}[f(\\theta, Z)]$, where $f \\colon \\Theta \\times \\mathcal{Z} \\to \\mathbb{R}$ is a function such that $f(\\cdot, z)$ is convex and $L_p$-Lipschitz for all $z \\in \\mathcal{Z}$. The problem of traceability has to do with the extent to which algorithms that solve the above problem have to memorize input datapoints, which would result in tracing algorithms being able to recover a significant part of the dataset with good probability. It is known that differentially private algorithms can be used to address the previous concern. However, the extent to which tracing is possible is not fully understood when one uses a non-private algorithm with error rate lower than that of the best private algorithm.\nThe present paper contributes to the understanding of the above question, by showing that, if one considers learners with error slightly lower than the minimax DP rate (for $\\varepsilon = \\Theta(1)$), then it becomes possible to trace a constant fraction of the input dataset, that is, the number of traceable samples is of the same order of magnitude as the input dataset. Three different results are given, corresponding to different ranges for $p$ ($p = 1, p \\in (1, 2]$, and $p > 2$). The results require proving new ``sparse'' fingerprinting-type lemmas (i.e., lemmas for distributions supported over vectors with a small number of non-zero components). As a by-product, a new lower bound for DP-SCO for $p > 2$ is obtained, as well as an improved traceability result for mean estimation in $L_{\\infty}$-norm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The questions of memorization and traceability are of significant importance to the machine learning literature, and so is the problem of SCO. Additionally, the main result about constant fraction traceability being achievable when one has error slightly below the DP minimax risk is conceptually interesting. Thus, the results are definitely of interest to the community of NeurIPS.\n2) In the past, fingerprinting arguments have focused on algorithms for parameter estimation problems attaining small mean-squared error. However, due to the different focus in this work, it is necessary to consider distributions parameterized by points in the largest $L_{\\infty}$-ball that can be inscribed into the $L_p$-unit ball. The constructions considered for the distributions supported on sparse vectors (and the proofs of the associated fingerprinting lemmas for the subgaussian trace value) exhibit a degree of insight and technical novelty which I definitely consider to be enough for the paper to be accepted.\n3) The presentation and writing were accessible.\n\nWeaknesses:\n1) The authors mention that they found ``the ability of trace value to recover non-private lower bounds surprising''. I don't think this is really surprising, and I can explain why. The lower bounds obtained using the trace value rely on fingerprinting. Fingerprinting requires one to reason about the correlation between the input dataset and the output distribution of an algorithm. Within the context of the DP literature, the correlation terms are upper-bounded using the definition of DP. However, it is always possible to upper-bound these terms using the Cauchy-Schwarz inequality (or Holder's inequality), which would lead to non-private lower bounds (not sure if this is explicitly written somewhere). Thus, given the lower bounds reliance on fingerprinting-type lemmas, it doesn't seem entirely unexpected that trace value-based arguments can recover non-private lower bounds. That said, this is not a serious criticism by any means, since it's clear from the proofs in the appendix that implementing this in the present setting is not as simple as suggested above, but requires a non-trivial amount of work.\n\nOverall, I consider this to be a good work that merits acceptance.\n\nQUESTIONS:\nI don't really have any questions for the time being, but I can't rule out questions arising based on other reviewers' comments during the discussion period.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}