Title: Differentially Private High-dimensional Variable Selection via Integer Programming
Venue: NeurIPS 2025
Accepted: yes
Keywords: Variable Selection

Best Subset Selection

Differential Privacy

Exponential Mechanism

Mixed Integer Programming

Sparse Linear Regression

Sparse Classification
Abstract: Sparse variable selection improves interpretability and generalization in high-dimensional learning by selecting a small subset of informative features. Recent advances in Mixed Integer Programming (MIP) have enabled solving large-scale non-private sparse regression—known as Best Subset Selection (BSS)—with millions of variables in minutes. However, extending these algorithmic advances to the setting of Differential Privacy (DP) has remained largely unexplored. In this paper, we introduce two new differentially private estimators for sparse variable selection, levering modern MIP techniques. Our framework is general and applies broadly to problems like sparse regression or classification, and we provide theoretical support recovery guarantees in the case of BSS. Inspired by the exponential mechanism, we develop structured sampling procedures that efficiently explore the non-convex objective landscape, avoiding the exhaustive combinatorial search in the exponential mechanism. We complement our theoretical findings with extensive numerical experiments, using both least squares and hinge loss for our objective function, and demonstrate that our methods achieve state-of-the-art empirical support recovery, outperforming competing algorithms in settings with up to $p=10^4$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the sparse variable selection in the context of Differential Privacy (DP), which aims to improve interpretability and generalization in high-dimensional learning while protecting the confidential and personal information. This paper introduces two new differentially private estimators for sparse variable selection based on modern MIP techniques. Their framework is broadly applicable, and they provide theoretical support recovery guarantees in the case of sparse regression. Finally, extensive numerical experiments, using both least squares and hinge loss, demonstrate the effectiveness and scalability of their methods.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is clearly written and well-structured. This paper is the first to apply MIP techniques for sparse variable selection in the context of DP. The work addresses an important problem in the field of variable selection.\n\nHowever, the computational results could be improved. Specifically, the authors do not discuss why the Top-R method performs better as the sample size n increases. Throughout the computational experiments, the sparsity parameter s is set to 5 or 8, much smaller compared to the number of features. This may not be representative. It is necessary to try larger values of s to evaluate the methods. \n\nWhile the two DP algorithms are general, the support recovery results are limited to the sparse regression setting. In addition, they only provide MIP formulations and algorithms for least squares and hinge loss objectives. It remains unclear how MIP techniques extend to general loss function.\n\nQUESTIONS:\nPlease see weakness above.\nA minor comment is: the notation $\\rho$  is used to represent different concepts.\n\nLIMITATIONS:\nSome results of this paper are limited to the sparse regression problem, as detailed in the weakness. It would be beneficial to extend these results to broader settings and include numerical results for a wider class of loss functions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces recent developments in Mixed Integer Programming (MIP) for the Best Subset Selection (BSS) problem to the differential privacy community. It presents two selection procedures: Top-R, which is based on the exponential mechanism and either selects one of the top R subsets or samples a subset uniformly at random; and the \"Mistakes\" method, which applies the exponential mechanism based on the distance (the number of mistakes) from the optimal solution. To run the Top-R and Mistakes mechanisms, one needs to get the top R subsets or the best subsets with a fixed number of mistakes, which the authors do by sequentially solving modified MIP problems. This approach allows private BSS problems to scale up to $10^4$ parameters.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors developed an efficient algorithm for differentially private Best Subset Selection by modifying the exponential mechanism to sample not from the full exponential space of all possible subsets, but only from the best ones. To my knowledge, this adaptation of the exponential mechanism is original, though similar ideas may potentially exist. I consider this paper an important contribution: in numerical experiments, the Mistakes and Top-R methods significantly outperform existing alternatives. Furthermore, the paper provides a non-trivial theoretical analysis of the utility guarantees of the proposed methods in terms of support recovery (Theorems 3 and 4). For these reasons, I support the acceptance of the paper.\n\nWeaknesses:\n\n1) The theoretical guarantees for support recovery appear to be weaker than those provided by the MCMC method [22]. \n\n2) The privacy analysis of the Mistakes method (Theorem 2) relies on extra assumptions that may not hold in practice, potentially limiting the method’s applicability.\n\n3) The presentation of the paper could also be improved. The conclusion section is unnecessarily short, and the font sizes in the plots are too small, making them difficult to read. I would also suggest renaming the \"Mistakes\" method, as the current name might be misleading. Furthermore, the formulas in the Appendix need to be reformatted to fit within the page margins.\n\nQUESTIONS:\nYou state that Algorithm 1 satisfies $(\\epsilon, 0)$-DP, but according to Theorem 1, it actually satisfies $(\\epsilon', 0)$-DP unless $T = \\infty$. Is this a typo?\n\nAre there similar modifications of the exponential mechanism in the literature? If so, it would be advisable to mention them in the related works section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the problem of sparse variable selection with differential privacy. Here we have some per-example loss (such as least squares or the hinge loss) on a model $\\beta$, a dataset $X, y$, and our goal is to find a model $\\beta$ that minimizes the empirical loss on $X$ subject to the model being chosen with differential privacy, and $\\beta$ having at most $s$ non-zero coordinates. Variable selection focuses just on selecting the right $s$ coordinates; in particular, the success of a variable selection algorithm is the fraction of non-zero coordinates in $\\beta^*$, the optimal model, that are also chosen by the algorithm.\n\nThe authors consider an exponential mechanism-based approach to this problem. In their first variant, the loss for choosing a certain set of $s$ coordinates is the minimum loss achieved by any model only supported on those coordinates. They are able to show a bound on the sensitivity of the exponential mechanism under mild assumptions (that can be enforced via clipping), but since there are $\\binom{p}{s}$ different sets of coordinates to choose, running this exponential mechanism is infeasible. They instead propose the following modification: rather than compute the loss for all sets, compute it for the best r sets. For all other sets, set their loss to the r-th best set's loss. They show this is also private, and in fact gets better privacy than the \"default\" value of $r = \\binom{p}{s}$ improving as r decreases. They also show that the top R supports can be computed by solving a series of mixed integer programs which are efficient in practice for p ~= 10000. This eliminates the need to enumerate all solutions. The second variant the authors propose is to set the loss in the exponential mechanism to be the number of wrong coordinates in the set. This is easier to sample from but requires a separation between the loss of the first and second for privacy.\n\nTheoretically, the authors show \"top-r\" recovers the best selection with high probability when the smallest non-zero coordinate of the optimal model is $\\Omega(\\sqrt{\\max\\{1, s^2 / \\epsilon\\} log p / n})$ for a moderate value of r. The non-private requirement is $\\Omega(\\sqrt{\\log p / n})$, so top-R matches the non-private rate up to constants when $\\epsilon \\rightarrow \\infty$ or when the number of coordinates to pick is small. Mistakes requires the non-zero coordinates to be $\\Omega(\\sqrt{(s / \\epsilon) \\cdot \\log p / n})$, i.e. again is optimal when $\\epsilon$ is sufficiently large but requires the additional separation condition for privacy. Empirically, the authors compare top-r and mistakes to an MCMC-based approach of the previous work of Roy et al., and also the Samp-Agg algorithm of Smith et al. Mistakes recovers almost all coordinates in most settings, and top-R outperforms Samp-Agg and outperforms the MCMC based approach when the dataset size is large enough.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Best coordinate selection is an interesting and well-studied problem.\n* A number of theoretical insights in combination are needed for the results. e.g. top-R requires the combination of a sensitivity bound which is non-trivial, a cleverly-designed MIP to isolate the top-R sets without brute-force enumeration, and the authors also give a privacy proof which shows that imposing a ceiling on the loss function in the exponential mechanism improves its utility.\n* The algorithms are tight in that match the non-private best rates of recoery as $\\epsilon \\rightarrow \\infty$.\n* The presentation makes the flow of ideas appear to be natural which makes for an easy read.\n\nWeaknesses:\n* The mistakes algorithm's privacy guarantee relies on a separation between the loss of the best and second-best datasets, i.e. the privacy guarantee doesn't hold for all pairs of datasets. While the authors argue under certain sampling assumptions for the datasets, this separation holds with high probability, DP is inherently a worst-case statement about all pairs of distributions so \"average-case\" DP definitions are generally problematic in nuanced ways (see e.g. https://differentialprivacy.org/average-case-dp/ for more discussion) and such definitions shouldn't be promoted in the literature. So the mistakes algorithm is probably not desirable (except maybe as a theoretical ideal) unless e.g. the separation condition can be forced using clipping-style arguments.\n* The empirical comparison studies the chance of recovering the right set of coordinates, but the final downstream objective is actually to minimize the model loss and not just find the best coordinates. Samp-Agg in particular does minimize the final loss, so a comparison to Samp-Agg is a bit unfair because (i) Samp-Agg is not optimizing for the highest % of correct coordinates, and (ii) Samp-Agg is spending part of its privacy budget on the model optimization, whereas top-R and mistakes do not spend any privacy budget on this aspect of the problem. \n* The combination of the two above points makes the empirical comparisons a bit unconvincing, since (i) mistakes has a problematic privacy guarantee, (ii) top-R does not uniformly beat MCMC, and (iii) top-R beats Samp-Agg, but that comparison is unfair.\n\nQUESTIONS:\n-Can the separation between the loss of the first and second-best solution be enforced in any clipping-like way? As discussed before, requiring a high-probability condition to hold for the DP guarantee of mistakes is somewhat problematic.\n-If we were to e.g. run objective perturbation or another DP-ERM algorithm over the coordinates selected by top-r or mistakes using the same overall privacy budget as Samp-Agg, would top-r and mistakes still outperform Samp-Agg?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSparse variable selection is important for high-dimensional data, it can improve the interpretability and reduce the overfitting of the models. This paper proposes two new and scalable differentiable privacy estimators for sparse variable selection, “top-R” and “mistake”. And both of them employ the idea of mixed integer programming to generate the candidate supports, avoiding the exhaustive search in the exponential mechanism. They also prove that these two estimators satisfy pure (ε,0)-DP. And the best subset selection matches the non-private minimax rate of top-R in the low privacy regime and those of mistakes in the high privacy regime. Experiments on least squares regression and hinge classification prove that the proposed method is able to recover the correct support better than SOTA, and is still computationally feasible.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1) This paper gave us an insight into using MIP solvers to approximate the exponential mechanism and solve the problem that effectively solves combinatorial optimization problems under the strong guarantee of pure-DP. \n2) The author provided solid theoretical proof. They provide a detailed proof for pure (ε,0)-DP and analyze the support recovery for BSS. \n3) They gave an empirical comparison with state-of-the-art methods, showing great results.\n\nWeakness: \n1) The proposed methods rely on several assumptions (1-4), but may not be practical for highly correlated features. How will privacy guarantees or support recovery degrade when these assumptions are not satisfied?\n2) Experiments only focus on support recovery (i.e., identifying non-zero coefficients), but do not mention the prediction accuracy. Furthermore, the paper should address how to choose several hyperparameters and discuss the sensitivity of choosing them.\n\nQUESTIONS:\nHow to choose the parameters R and λ? An ablation study would be useful.\nIn practice, how should one set the clipping bounds bx and by to balance bias and privacy?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}