Title: Replicable Distribution Testing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Replicability

Distribution Testing
Abstract: We initiate a systematic investigation of distribution testing in the framework of algorithmic replicability. Specifically, given independent samples from a collection of probability distributions, the goal is to characterize the sample complexity of replicably testing natural properties of the underlying distributions. On the algorithmic front, we develop new replicable algorithms for testing closeness and independence of discrete distributions. On the lower bound front, we develop a new methodology for proving sample complexity lower bounds for replicable testing that may be of broader interest. As an application of our technique, we establish near-optimal sample complexity lower bounds for replicable uniformity testing---answering an open question from prior work---and closeness testing.


Peer reviews:
comments: SUMMARY:
The focus is on replicable testing (uniformity, closeness, etc); that is we are interested in testing algorithms that with high probability output the same thing on two different independent draws (using the same randomness). 

The main result of the paper is thm 1.3, setling down (up to polylog factors) the sample complexity of replicable uniformity testing, showing that the existing algorihm is in fact nearly optimal. As an aditional application of the developed techniques (which to some extent mimic the previous work up to a new random-walk argument), the authors provide lower bound on the sample complexity of closeness testing and give a (linear time) algorithm matching this lower bound up to polylog factor. The final interesting point of the paper I want to discuss here is the observation that replicable testing has suspiciously similar complexities with high probability testing and I like that it is emphasized as an open question.

STRENGTHS AND WEAKNESSES:
The paper is very dense and was not that easy to read, but this is to be expected as it is very technical and I do not have suggestions in this regard. The studied problem/field is rather new and is somewhat niche, but I believe the replicability is an important property of testers and should be understood. To this end, It is likely that this paper will become a canonical reference.

Overall, I consider the presented results in the paper to be very strong and (unless there is an error that I missed, **I haven't checked technical details**) should be clearly accepted. 


typos in alg 1, line "2: ".

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies replicable property testing of discrete distributions, under the shared-randomness replicability definition proposed by Impagliazzo et al. Recent work of Liu and Ye studied the specific problem of replicable uniformity testing. This paper strengthens and generalizes their results in several dimensions. First, it introduces a framework for proving replicable distribution testing lower bounds, which gives nearly tight results for both uniformity and closeness testing. The framework takes the same conceptual approach as Liu and Ye, who were able to prove lower bounds for symmetric testers, but bypasses the need for this assumption by identifying and studying a particular random walk to show that an arbitrary tester behaves as though it were symmetric in the right way.\n\nSecond, it designs new replicable testers for closeness and independence testing. Building again on the work of Liu and Ye, it argues more generally that testers which involve thresholding a test statistic can be made replicable by randomizing the threshold. Getting this to work for independence testing involves some additional ideas for stabilizing the test statistic, which were originally introduced for differential privacy.\n\nSTRENGTHS AND WEAKNESSES:\n(+) Distribution testing offers an important and well-studied class of problems as fodder for identifying optimal replicable algorithms.\n\n(+) We're still lacking strong / general-purpose techniques for proving replicability lower bounds, so the framework this paper offers (in particular, showing that the approach suggested by Liu and Ye does not require as strong assumptions about the structure of the learner as previously thought) is quite welcome.\n\n(+) Both the positive and negative results identify broadly applicable principles for replicable distribution testing and algorithmic replicability more generally.\n\n(+) The writing is clear and appears sound (though I did not check details carefully).\n\n(-) The paper's novelty lies in its technical contributions (which are nice), but it doesn't offer much that is new conceptually.\n\nQUESTIONS:\n* Does the lower bound framework give anything nontrivial for replicable independence testing?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of distribution testing with replicability constraints. Given samples from k distributions, the distribution testing problem aims, in general, aims to design a \"replicable\" tester (algorithm) that distinguishes between the cases when all k distributions satisfy a certain property or if one of them is far from the property. This work in particular focuses on uniformity testing, closeness testing, and independence testing problems. These distribution testing problems without the replicability constraint has been well studied. The main results show tight tradeoffs in sample complexity with this added constraint.\n\nSTRENGTHS AND WEAKNESSES:\nThe results are a good addition to the literature on distribution testing, and the techniques while similar to ref 27 (in the manuscript) require certain delicate balancing and are quite novel. The paper overall is quite well-written. The main technical differences from [27] while adequately addressed is a bit confusing at first without reading the definitions that follow in later sections. In my personal opinion, a slight rephrasing could help, but is not a major concern. Hence i would recommend accepting. \n\nConnection to Tolerant Testing: It is however unclear why a tolerant tester that distinguishes between $\\xi(\\rho)$, and $\\epsilon-\\xi(\\rho)$ for independence testing is not sufficient. At a high level, it seems intuitive that two independent sets of samples from the same distribution will be close, and hence from triangle inequality, should ensure that two induced distribution be either close or far from the property.  \n\nConnection to Private Testing: Another connection that has been leveraged, but not discussed is the connection to differentially private testing. This stems from the intuition that modifying even a single sample should not change the tester's output with high probability. Will the lower bounds of differentially private independence testing carry over to this setting?\n\nQUESTIONS:\nPlease see the previous section for connections to tolerant testing and private testing.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
Given sample access to a distribution, property testing is the problem of determining if the distribution has property $P$ or is far from any distribution with probability $P$. An algorithm is replicable if repeated runs of the algorithm give the same answer whp. In this case, that corresonds to consistently classifying a particular distribution the same way whp, regardless of the exact set of samples you get for the distribution. The paper gives upper and lower bounds for replicable property-testing algorithms for a couple of key properties, including tight bounds for closeness testing and n-item uniformity testing.

STRENGTHS AND WEAKNESSES:
+ Tight sample complexity bounds for replicability for some fundamental problems
+ Neat random walk argument to establish that on similar distributions the tester will behave similarly

? I did not verify the proofs, which are extremely long and technical

QUESTIONS:
There's an indirectly related proof of lower bounds for replicability in https://arxiv.org/pdf/2304.03757, https://arxiv.org/pdf/2503.15294 and https://arxiv.org/pdf/2304.02240v1 Both show roughly that you can't replicably distinguish between things if things more or less continuously change into each other. There's a follow-up with sample complexity stuff here https://openreview.net/pdf?id=2lL7s5ESTj.
Those papers required that there is a set $S$ of size $k$ s.t. $A[samples, randomness]\inS$ whp, which is slightly different but related to the constraint that $A[sample1, randomness] = A[sample2,same randomness]$ whp. But I think it could simplify your lower bound proofs.

From what I can tell, your main technical contribution is the random walk analysis to establish the space of similar distributions. Is that fair to say?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-10