Title: Channel Simulation and Distributed Compression with Ensemble Rejection Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Channel Simmulation

Distributed Compression

Probabilistic Method
Abstract: We study channel simulation and distributed matching, two fundamental problems with several applications to machine learning, using a recently introduced generalization of the standard rejection sampling (RS) algorithm known as Ensemble Rejection Sampling (ERS). For channel simulation, we propose a new coding scheme based on ERS that achieves a near-optimal coding rate. In this process, we demonstrate that standard RS can also achieve a near-optimal coding rate and generalize the result of Braverman and Garg (2014) to the continuous alphabet setting. Next, as our main contribution, we present a distributed matching lemma for ERS, which serves as the rejection sampling counterpart to the Poisson Matching Lemma (PML) introduced by Li and Anantharam (2021). Our result also generalizes a recent work on importance matching lemma (Phan et al, 2024) and, to our knowledge, is the first result on distributed matching in the family of rejection sampling schemes where the matching probability is close to PML. We demonstrate the practical significance of our approach over prior works by applying it to distributed compression. The effectiveness of our proposed scheme is validated through experiments involving synthetic Gaussian sources and distributed image compression using the MNIST dataset.


Peer reviews:
comments: SUMMARY:
This paper proposes Ensemble Rejection Sampling (ERS) for channel simulation and distributed compression. Theoretically, ERS achieves near-optimal coding rates matching for channel simulation and distributed matching probabilities competitive with the Poisson Matching Lemma (PML) as batch size increases. Experiments on Gaussian sources and MNIST validate ERS’s advantages over importance sampling (IML) in bias reduction and distortion control.

STRENGTHS AND WEAKNESSES:
**Strengths:**

1. The paper provides significant theoretical advancements, including a new coding scheme for RS that achieves near-optimal rates and extends prior discrete results to continuous settings. 

2. Synthetic experiments show ERS outperforms IML in low-distortion regimes, and MNIST confirms lower MSE with unbiased samples.

**Weaknesses:**

1. Only simple datasets are tested and no high-dimensional or real-world benchmarks are adopted. The scaliability of ERS is unclear. Moreover, comparision to SOTA compression methods is missed.

2. While the ERS matching lemma is new, core mechanisms, such as batching and Gumbel tricks, build heavily on prior IS/RS work.

QUESTIONS:
refer to Strengths And Weaknesses.

LIMITATIONS:
limitations and potential negative societal impact are not discussed.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new coding scheme for channel simulation based on standard Rejection Sampling (RS) and extend this analysis to Ensemble Rejection Sampling (ERS). They derive the coding cost for both methods, showing that they are first-order optimal. The paper then applies ERS to the problem of distributed matching, for which the authors derive a new bound on the matching probability, establishing a counterpart to the Poisson Matching Lemma (PML) and Importance Matching Lemma (IML). The practical utility of the proposed ERS-based method is verified in a distributed compression setting on both synthetic Gaussian sources and the MNIST image dataset.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- **New Rejection Sampling Channel Simulation:** The paper introduces a novel coding scheme for standard Rejection Sampling (RS) based on grouping and sorting proposals. It bypasses the suboptimality of naive runtime-based coding for RS, achieving a near-optimal rate of $I(X;Y) + O(\\log I(X;Y))$.\n- **New Ensemble Rejection Sampling Channel Simulation** The authors extend the RS coding technique to Ensemble Rejection Sampling (ERS) and also show it to be near-optimal.\n- **New Distributed Matching Lemma:** A major theoretical strength is the derivation of a distributed matching lemma for ERS. This result serves as a new tool for one-shot distributed information theory, alongside existing Poisson Matching Lemma (PML) and the Importance Matching Lemma (IML). Its existence expands the toolkit available for analyzing distributed compression and communication problems.\n- **ERS Distributed Compression:** Authors show how ERS matching can be used for distributed compression with decoder-only side information, mirroring IML scheme. Unlike in IML, which produces biased samples, the chosen sample $Y$ comes from exact distribution.\n\n## Weaknesses:\n\n- **Dependence on Global Bounding Constant:** A practical limitation of the proposed RS and ERS schemes is the requirement that both the encoder and decoder know the global worst-case density ratio $\\omega = \\sup_{x,y} \\frac{p_{Y|X}(y|x)}{q_Y(y)}$ beforehand. This contrasts with schemes like Poisson Functional Representation (PFR) or Greedy Rejection Sampling (GRS) which do not require such a global constant to be known to the decoder. For PFR and GRS only the local constant is needed $\\omega_x = \\sup_y \\frac{p_{Y|X}(y|x)}{q_Y(y)}$ and exclusively at the encoder.\n- **Computational Complexity:** The average computational complexity of the proposed methods scales with the global constant $\\omega$, rather than the instance-specific ratio $\\omega_x = \\sup_y \\frac{p_{Y|X}(y|x)}{q_Y(y)}$. This can lead to significant inefficiency. Consider the following example: $p(y|x=0)=Unif[0,1), p(y|x=1)=Unif[0,c), p(x=1)=c$; average PFR/GRS computation $<2$, while average RS and ERS computation $>1/(2c)$ can be arbitrarily large.\n- **Looser Rate Bounds:** While the coding rates for both RS and ERS are shown to be asymptotically optimal (i.e., achieving the leading term $I(X;Y)$), the upper bounds contain larger logarithmic and constant terms compared to the current channel simulation methods, such as those shown by PFR ($I(X;Y) + \\log(I(X;Y)+1) + 4$).\n\nQUESTIONS:\n1.  **Practicality of PML as a Baseline in the MNIST Experiment:** In the distributed image compression experiment, you argue that PML is inapplicable because the upper bound of the density ratio $p_{Y'|X'}(y'|x')/q_{Y'}(y')$ is unknown when learned by a neural network. However, this seems to warrant more discussion:\n\n    **1a.** PML, like PFR or GRS, only requires the *local* constant $\\omega_{B,x'} = \\sup_{y'} p_{Y'|X'}(y'|x')/q_{Y'}(y')$ at the decoder for a given side information $x'$. Since the decoder has access to the neural network representing these distributions (in this case a Gaussian), why can it not compute or tightly bound this local constant on the fly for each sample and use a PML-based scheme?\n\n    **1b.** Furthermore, your ERS scheme operates with a chosen group size, which is directly related to the density ratio. Could you clarify why the information implicit in the choice of group size cannot be used to parameterize a PML-based competitor for a direct comparison?\n\n2.  **Computational Cost of the Sorting-Based Coding:** The rejection sampling coding scheme has some similarities to PFR, where uniform random variables serve as arrival times (though it's not quite a Poisson process since $N$ is fixed). Can you think of a more computationally efficient method for coding the sample--ideally on the order of $\\omega_x$ ? Or must all the uniforms in the group be drawn every time? Finally, is there any downside, in terms of rate, to making groups larger than $\\omega$?\n\n\n### Minor Suggestions for Presentation:\n\n*   **Clarity of Indices:** The use of same letters for different indices (e.g., group index, batch index, local index) can be slightly confusing to follow.\n*   **Figure Readability:** The font size in the figures is smaller than the main text and can is difficult to read. Font size should be at least as large as the main text, ideally larger, to ensure clarity.\n*   **Context in RD Plots:** In Figure 5 (Right), adding a curve for the theoretical Wyner-Ziv rate-distortion bound would provide valuable context. It would help to assess the gap between all the practical methods and the theoretical limit.\n\nLIMITATIONS:\nYes, the authors briefly mention improving runtime efficiency as a future direction. However, the paper would benefit from a more direct discussion of its two key limitations and their practical context:\n1.  The computational complexity can be significantly higher than competing methods since it depends on the global density ratio bound not a local one.\n2.  The requirement for this global bound to be known by both parties is a stricter assumption than that of several existing channel simulation schemes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies problems of channel simulations and (distributed) matching probabilities. Although PFRL and PML-based coding schemes can achieve near-optimal coding rates for these problems, they are impractical as they require an infinite number of proposals (unbounded termination time). To address this issue, the authors have developed a new coding technique for RS (for distributed matching) and ERS (for both channel simulation and distributed matching), demonstrating that it can achieve near-optimal performance for the considered problems. The practicality of the proposed approach is demonstrated through experiments.\n\nSTRENGTHS AND WEAKNESSES:\nOn the strong points:\n\n-\tThe new coding technique and its analysis for RS are novel and interesting.\n\n-\tSimilarly, the analysis for ERS, particularly for distributed matching problem is involved and novel.\n\n-\tThe proofs appear correct and rigorous to me. (I have not gone through all the proofs; but even those over which I could not go in details I have glanced over them.)\n\nOn the weak points:\n\nFor details, please refer to the below `Questions' section.\n\n-\tThe benefits of ERS and the proposed coding schemes require further elaboration.\n\n-\tThe application of the proposed approach to machine learning problems is not discussed in detail and remains unclear. For a paper submitted to a top-tier ML conference I feel that this aspect should be largely improved. For the moment the paper appears to be more suitable for a conference such as ISIT or the Info. Theory Transactions.\n\n-\tThe boundedness of the ratio can be restrictive in practice; as one needs to ensure this by using a very large $\\omega$; which results in longer delays.\n\nQUESTIONS:\n-\tAs mentioned by the authors, the main pitfall of PMC-based approach is their termination time. However, when using RS or ERS in practice, it is also necessary to choose a maximum termination time.\n\nTo better explain this; consider Proposition 4.1. that establishes a bound that is independent of $\\omega$. This means that letting $\\omega \\to \\infty$ does not induce any (considerable) penalty for the coding rate. However, this comes at the cost of a larger $K$; hence more common randomness and larger delays. For example, even when the intended $K$ is found, in this scheme to send $\\hat{K}$, one must wait until end of the group (of size $\\lceil \\omega \\rceil$) in which $K$ is located, in order to sort them. Thus, in practice, for large values of $\\omega$, an upper limit must be set. The question then is what effect such a limit has on the quality of the simulated sample.\n\n-\tIt is unclear what benefit ERS has over RS for channel simulation problems. In other words, what benefit does the analysis offer for general $N$? As mentioned by authors, the analysis for $N=1$ and $N=\\infty$ is similar to that in previous works.\n\n-\tIn [39], the issue of unbounded termination time in PFRL for channel simulation is addressed using a simple Gumbel-max trick. Could a similar approach be used for PML-based coding schemes and the distributed matching problem?\n\n-\tThe definition of $\\mu_1'$ is deferred to Appendix F.5. Is it referred to as $\\mu_1^{cond}$ therein? If so; they do not only depend on $N$; but also on $\\omega$ and distributions. It would be better to at mention this.\n\n-\tAs he authors mentioned, the main contribution of the paper is the analysis of ERS for matching probability. However, if I understand correctly, it has not been used or verified in the experimental part. Moreover, I do not fully understand why, in the context of 'lossy compression', one would be interested in matching probability in Proposition 5.6. Instead, one should be interested in the incurred distortion, as considered in the experiments.\n\n-\tI am also concerned about the practicality of these approaches in real life, for example in image compression, where the distributions are unknown and the data is high-dimensional. In the experimental section of the paper, an example is given of a neural contrastive estimator being used to approximate the ratio. However, such approaches perform poorly in the high-dimensional regime. This limitation is not specific to this paper, but to similar papers as well. Do the authors have any suggestions for how such approaches could be used in real-life applications?\n \nSome of the typos: some mispelled words such as `Simmulation`, missing space at line 318, many places $\\ldot$ should be replaced by $\\cdot$, e.g. $\\mu(\\cdot)$ in line 323.\n\n=======================\n\nPOST REBUTTAL:\n\nThis is a rather interesting paper. There are still some minor concerns on the computability of the log density ratio in high dimension; but given the clear nature of this paper, which is theoretic, and the technical novelty, I vote for acceptance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework for channel simulation and distributed compression using Ensemble Rejection Sampling (ERS), a generalization of standard rejection sampling (RS). It addresses two central problems: сhannel simulation and distributed matching.\nKey contributions include:\n- A novel RS-based compression method achieving coding costs close to the mutual information bound.\n- A distributed matching lemma for ERS, extending prior work like the Poisson Matching Lemma (PML) and Importance Matching Lemma (IML).\n- Bounds and empirical validation on synthetic data (Gaussian sources) and MNIST image compression tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper is well-written and clearly outlines its contributions in a structured manner, supported by theory and experiments. Also I would emhasize the presence of theoretical justification and well-structured proofs.\n\nWeaknesses\n- While the theoretical complexity is discussed, practical runtime and memory overhead for ERS (especially in high-dimensional settings) could be elaborated further.\n- The experiments are limited to synthetic Gaussian sources and MNIST. Including more complex or real-world datasets (e.g., CIFAR, ImageNet) could strengthen the empirical claims.\n- Broader potential implications (e.g., in privacy-preserving communication or edge AI) could have been mentioned.\n- There are several typos, e.g. simmulation\n\nQUESTIONS:\n- Channel simulation. What is the main task? Communicate a sample $Y$ or generate it from $P_{Y|X}$?\n- Distributed matching. Please mention why the probability (2) is of the most interest. Am I right that $P_A$ and $P_B$ are discrete distributions? \n- How does the choice of batch size N in ERS affect practical runtime and memory usage in large-scale settings?\n- Can ERS be directly extended to domains beyond images, such as video or NLP, where the proposal and target distributions may be more complex?\n- How would ERS perform in scenarios requiring privacy-preserving sampling (e.g., differential privacy)?\n\nLIMITATIONS:\nThe experiments are limited to synthetic Gaussian sources and MNIST. Including more complex or real-world datasets (e.g., CIFAR, ImageNet) could strengthen the empirical claims.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}