Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors discuss Bayesian pseudocoreset methods that uses reverse KL, KL and Wasserstein metrics special cases. They link these algorithms to data distillation as different posterior choices. Their main method is the Bayesian pseudocoreset algorithm that replaces reverse KL with a forward KL algorithm. They test these algorithms with Cifar-10 dataset with random coresets baseline.
strengths_and_weaknesses: Strengths:

Focusing/unifying on the Bayesian coresets and distillation methods is a good direction and influence for the academia. The paper might have a good potential to be proceeded with more future works.

Weaknesses:

The baseline is a random coreset which is a very weak baseline. However given that they apply their algorithm on an image dataset, this might be acceptable, and to be improved as a future work.

The paper needs to be organized and motivated better. The authors might more clearly mention the relations and related works for data distillation and bayesian coresets method, then unify them. There are some optimality requirements for these relations, it could be organized better. It would be good to emphasize more on the reasons that they were able to make the Bayesian algorithm tractable with a divergence perspective.

The ideas and proofs are too straightforward and the experiments are too preliminary. 

It would be good to do one more pass for proofreading.
questions: Have you compared your method with another baseline? At least another coreset algorithm a bit more intelligent than random, such as weighted summation or using input points distances? 


limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper unifies the Bayesian Pseudocoresets and existing dataset distillation methods in certain settings. It investigates multiple choice divergence measurements and proposes an algorithm using forward KL-divergence to construct Bayesian Pseudocoresets.
strengths_and_weaknesses: - ***Strengths***:
1. The direction of unifying existing data condensation and Bayesian Pseudocoresets is well-motivated.
2. The effectiveness of the proposed method is well illustrated.
	
	
- ***Problems***: 
1. Proposition 3.1 relies on two factors: (1) $\Vert \theta_{t} - \theta_{t-1}\Vert$ is sufficiently small so that the expansions in eq 24 and eq 25 hold; (2) the gaussian approximation. The second approximation might be commonly used in practice, yet this choice should be carefully elaborated on since it plays a key role throughout the paper. The first assumption actually limits the efficacy of the proposition to large t when \theta_t is close to local optima instead of the whole trajectory as in Data Condensation (eq 6). Then the connection between BPC-rKL and the existing Dataset Distillation method is not as exact as claimed. 
2. In section 3.2, the discussed connection between BPC-W and MTT accredits mostly to the simplified setting using the gaussian approximation. As the distance of two Gaussians is minimized when the variance is shared and the distance of the mean is minimized.
3. Theoretical-wise, the paper mainly leverages the gaussian approximation to unify the Bayesian Pseudocorset with data distillation methods. The strong reliance on the gaussian approximations without adequate elaboration downgrades its contribution.
4. In section 5, since BPC-rKL and BPC-W are not exactly equivalent to existing methods, the dataset distillation methods and the original Bayesian Pseudocoresets algorithm are expected as baselines while missing. Especially when the implementations of these algorithms are available.

***Minor Issues***:
1. Line 422, I think it should be $\approx$ rather than =.
2. Line 419, the expectation seems to be over $\pi_x$ rather than $\pi_u$.
questions: 1. The assumption of proposition 3.1 should be defined clearer that $\Vert \theta_{t} - \theta_{t-1}\Vert$ should be sufficiently small.
2. Results shown in figure 1 is not very illustrative due to their size. Could you pick some representatives and move the rest to the appendix? Could you demonstrate the differences quantitively rather than qualitatively? 
3. Could the performance gap between the using psuedocoreset and the entire dataset be indicated by the divergence?
4. Could you clarify the bias amplification on the downstream task mentioned in line 299? The Bayesian Pseudocoreset algorithm [Dionysis et al. 2020] claims weight w could potentially help with reducing the computation burden when nonzero entries are small. Is there any reference or empirical results substantiating the claim that m doesn't help with the performance?
limitations: The limitation is well discussed. The negative societal impact is not applied.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper reinterprets heuristic dataset distillation methods as approximate Bayesian pseudocoreset procedures, with different choices of divergence between the true posterior and the pseudocoreset approximation. Motivated by this analysis, they propose a new Bayesian coreset procedure based on an approximation to the forward KL divergence, and show empirically that it outperforms alternatives.
strengths_and_weaknesses: This paper offers an insightful and unifying new view on heuristic dataset distillation techniques, which leads to improved practical methods. It is original (to my knowledge) and clearly written, and a significant contribution both to the dataset distillation and the Bayesian coresets literatures.
It suffers from two main weaknesses in terms of quality. First, the theoretical analysis is extremely rough, involving severe approximations (essentially treating the posterior as a Gaussian with zero variance, i.e. a delta function). There is a rich literature on Gaussian approximations to the posterior, including in particular Bernstein von Mises theorems. Moreover, SGD can be viewed as providing a Gaussian approximation to the  Bayesian posterior (Mandt, Hoffman, Blei, Stochastic Gradient Descent as Approximate Bayesian Inference). It would be valuable to establish more general and rigorous results in terms of Gaussian posterior approximations, and then derive the heuristic approximations as limiting cases when the covariance goes to zero (i.e. when the number of datapoints goes to infinity). The second major weakness is the empirical analysis. Despite motivating the work by Bayesian inference, there is not a rigorous evaluation of the quality of the posterior approximation (e.g. is the uncertainty correct), only evaluation of accuracy and log likelihood. 
questions: Is there a general recipe for constructing Bayesian pseudocoresets using Gaussian approximations? In particular, can you enumerate the different choices of how the trajectory is managed, what divergence to pick, etc. Can you quantify the various sources of error in this approximation?

Small Suggestions:
Equation 1 has a typo, the comma in the middle (which makes it look as if the prior pi_0 is defined to be the posterior). Same typo appears in Eqn. 2

Equation 4: I believe u_m should not be bold, since it’s one datapoint, and the gradient on the RHS should be with respect to u_m not u, since the overall gradient should have dimensions the size of a single datapoint.

Line 67, the index should run to capital N not lowercase n. Might want to do the same in the contrastive divergence section, for consistent notation.

Line 89 - Typo: Traning -> Training

Line 180 - Check the grammar here and in the following lines (extends -> extend, etc.)

Line 193 - In fact, [16] and other Bayesian coreset methods demonstrate applications to real data (some real data is low dimensional, believe it or not).

Lines 215 - this paragraph is very unclear to me.

limitations: The authors are honest about some serious limitations of the method’s performance. I think it may be worth discussing privacy in the societal impacts section; pseudocoresets may potentially help increase data privacy while enabling downstream analysis on the one hand, while on the other hand their privacy properties have not be thoroughly studied
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies learning Bayesian pseduocoresets, which is the task of a small set of "data" by conditioning on which the posterior is close to the posterior conditioned on the full dataset.
It establishes the framework in which two important ingredients are (1) the choice of divergence measure and (2) the choice of variational approximation.
Following this, it first recasts/reinterprets two existing, successful dataset distillation methods are Bayesian pseduocorsets learning with reverse KL/Wasserstein distance with Gaussian approximations to the posterior.
Then it proposes to use the forward KL which in theory doesn't suffer form the mode-seeking behaviour of reverse KL.
On a few real-world datasets, the proposed choice of forward KL is better than existing choice of divergence measures in terms of NLL and accuracy.
strengths_and_weaknesses: Pros
- The paper studies an important problem of scaling Bayesian methods to large datasets by learning Bayesian pseudocoresets. This approach can benefit a wide range of Bayesian inference methods.
- The paper establishes the framework of this problem, which is novel, and studies the relation to existing dataset distillation methods. The paper is significant as it (1) gives justification of some existing heuristics-based methods and (2) paves the way of further research in this direction.
- The paper is clearly written with good quality.
- Related works are discussed adequately.

Cons
- The experiments might be improved a bit.
  - I'd like to see at least one synthetic dataset where the posterior is tractable and see how the posterior each method learns differs from the true one.
  - Some classic Bayesian inference datasets could be used so that naive HMC results could be provided as a reference.

Writing
- Equation (1) is broken: I assume the $\pi_0$ in the middle shall be removed.
questions: - There are some discussions regarding contrastive divergence in section 3.3. I wonder if the divergence from [1] is related there or maybe a good choice of divergence in this framework.

[1] Ruiz, Francisco, and Michalis Titsias. "A contrastive divergence for combining variational inference and mcmc." In International Conference on Machine Learning, pp. 5537-5545. PMLR, 2019.
limitations: Limitations and potential negative societal impact are addressed.
review_title: null

Meta-review: metareview: There was a consensus among reviewers that this paper should be accepted. The authors formulate dataset distillation methods as approximate Bayesian pseudocoreset procedures which appears to be a novel viewpoint. They further propose a new coreset procedure and show that it performs well. The paper appears to be well-written.
confidence: Certain
Decision: Accept