Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper introduces an online MCMC framework for semi-supervised learning with variational autoencoders. The idea of the paper stems from physics problems, where one typically wants to infer certain parameters of a data generating process from empirical data. An example of this is quantitative MRI. 

The authors consider the model of data generating process known, and also assume a distribution mismatch between the simulated data, as well as the empirical data, which may include certain confounding factors that may be impossible to model. During the the training of a VAE, the authors progressively improve the psudolabels via a Metropolis-Hastings scheme. The key novelty of the paper is that the proposal distribution for the sampler is an posterior distribution $\hat p(\mathbf{y} | \mathbf{x})$, modeled by the VAE itself.

main_review: The most interesting part of this paper is a novel application of an existing machine learning result -- its adaptation to the MR physics problem. The release of the code also makes this work rather strong. The manuscript is well written. However, I believe that the experiments in the paper lack insights, rigor, and comparison to alternative methods.

Things to fix:

Equation (7) : please define what is $n$.

Line 177: use \eqref to refer to equation (14)


Experimental design issues:

-- Lack of baselines and proper comparisons. I believe that in order to truly prove the benefits of the proposed technique, one has to collect a sufficiently large dataset, and show that e.g. proposed methods for inference of quantitative MRI parameters boosts the metric of e.g. prediction of brain conditions. I would argue that acceptance rate of the sampler is not the most appropriate metric and substantially limits the level of experimental validation.

-- In the current context, the authors have only MCMC baseline, but what about machine learning methods? E.g. one off-the-shelf semi-supervised baseline could be to train a simple regressor of parameters in a semi-supervised way using e.g. Pi-model or other techniques. 

-- How is the method generalizing from being trained on the first scan to the second scan? 
- One very critical issue that I see with in paper, is the convergence of the sampler under different domain shift levels. One could test this in a simulator, and then analyze when the method actually stops converging. E.g. visualizing Wassestein distance vs. the level of distribution mismatch would definitely add important insights

- The penultimate issue I have with the experiments in the paper, is that the authors validated only one application: brain. qMRI is investigated in the knee imaging domain. I would propose the authors to strengthen this part as well. 

-- The final issue is that the method seems to work for any likelihood function. I would suggest the authors to evaluate it also on CIFAR10/100 classification benchmarks, and compare to classical VAE. The authors can conduct the distribution mismatch experiments there as well.

limitations_and_societal_impact: This work does not have any negative societal impact.

My main points for improvements are summarized in the main review. Here is the summary of limitations:

- Lack of SSL baselines
- Lack of clinically-relevant evaluation
- Generalization of the sampler is not investigated (from scan 1 to scan 2)
- Narrow domain in evaluation (the method seems to be general though)
- Lack of insights on distribution shift level
- Only one application: one can also conduct experiments on image classification, and inference of qMRI parameters from knee images.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper addresses semi-supervised learning problem. The authors propose to adopt Metropolis-Hastings Markov chain Monte Carlo (MH-MCMC) into the sampling process of posterior in conditional variational autoencoder (CVAE) learning.
Experimental results on MRI datasets demonstrate that the proposed method performs better than existing methods regarding the number of samplings.
main_review: ### Originality:
* This work proposes a novel combination of CVAE for semi-supervised learning and MH-MCMC.

### Quality:
* The problem setting this paper addresses is not clear.
    - The problem setting is finally described in the experiment section, so I cannot find whether it represents the scope of the proposed approach or the scope of the specific experiments conducted here.
    - In Section 2.3, it would be better to focus more on the machine learning perspective.

* The motivation to use MH-MCMC is not described in the main text.
    - The authors do not specify a clear reason/purpose to introduce this combination of CVAE for semi-supervised learning and MH-MCMC.
    - The specific combination would be novel, but there have been many combinations of other sampling methods including MCMC, and CVAE learning. Therefore, they should differentiate their approach from them.
    - In Section 2.1 (related work), it would be better to focus on comparing the proposed approach and existing works.

* In experiments, although the authors claim the efficiency of the proposed approach, they did not provide a comparison on actual computational time. Thus, it limits the significance of the analysis. (I acknowledge that they provided the analysis in terms of the number of sampling.)


### Clarity:
* The problem setting and the motivation of the proposed approach are unclear, as stated above.

* Minner comments.
    - What is the difference between $q$ in Eq.(1) and $\hat{p}$ in Eq.(2)? Both seem to represent approximate posterior.
    - In Eq.(10), $L_z$ is not defined.


### Significance:
* Unclear scope of this paper addresses degrades significance.
limitations_and_societal_impact: They addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Variational inference and distribution sampling are two popular methods for statistical inference. The present paper combines these two approaches in a semi-supervised learning setting, where the distribution of unlabeled data may differ from that of labeled data. Experimental results are provided for an MRI dataset, where the task is to infer the imaging/tissue parameters.
main_review: Variational inference and distribution sampling are both powerful approaches, and combining them is likely to be impactful. Here, the authors present a reasonable and novel strategy for semi-supervised learning.

- Overall, I find the method interesting, but experimental justification could be significantly expanded: (i) How would other semi-supervised methods, for instance the cited Kingma et al paper, perform on this task? (ii) Could the authors add 1-2 more tasks that are studied in previous semi-supervised learning papers?
- Somewhat related to the previous point, how would this method extend to scenarios with discrete labels?
- The authors already discuss sampling-related distributions of the proposed method. But, I think it would be valuable to put this in a better context: For some of the more popular applications in ML conferences such as image/audio analysis, language processing, recommendation systems, how applicable is this approach? It'd be ok if applicability is low for many tasks, but I believe such discussion would improve the paper.
- I find the review of previous work somewhat missing. A better survey of the semi-supervised learning literature would significantly improve the better. I think the current flow converges to CVAE very quickly.
- The presentation of the material is clear.

----------------------------- UPDATE -----------------------------

I like the main idea behind this paper. I also think that the improvement plan proposed by the authors sounds reasonable and interesting. However, considering that it is certainly possible for this set of experiments to reveal no significant practical improvements with respect to baselines, I decided to keep my score at its current level.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a scheme for parameter inference that builds on both simulated/labelled data and real/unlabelled data.
It uses MCMC with a CVAE proposal distribution to infer the data labels, and then trains the CVAE on the mixed data, bootstrapping the process.
They evaluate their method on an MRI task.
main_review: # Overall review

I really liked this paper and have tentatively recommended acceptance.
The main caveat in my recommendation is that I couldn't fully read Figure 3 - I have asked for clarification below.
Moreover, because I like what you've done so much, it feels like a shame that you haven't done more to explore the opportunities of the algorithm more generally.
I've left some thoughts on this below.
Whether or not the paper is accepted in its current form, I would encourage you to explore this a bit further in order to increase the impact and significance of the work (I think you have a little extra space in the paper that could be used for this).

# Major comments

Your main contribution seems to be a strategy for inferring parameters for generative models in a label-efficient way.
I like your overall strategy.
I am mindful of the fact that I think my research background is slightly different than yours (just inferring from the paper), so some of our assumed context may differ.
I've tried to account for this in my review, but we should be extra careful not to accidentally talk past each other, and if I seem to have misunderstood your points then please help me understand.

I don't have the background to know if your interpretation of MRI imaging as mixing simulated labelled data with unlabelled real data is novel.
If it is you should make this more explicit, and if not you should cite the source.

I cannot comment on the correctness of the work between lines 139 and 163.

Personally, I am most interested in the algorithmic contribution: the mixing of MCMC pseudo-data alongside simulated labelled data.
Here, it feels like there are some design options that could benefit from more investigation and discussion.
- What are the trade-offs in performing more than one MCMC step for each new label?
- What heuristics should be used to balance the gradient-based optimization of the proposal distribution/inferred parameters against the MCMC label inference?
- Are there advantages to 'warm-starting' the CVAE proposal in various ways?
- Why do you randomly select labelled or unlabelled points rather than using a steady ratio between them? What might the best ratio be? Does this change during training?

It seems as though the impact of this work is much greater if the method can be generalized to other settings and data-sets.
I think I'm sufficiently interested in the method already to tentatively recommend acceptance, but I think you could get much more out of this with more exploration of the algorithm itself.

Certainly, the paper would benefit from an algorithm box.
If nothing else, this might force you to enumerate the design assumptions that you make in implementing the algorithm, and give you an opportunity to discuss alternative choices.

Can you also say a little more about the hyperparameter choices made in S3.3? A number of these choices seem quite specific.
If you're just using something from a previous implementation that's absolutely fine, but it's nice to know what the origin of the choices is.

This comment is the most important for me: I cannot read Figure 3 and it seems the most important figure in the paper.
This is your opportunity to show that adding the unsupervised data improves your model, which is your main contribution.
I think this could be improved just by switching colors---unfortunately the line-shape makes it very hard to distinguish dashed lines from full ones.
I'm going to give benefit of the doubt for the moment that the results show the Metropolis-CVAE helping on at least a few of these.
If it is at all possible, please replace this figure with a more legible one during the review stage, as my provisional score is sensitive to this graph showing sufficient difference between the two methods.

If you can think of anything further that can be done to highlight performance differences between the CVAE and Metropolis-CVAE, these would be most likely to reaffirm my belief that this paper should be accepted.

# Minor remarks

The abstract could be much shorter.

Does the use of two different data sources for the MRI data complicate things?
How do you deal with the mismatch?
(Other than the masking - after all the voxels are different sizes etc.)

I usually don't work with VAEs, and I understand that truncating the Gaussian is more usual in that domain.
It does, however, mean that the KL divergence estimate in equation (18) is wrong because the approximate posterior which is sampled from is not a Gaussian.
I know most people ignore this, so I don't hold it against this paper, but it seems worth noting.
Can I just check also whether you truncate by resampling or by clamping values greater than 1 down to 1?
These are quite different distributions.

Computationally, checking each step whether you are using one loss or the other, then waiting for your CPU to return the pseudo-label, and then doing a backwards pass, is very bad (removes most of the advantage of GPU by blocking parallelization).
I think it's beyond the scope of this work, but it would be valuable to consider possible parallelization strategies or ways to pre-compute the pseudo-labels in batches.
limitations_and_societal_impact: These were adequately addressed.
review_title: null

Meta-review: This paper describes an algorithm reminiscent of the dynamics of an Monte-Carlo expectation-maximization (MCEM) algorithm where the E step is replaced by MCMC, here the classical MH algorithm. During the training of a VAE (similar to M step), the authors progressively improve the psudolabels via a Metropolis-Hastings scheme. (The derivation here seems to be pretty similar to and it might be informative if the authors would point out key differences of their approach.) One novelty of the paper is that the proposal distribution for the sampler is a posterior distribution modeled by the VAE itself. 

Overall, the reviewers seem to have liked the key idea but raise major concerns regarding the lack of sufficient experimental evaluation at this stage. I think this is a major concern and I encourage the authors to carefully address the reviewers extensive comments and submit to a future venue.


Decision: Reject