Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a complete latent likelihood (CoLLike), an alternative to the marginal likelihood (MaL) objective that is standard in ML algorithms (MaL) today. A systematic comparison between COLLike and Mal in the KL framework is compared. By discussing posterior collapse and high deviation of the aggregated posterior from the prior, authors show CoLLike can prevent these drawbacks. Empirical studies on image and NLL tasks show consistent improvement of CoLLIke over MaL. 
strength_and_weaknesses: Strength:
This paper address a fundamental problem in all machine learning training and it potentially has a big impact. 
Many connection and differences with existing objective are discussed - it was enlightening.

Weakness:
Some concern on the permutation estimation exist due to inefficiency. 


Comments:
- " target conditional is actually specified and equals $p_{\theta} (z|x)$ if we ask what distribution we want to mimic":  the exact form of $p_{\theta} (z|x)$ is still not specified, and it could be anything since it canceled out. If $p_{\theta} (z|x)$ is changed to any other distribution,  MaL  remains the same. This seems to contradict the statement. 
- " the CoLLike objective allows learning models with reverse factorization": ELBO has been used in VAE like structure, which resembles the function form. In addition, it would be good to provide examples on advantage or necessity of reverse factorization.
- some references are not cited correctly. e.g., " InfoMax principle Huszar (2017))".
- Eq 3: does the permutation mean all possible label assignments for all samples? How is it a permutation if the number of each class within Z are different? 
- "since the set of all possible π values is countable and finite, optimization can be performed by an exhaustive search." it seems for large sample size N, this is not truly feasible with exhaustive search. Even with Hungarian complexity O(N^3), it is still quite a big number, in particular in large datasets popular nowadays. 
- Hence, it would be interesting to see the time comparison between different objectives in the experiments. 
- " we emphasize our focus on learning useful z|x instead of simplifying the model with factorized x|z conditional" can authors elaborate on this point? what if x|z is not factorized?
- "To reduce the high variance of CoLLike unsupervised classification": authors should have discussed this point earlier of this work. 
clarity,_quality,_novelty_and_reproducibility: quality: fundamental contributions with sound justifications.

clarity: presentation is clear 

originality: novel 
summary_of_the_review: a fundamental and potentially impactful work to address the training objective in ML algorithms today.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors propose a new learning objective for fitting latent variable models. The basic idea is to sample a set of latent variables from the prior and find a permutation of those that aligns well with the observed data. The method provides a point estimate of the latent variables, but the paper is written from a perspective where it is suggested as an alternative to marginalising over the latent variables rather than focusing on the effect of the constraint imposed over standard joint likelihood. The approach is demonstrated primarily in context of supervised classification problems, where it is shown to find latent variables that align well with known classes.
strength_and_weaknesses: Strengths:
- The paper focuses on a fundamental learning problem and hence has potential for significant impact if the contribution is considered valuable by the community
- The core idea of constraining the possible values for the latent variables based on a sample from the prior is interesting
- The learning algorithm seems to work well, in the sense that we observe empirically the properties we should expect from the objective (higher MI, lower KL between the prior and the conditional etc)

Weaknesses:
- The objective is poorly motivated, using only high-level argumentation like posterior collapse in some specific latent-variable models as justification for limitations of marginal likelihood, and it relies on possibly questionable assumptions (sharp distribution for the latents). The main story focusing on its relationship with marginal likelihood is odd.
- The presentation is on a too high level. For instance, the main method is described only in terms of pseudo-code and the experiments are near impossible to parse from the main paper, for instance to figure out how the proposed method was used in these.
- The computational cost of operating with permutations is dismissed lightly; the authors mention it as a challenge and outline possible solutions on a high level, but do not transparently communicate what it means in practice. The experiments completely omit all discussion on the computational cost or even whether the proposed speedups were used.
- The notation and terminology are confusing and non-standard, making the paper difficult to read. For example, the phrases "unknown $z|x$ target conditional" or "target $z|x$ distribution" used in Introduction do not mean anything at that stage. Similarly, Section 3 is written in rather confusing terminology and makes standard concepts sound unjustified choices; the choice of conducting learning in a manner that is faithful to the modelling assumptions is called "keeping the model posterior unchanged" etc.
clarity,_quality,_novelty_and_reproducibility: The authors approach a fundamental question of what learning means, but unfortunately the paper is written in quite a confusing manner from a narrow perspective. I would assume a paper proposing a replacement for standard learning objectives to analyse the problem from a proper statistical perspective with rigorous mathematical treatment, rather than talking about vague concepts like "the target conditional". It seems that the motivation for the work comes from specific problems (posterior collapse etc) in specific types of latent variable models and the proposed objective indeed may be a good heuristic for addressing these specific aspects, but the paper suggests it would be a general solution for learning any latent variable models. From this perspective it is highly questionable and difficult to read.

Already the starting point is odd. The paper proposes a specific kind of a point estimate for $z$ as an alternative for marginal likelihood (that integrates over $z$), rather than relating it to standard point estimates that would be much the natural baseline for this. This discrepancy remains throughout the motivation, model description and experiments, where (unsurprisingly) metrics that favor point estimates are shown to be better. The paper would be easier to understand and follow if it focused on explaining the effect of the specific constraint for $z$ (namely that the set of all $z$ must be compatible with a sample drawn from the prior), rather than attempting to describe the objective as alternative to marginalising over the latent variables.

It is good that the authors clearly mention the main assumptions, but I have trouble accepting either one of them in general case. First of all, why should the "marginal of the target conditional in the latent domain" (sic) correspond to the prior $p(z)$? The standard practice in latent variable models is to use priors that are broad to allow for the data define the actual posterior, and models with no posterior concentration would be considered poorly defined. I understand the need to avoid extreme posterior collapse in flexible deep models, but nevertheless we should typically expect $p(z|x)$ to differ quite notably from $p(z)$. You later show empirically that you indeed reach $p(z|x)=p(z)$ which nicely shows the algorithm works well and that you can satisfy the assumption, but I would consider that as negative result from the perspective of learning a useful model for the data.

Similarly, I do not understand the assumption of confidently assigning a single $z$ to each $x$. Quite contrary, I would expect $p(z|x)$ to be broad (assign clearly non-zero probability for non-diminishing set of $z$) for many common use-cases for latent variable models. For clustering, we need probabilistic models exactly when the clusters overlap severely, in recommender engines we use latent variable models because for new users we know we cannot determine $z$ accurate before seeing enough data etc. In brief, the whole assumption seems to contradict the most common use cases for the model family. Even if I somehow misunderstood the statement or we narrow down to some specific sub-family of latent variable models where this assumption makes sense, the justification 'as in any real-world dataset' is nowhere near adequate. Finally, I do not see how 'assignment of a single $z$ to each $x$' would assure the second assumption. It certainly assigns one $z$ for each $x$, but does not say anything about the actual $p(z|x)$ in the sense that we could confidently tell for future $x$ what the latent variable is. Then again, the proposed approach does not even seem to define $p(z|x)$ outside the training instances.

The presentation of the algorithm is poor; you only provide a pseudo-code rather than explaining it mathematically. As you anyway provide code that likely includes similar parts, it would be better to describe the method properly in the main paper. Furthermore, the discussion of possible speedups remains on such a high level that we do not even know whether you have derived the details (and used these in the experiments) or whether these are just ideas for future work.

The empirical experiments are similarly vague. It seems you use 'MaL' to refer to a solution of some approximation (which seems to vary from case to case) and explain the methods on a very high level ("we use Glow-like normalising flow"). I know the details are provided in the Supplement, but the presentation in the main paper is still too simplified. As a practical example, your experiments do not say anything about the computational cost of the approach. You clearly mention in technical sections that the method has high cost because of the permutation and propose a few techniques for addressing it, but you do not even say whether those were used and you do not say anything about the actual cost. Now the reader has no way of knowing whether the MaL and CoLLike are approximately as fast, or whether one of them is 1000 times slower. For a method with known computational issues you absolutely need to say something about the cost in the empirical section.

The empirical results are good in the sense that they confirm the learning algorithm, but do not tell much about the specific formulation. Replacing integration over $z$ with point estimates indeed increases MI between the estimate and the ground truth, due to reduced uncertainty, and this is what you see. You would most likely get a similar result when comparing the solution of k-means and the posterior of a probabilistic mixture, but it would not imply the former is a better objective. 

Questions and minor remarks:
- What is the exact relationship to EM? It sounds like the most closely related method, rather than MaL, yet it is only briefly mentioned. Should there perhaps be some comparison as well? Even EM is 'more advanced' than your method as it approximately integrates over $z$ rather than finding a point estimate, but still a better match for comparisons.
- Does your learning algorithm propagate gradient information through LAP? Not easy to figure out from the pseudocode as it may depend on how LAP is implemented. 
- It sounds like your approach works in 'transductive' setting, in the sense that you assume all $x$ to be available at training time. Can you still estimate $z$ for new samples in an inductive manner? If yes, how? This should be discussed.
- How would the model work in a case where the true $p(z|x)$ are very broad (e.g. a clustering case where the clusters are almost -- or perhaps completely -- overlapping)? Would it still find a good objective value? I strongly feel that illustrating the model in some simple examples, in addition to the main goal of flexible latent-variable models, would help clarifying the key concepts. For instance, a simple mixture model case would show the readers how the optimal solution is similar to what k-means would produce, in the sense that it is a hard clustering where the global optimum is one that assigns each sample to the closest cluster (unless I'm mistaken). An example like this would show what the effect of sampling the set of $z$ from the prior is in practice.
- How would the method work if the prior is way off? For instance, if you assume Dirichlet prior in clustering with constant cluster sizes but the data strongly suggests some clusters have ten times more samples?
- Constrained K-means does have a probabilistic interpretation; see e.g. Jitta and Klami "On controlling the size of clusters in probabilistic clustering" (AAAI, 2018).

summary_of_the_review: A bold attempt of re-defining the fundamental learning objective of latent variable models, but there are too many open questions regarding the theoretical foundations and also the empirical side. The biggest issue is that the approach is compared against marginal likelihood when it actually corresponds to a specific way of constraining a point estimate for the complete likelihood, and because of this mis-positioning the paper actually avoids discussing what the effect of sampling the set of possible latent variables from the prior is in practical cases or even in easy examples. From the perspective of general latent variable models the assumptions made for deriving the new objective are either shaky or even wrong, but it may be that for specific types of models they provide reasonable inductive biases. However, for that the paper would need to be re-written from a very different perspective.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Complete Latent Likelihood (CoLLike) which utilizes permutation for matching in the latent domain. The authors argues that permuting the latents and then matching to inputs leads to expressive models with exact likelihood and no posterior collapsing, compared to MaL. Also, the suggested CoLLike can link likelihood and optimal transport.

strength_and_weaknesses: The experimental results show the efficacy of the suggested CoLLike.

However, it seems that the computational complexity of O(N^3) can be problematic, where N indicates the number of data instances.

clarity,_quality,_novelty_and_reproducibility: The authors provided their code as supplementary material, and the quality and the novelty are fair.

For the clarity of the paper, here are some questions to the authors.

- In the beginning of Section 2, the authors should mention that x is an input and y is a label to be predicted, even though it is convention. Or, am I wrong?

- I wonder why the authors represent the latent variable z as “label”. Does the latent z is limited to some attributes, for example? Can’t it be in any shape? It would be better to provide a graphical notation of x, y, and z, together.

- Sampling from the prior ensures the first requirement, … -> probably (x_N, y_{pi(N)}) ?

- Figure 1 is very hard to follow. What are the circles indicates? Where is permutation pi in the figure?

- The python-like pseudo-code should be written with more consistency.

- The complexity of CoLLike seems to be problem. Even though the authors utilized the combinatorial linear sum assignment problem with complexity of O(N^3), the problem is that N is the number of data instances which can be a huge number. Could you compare the wall-clock runtime of CoLLike against the baselines? Also, I’m curious about the effect of additional techniques: minibatch assignment, low-dimensional discrete latents, and factorized conditional. How much do they reduce the wall clock runtime in the experiments?

- Also, how exact is the combinatorial linear sum assignment problem? Does it always solvable?

- I suggest bring the objective of CoLLike and the combinatorial linear sum assignment problem to the main paper. Those should be treated in the main body.

- It seems that there are no direct comparison between CoLLike and ELBO in the experiment section. Does it due to that the ELBO does not have tractable likelihood? Can't it be approximated by Monte Carlo estimation, for example, in [1,2]?

- Could you provide more information about the experiment setting in Section 6.2? 

- In data generation perspective, how ones can generate data from the latent prior p(z)? Could you explain the sample generation process?

- I suggest authors to have proof-reading from the natives.

[1] Stick-Breaking Variational Autoencoders, https://openreview.net/pdf?id=S1jmAotxg 

[2] Dirichlet Variational Autoencoder, https://www.sciencedirect.com/science/article/pii/S0031320320303174


summary_of_the_review: The paper is fairly-written in general. However, the authors are missing a critical point, which is an analysis on the computation complexity. Following from the authors, the provided algorithm has O(N^3) complexity, where N indicates the number of data instances. My major concern for this paper is that the utility of the proposed method may vary with computational time. Also, the clarity of the paper needs be improved. Therefore, at this moment, I'm leaning to the negative side on this paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The submission proposes a Complete Latent Likelihood (CoLLike) objective, which is given as (3), as an alternative to commonly used log-likelihood objectives (which the authors call Marginalized Likelihood, MaL) for learning latent variable models. In essence, when the prior over latent variables are known or given, the proposed method generates i.i.d. latent variables from the prior, and find a permutation between the generated latent variables and observed samples. 

The motivation comes from scenarios where computing a marginalized probability by integrating over latent variables can be intractable, typically required in EM style algorithms. Instead, by finding the optimal pairing of observed samples and (generated and fixed) latent variables, which can basically be reduced to a bipartite matching (or assignment) problem, authors could avoid the intractable integration step. 

Experiments are performed on datasets with relatively small number of latent classes, and compare the performance of CoLLike and MaL. 
strength_and_weaknesses: 
There are a number of controversial heuristics unjustified in my view. 

1. One basic principle of the MLE is its (asymptotic) consistency (which is why in the first place people want to do ML estimation). What can be said about the proposed algorithm? Is it consistent at least? 

2. The proposed algorithm is not a strict generalization of K-means, since empirical distribution of true labels does not necessarily match the true prior. Impact of such sampling errors should be considered (e.g., when clusters are very well-separated). 

3. For AG News data, the huge gap in Accuracy is suspicious. More thorough and detailed discussion on this phenomenon (on why it happens) should follow. 



Other weaknesses

- Discussions in Section 5 are reasonable, but they are sort of trivial and do not add much value to the paper in my opinion. 

- What can be done if the prior of latent variables are not known in advance? 

- In the experiments, it is not clear what NLL / Agg. KL stand for (and also Accuracy is not precisely defined).

- It is unusual to use $-\log$ as a distance measure for OT. So why this should be better than using a typical Euclidean distance measure?

clarity,_quality,_novelty_and_reproducibility: While the claim of the paper is clear, the proposed algorithm is a naive variant of MLE, and is not supported by convincing reasons. 
summary_of_the_review: I do not think the paper contains enough contribution or originality to be published.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a Complete Latent Likelihood (CoLLike) objective, as an alternative to commonly used marginalized likelihood objectives for learning latent variable models. The basic idea is that when the prior over latent variables are known or given, the proposed method generates i.i.d. latent variables from the prior, and find a permutation between the generated latent variables and observed samples. The tackled problem is fundamental, important, and interesting, and the empirical results illustrate the efficacy of the proposal. However, as reviewers pointed out, several issues are unclear (e.g., what if the prior is not known in advance?), the presentation is to be improved, and the authors may consider including an analysis of the computation complexity of the proposed matching procedure.
justification_for_why_not_higher_score: As reviewers pointed out, severaAl issues are unclear (e.g., what if the prior is not known in advance?), the presentation is to be improved, and the authors may consider including an analysis of the computation complexity of the proposed matching procedure.
justification_for_why_not_lower_score: The tackled problem is fundamental, important, and interesting, and the empirical results illustrate the efficacy of the proposal.
Decision: Reject