Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A view-consistent loss is designed, which is based on PGM. The PGM is quite simple to understand.
main_review: 1. I want to know how many branches is good and why don't you use multiple heads? e.g., GAT uses multi-head.
2. In multiview learning, I want to know how to model diversity? The consistency is easy to model, but the diversity is not well addressed. Even CCA can be regarded as a view-consistency modeling.
3. I want to know why you design the model? for what applications?
summary_of_the_review: The view-consistency modeling does not give a supervise or somehow it is not new enough. 
The diversity loss is not well addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: the submission proposed a semi-supervised learning framework that leverages the benefits of multi-view learning with neural networks. On labelled data, besides minimising the empirical risk, the objective function encourages diversity by driving the correlation between individual functions with the mean function; on the unlabelled data, the objective forces multiple views to produce consistent features or predictions. 
main_review: 1. The ablation study missed an interesting question:  would increasing the number of views with either consistency or diversity loss surpass the combined loss with fewer views?

On one hand, increasing the number of views certainly increases the chance of producing more diverse features or predictions, but on the other hand, having more views on the same data also leads to a higher probability of consistent predictions from multiple views. It would be good to see how the number of views plays in the game.

2. The fundamental grounding of multi-view learning is that, the utility of all views is maximised if, given the label, all views are independent of each other. 

The paper made an argument in the intro that negative correlations between or among individual views would be helpful, which doesn't make sense. Referred literature in this paper talked about decreasing correlations by including a penalty term, but they didn't mean that we would like to see negative correlations.

3. In ensemble learning, at least in bagging, the improvement over individual learners is often more significant when these learners are more diverse (have larger variance), but again, the baseline model here is a trained base learner. It is highly likely that an ensemble of model A would still be worse than a single model B. There is always a tradeoff between the diversity (variance) of the base learner and its accuracy. I hope the authors could be more precise on their wording in the intro. 

4. I am not super positive about the derivations in Section 3. Let me elaborate a bit.

Providing that $$f_m \sim \mathcal{N}(f_c, \sigma_m^2), \forall m=1,...,M$$ with the distribution of f_c unknown.

The joint distribution should be $$p(f_c, f_1, ..., f_M)=p(f_1, ..., f_M|f_c)p(f_c)=p(f_c)\Pi_{m=1}^Mp(f_m|f_c)=p(f_c)\Pi_{m=1}^M\mathcal{N}(f_c, \sigma_n^2)$$, which is different from Eq. 2.

The marginal distribution should be $$p( f_1, ..., f_M)=\int p(f_1, ..., f_M|f_c)p(f_c) df_c=\int p(f_c)\Pi_{m=1}^M\mathcal{N}(f_c, \sigma_m^2)df_c$$, which doesn't give us Eq. 3 since we don't know  the distribution of f_c.

Let's move on to the posterior distribution of the consensus function: 

$$p(f_c | f_1, …, f_M) = p(f_c, f_1, …, f_M) / p(f_1, …, f_M)= p(f_c) p(f_1, …, f_M | f_c) / \int p(f_c) p(f_1, …, f_M | f_c) df_c$$

which may not be a gaussian distribution. 

A quick fix is to also assume that p(f_c) is a gaussian distribution, although this assumption seems to be very strong. 
summary_of_the_review: Overall, this submission's empirical contribution is more appreciated than its unclear theoretical justification. However, the arguments made in the intro were too vague and also misleading. Therefore, I do not vote for accepting this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper concerns the semi-supervised problem. Different from SOTA deep semi-supervised methods, the proposed DiCom employs a diversity measure on the labeled multi-view data, and combines diversity with consistency based on underlying probabilistic graphical assumptions. Experiments verify the effectiveness.
main_review: 1.	The consideration of diversity for semi-supervised learning is interesting, and the reviewer considers that it is helpful for improving performance.
2.	The proposed method combines diversity with consistency based on underlying probabilistic graphical assumptions is a meaningful idea.

weakness
1.	The idea of this paper is direct. This paper still considers traditional semi-supervised learning, and the multi-view refers to the view generated from instance augmentation. Why only measure the diversity of labeled instances and not extend to unlabeled instances? Are there relevant experimental verifications?
2.	What is the advantage of combining diversity with consistency based on underlying probabilistic graphical assumptions? Several multi-view methods have already considered the diversity with consistency simultaneously, but the authors choose to ignore them, e.g., “Exclusivity-Consistency Regularized Multi-view Subspace Clustering”, “End-to-End Adversarial-Attention Network for Multi-Modal Clustering”. More analyses and comparisons are expected, the reviewer considers that these ideas can also be transformed into the semi-supervised framework.
3.	Although this paper considers semi-supervised regression, why can't the semi-supervised classification method extend to the regression problem by replacing the loss function? The SOTA comparison method is in 2018, more comparison methods recently are expected.


summary_of_the_review: This paper concerns the semi-supervised problem. Different from SOTA deep semi-supervised methods, the proposed DiCom employs a diversity measure on the labeled multi-view data, and combines diversity with consistency based on underlying probabilistic graphical assumptions. Experiments verify the effectiveness. However, there exist several shortcomings for this paper, so I recommend weak rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a method for **semi-supervised multi-view regression** based on **data augmentation** and an **undirected graphical model**.
The author derived a "**consistency**" term for unlabeled data and a "**diversity**" term for labeled data based on their log-likelihood and combined them linearly with two hyperparameters.
The proposed method was evaluated on tabular and image data and analyzed via ablation study.
main_review: ## Strengths

This paper studied the semi-supervised regression problem and tackled it with a technically sound approach.
It sheds light on the effects of diversity and consistency in multi-view learning.
Besides, this paper is mostly clear written.
It is easy to follow the derivation and verify its correctness.

---

## Weaknesses

My biggest concern is its novelty.
As cited in the paper, most of the probabilistic graphical model definition, marginalization, and likelihood derivation is quite similar to those used in [Yu et al., 2011].
Although [Yu et al., 2011] used the Gaussian process while this work mainly used neural network and gradient-based optimization, the techniques for deriving the learning objective are basically the same.

Concretely, if I didn't misunderstand anything, the definition of the undirected graphical model is roughly as follows:

- $p(y, f_c, f_1, \dots, f_M, x) = p(y, f_c) \prod_{m=1}^M p(f_c, f_m) p(f_m, x)$

- $p(f_m, x) \propto F_m(\eta_m(x), \theta_m)$ (multi-view neural networks)

- $p(f_c, f_m) \propto \exp\left(-\frac{(f_c - f_m)^2}{2\sigma^2_m}\right)$ (consensus potential, "within-view potential" $p(f_m)$ in [Yu et al., 2011] does not appear here)

- $p(y, f_c) \propto \exp\left(-\frac{(y - f_c)^2}{2\sigma^2}\right)$ (output potential, Gaussian noise model for regression)

Then, both [Yu et al., 2011] and this work marginalize the potential function to obtain:

- $p(f_1, \dots, f_M) \propto \int \prod_{m=1}^M p(f_c, f_m) \,\mathrm{d} f_c$ (similar to [Yu et al., 2011] Eq. (7) in Section 3.1 (Marginal 1) and Appendix A.1 (derivation) without the within-view potential)

And the conditional probability is calculated via Bayes' rule:

- $p(f_c | f_1, \dots, f_M) = \frac{p(f_c, f_1, \dots, f_M)}{p(f_1, \dots, f_M)} \propto \exp\left(-\sum_{m=1}^M \frac{(f_c - f_m)^2}{2\sigma^2_m}  + \sum_{m<k} \lambda_{m,k} (f_m - f_k)^2\right)$

It seems that [Yu et al., 2011] can be used for semi-supervised multi-view regression as well as classification.
I'd like to know the main challenge and unique contribution of this work compared with [Yu et al., 2011].

---

## Questions and comments

### Regression

- I might have missed something, but is there anything "regression-specific" in this work?
  If we just change the form of likelihood, can we apply the same technique to classification as well?

### Data augmentation

- How do you perform data augmentation on tabular data?
- How do you perform data augmentation after the backbone network in DiCoM-B?
- The data augmentation seems non-differentiable, so the backbone is pre-trained and fixed?

### Graphical model notation

- Figure 2 does not seem to be standard probabilistic graphical model notation so it's a little confusing.
  For example, edges should not be labeled and $\sigma^2$ and $\lambda$ should be fixed parameters.

### Others

- Eq. (1): Shouldn't it be $\sigma^2_m$? The assumption of $\sigma^2_m = \sigma^2_v$ is not given here.
- Eq. (2): $p(f_m, f_c)$ is not defined nor explained.

---

## References

- Yu, Shipeng, et al. "Bayesian co-training." The Journal of Machine Learning Research 12 (2011): 2649-2680.
summary_of_the_review: This paper proposed a technically sound method based on an undirected graphical model for semi-supervised regression.
However, compared to an existing formulation [Yu et al., 2011], its novelty and contribution are a little questionable and need to be further highlighted.
Hence, I think it's on the borderline for now.
review_title: null

Meta-review: The work proposed multi-view learning framework that combines diversity and consistency objectives for semi-supervised learning. While reviewers appreciated that simplicity of the proposed method, they raised concerns on the limited contribution on top of the original Bayesian Co-Training work. Although authors provided detailed rebuttals that addressed some of the reviewers' concerns, and one reviewer did raise their score, the other reviewers' scores remained unchanged. Given the work is closely based off the BCT work, I would like to see more detailed analyses on the importance of the changes brought in this work, such as changing the base learners and introduction of diversity objectives as pointed out by the authors.
Decision: Reject