Diverse and Consistent Multi-view Networks for Semi-supervised Regression
Authors: Cuong Manh Nguyen
Le Zhang
Arun Raja
Xun Xu
Balagopal Unnikrishnan
Kangkang Lu
Chuan-Sheng Foo
Venue: ICLR 2022 Submitted (2022)

Abstract: Label collection is costly in many applications, which poses the need for label-efficient learning. In this work, we present Diverse and Consistent Multi-view Networks (DiCoM) — a novel semi-supervised regression technique based on a multi-view learning framework. DiCoM combines diversity with consistency — two seemingly opposing yet complementary principles of multi-view learning - based on underlying probabilistic graphical assumptions. Given multiple deep views of the same input, DiCoM encourages a negative correlation among the views' predictions on labeled data, while simultaneously enforces their agreement on unlabeled data. DiCoM can utilize either multi-network or multi-branch architectures to make a trade-off between computational cost and modeling performance. Under realistic evaluation setups, DiCoM outperforms competing methods on tabular and image data. Our ablation studies confirm the importance of having both consistency and diversity.

Under review as a conference paper at ICLR 2022
DIVERSE AND CONSISTENT MULTI-VIEW NETWORKS FOR SEMI-SUPERVISED REGRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
Label collection is costly in many applications, which poses the need for label- efficient learning. In this work, we present Diverse and Consistent Multi-view Networks (DiCoM) novel semi-supervised regression technique based a multi-view learning framework. DiCoM combines diversity with consistency two seemingly opposing yet complementary principles of multi-view learning: based on underlying probabilistic graphical assumptions. Given multiple deep views of the same input, DiCoM encourages a negative correlation among the views' predictions on labeled data, while simultaneously enforces their agreement on unlabeled data. DiCoM can utilize either multi-network or multi-branch ar- chitectures to make trade-off between computational cost and modeling perfor- mance. Under realistic evaluation setups, DiCoM outperforms competing meth- ods on tabular and image data. Our ablation studies confirm the importance of having both consistency and diversity.
INTRODUCTION
Deep neural networks have achieved tremendous success across several domains, ranging from com- puter vision, natural language processing, to audio analysis (LeCun et al. 2015). However, to train neural networks that perform well typically requires large amount of labeled data. many cases, this requirement for large labeled dataset presents a challenge, because the annotation process can be
and thus expensive, especially when specialized expertise required. To address this challenge, semi-supervised learning methods (Van Engelen & Hoos 2020) that can achieve sim- ilarly high performance with less labeled data by using unlabeled data have been developed.
We focus on semi-supervised learning in the regression setting. There are several approaches for semi-supervised regression, including graph-based methods (Zhur & Ghahramanirh, 2002), co- training (Blum & Mitchell 1998) and entropy minimization (Jean et al. 2018). Consistency-based approaches that have been popular in the classification setting. such as Mean Teacher (Tarvainen & Valpola| 2017) and Virtual Adversarial Training (Miyato et al.) (018),, which reinforce the out- put consistency of the network under input perturbations, have also been adapted to the regression setting (Jean al., 2018). However, enforcing consistency alone may not be sufficient for good performance, and may lead to model collapsing (Qiao et al. 2018) or confirmation bias issues (Ke) al. 2019).
To address these issues, we draw inspiration from ensemble learning with neural networks for re- gression. A necessary and sufficient condition for an ensemble of learners to more accurate than any of its individual members is if the base learners are accurate and diverse (Dietterich) 2000). Therefore, the key component that can make break an ensemble is the diversity (or disagreement) among its individual regressors. If this diversity is insufficient, the ensembling may not result in better performance. On the other hand, overemphasizing diversity can degrade the learnability of the ensemble members. So far, the most successful mechanism to leverage ensemble diversity in regression Negative Correlation Learning (Liu & Yao 1999 Zhang et al. 2019).
In this work, we propose Diverse and Consistent Multi-view Networks for Semi-supervised Regres- sion (DiCoM) that elegantly unifies consistency and diversity multi-view learning framework. Based on probabilistic graphical assumptions, derive loss function that integrates both con- sistency and diversity components diversity is encouraged on labeled data, while consistency is enforced on unlabeled data. Furthermore, develop two variants of DiCoM: the first uses multiple

---

Under review as a conference paper at ICLR 2022
networks to achieve better performance, while the second employs single network with multiple branches to help with scalability. We compare DiCoM against state-of-the methods on eight tabular datasets and crowd-counting dataset, where show that DiCoM outperforms existing methods. We further perform ablation studies to analyze the importance of diversity and consis- tency, and the effect of varying the number of views in the model.
While other works have leveraged related ideas of complementary and consensus in multi-view classification (Xu et al. 2013); or explored commonality and individuality in multi-modal curricu- lum learning (Gong) 2017), these methods were developed for classification or clustering tasks, and cannot be easily modified suit -upi-supeeeieed regression.
In summary, the major contributions of this work are as follows:
We derive novel objective function from a probabilistic graphical perspective. Our objec- tive function unifies multi-view diversity and consistency and provides theoretical insights into the relationship between diversity-consistnnyy trade-off and the number of views.
We show the high flexibility of DiCoM, which can be adaptively scaled up to larger number of views while maintaining competitive performance.
We demonstrate the performance of DiCoM on both tabular and visual types of input data, where it outperforms competing methods. Our ablation studies validate the importance of having both diversity and consistency.
2
RELATED WORK
Semi-supervised Regression: Semi-supervised learning is a data-efficient learning paradigm that offers the ability to learn from unlabeled data. In recent years, much work has focused semi- supervised classification, and there have been far fewer studies on semi-supervised regression. For regression tasks, graph-based methods are among the first to be developed. One example is Label Propagation (LP) (Zhur & Ghahramanirh 2002) which defines graph of training data and prop- agates ground-truth labels through high density regions of the graph. Kernel methods have also been proposed, such as Semi-supervised Deep Kernel Learning (SSDKL) (Jean et al., 2018). 1)... This method minimizes the predictive variance in posterior regularization framework learn a more generalizable feature embedding on unlabeled data. Co-training regressors (COREG) (Zhou & Li, 2005) employs k-Nearest neighbor regressors, each of which generate pseudo-labels for the other during training: this helps to maximize their agreement on unlabeled data.
Apart from the aforementioned approaches, consistency-batt based methods are also gaining traction. Mean Teacher (MT) (Tarvainen & Valpola| 2017) enforces posterior consistency between two neural networks, a student and a teacher, the latter being an exponential moving average of the former in the parameter space. An orthogonal approach is to enforce consistency adversarially augmented input, as implemented in Virtual Adversarial Training (VAT) (Miyato et al.] 2018). These methods were originally developed for classification, and were subsequently adapted to regression tasks (Jean) let al. 2018). However, both MT and VAT maintain only a single trainable network, which may lead to problems such as confirmation bias (Ke et al. (201)) and overly-sensitive hyperparameters. In this paper, we show that consistency-based methods can be further improved with ensemble diversity.
Ensemble Diversity: Ensembles of neural networks have been extensively studied and widely used in many applications. Their effectiveness largely depends on the level of diversity (or disagreement) among members of the ensemble. It is well-understood that good ensemble must manage the trade- off between the accuracy of the individual learners and the diversity among them (Brown et al.) 2005] Tang et al. 2006). For regression tasks, a commonly- ensemble technique Negative Correla- tion Learning (NCL) (Liu & Yao 1999: Liu et al. 2000), which formulates a diversity-promodde loss using an ambiguity decomposition of the squared ensemble loss (Krogh et al. 1995). In this for- mulation, a correlation penalty term (also refered to an ambiguity term) measures how much each member's s prediction deviates from the ensemble output. When this penalty term maximized, the errors of individual learners become negatively correlated. It was theoretically proven (Brown al., 2005) that the strategy employed by NCL is equivalent to leveraging a waarianc--oovaiinae trade-off' (Ueda & Nakano, 1996) of the ensemble error.
2

---

Under review as a conference paper at ICLR 2022
Recently, NCL has been extended to semi- semi-supervised learning (Chen et al. 2018 where the cor- relation penalty term is extended to the unlabeled data. However, this method was demonstrated only tabular data. Another variant of NCL is Deep Negative Correlation Learning (DNCL) (Shi et al. 2018 /Zhang et al. 2019). which is designed for visual regression tasks in a purely supervised learning setting.
Multi-view Learning: A dataset is considered as having multiple views when its data samples are represented by more than one feature set, each of which is sufficient for the learning task. Although each view is supposed to be sufficient for learning the task, a model trained on only one single view often faces the risk of overfitting, especially when labeled data is limited (Xu et al. 2013). To address this problem, multi-view learning assigns a modeling function to each view and jointly optimize these functions to improve overall generalization performance (Zhao et al. 2017). By ana- lyzing the development of various techniques, Xu et al. (Xu et al. 20133 summarized two significant principles that underpin multi-view learning: consensus and complementary. The consensus prin- ciple states that multi-view technique must aim maximizing the agreement on different views. This is similar to how consistency-based semi-supervised learning methods works: for instance, MT enforces agreement with its past self. The complementary principle states that in order to make improvement, each view must contain some information that the other views do not carry. In other words, the views should be sufficiently diverse. This is related to diversity regularization in ensem- ble learning, where individual learners are encouraged to give diverse predictions. Thus, multi-view learning offers a unifying perspective of both consistency and diversity.
PROPOSED METHOD
We start by describing how multiple deep views can be generated from input data. Then, we pro- pose our multi-view learning framework for regression, in which multiple deep views can be si- multaneously optimized via backpropagation. We then discuss the graphical models that govern the probabilistic dependencies among the ground-truth label and the deep views. Finally, we derive the DiCoM loss function using these graphical models and provide few insights.
Consider a regression task where the goal is to estimate a label y E R from an in- put œ. To create multiple views, our first approach to use M neural networks F2.. FM. each parameterized by 0,, 0z...................... 0M. respectively. By applying different data augmentations Vh, 1/2,- ................................ .7M on the original x, we generate M different augmented inputs : = 7m (x) V m 1,..., M.. With each augmented input, the corresponding neural network produces regression output f=) Fm (x"". .m)) V m 1....,,,, M. Due to the different augmentations and network parameters, each output f'm can be treated as one deep view of the original input We call this multi-network setup DiCoM-N, where the stands for 'network' (see Fig. (a)).
View creation:
The second approach is to utilize a single network with a shared backbone B and multiple branches F,, F2,...,,FF.
We use 0g to denote the learnable parameters from the backbone and 0,, .2....,, 00 to denote the parameters of the branches. The hidden features generated by the back- bone serve as input to the branches. While the backbone still applies a random augmentation to the input x, each branch F.. applies its own random augmentation 7mm well. Thus, regression outputs fi, f2...... fM from the branches can be considered as deep views of the original input. We name this setup DiCoM-B, where the B is short for 'branch' (see Fig. (b)). Multi-branch technique was widely adopted for supervised classification (Xie et al. 2017). In this work, it allows us to harness the power of multi-view learning with a relatively lower number of trainable parameters.
Multi-view learning framework for regression: Regardless of how they are generated, the deep views are used together with the true label y to compute a semi-supervised loss function EDICOM. During training phase, DDICOM is phooaprrpegated simultaneously through the deep views to opti- mize network parameters 01, .2.....8, Au (including 0g in the case of DiCoM-B). During inference, all augmentations are removed so that the forward pass is applied on the raw input x. The final prediction is computed the average of all deep views: (xx)) ΣM 1 (x). The general Di- CoM framework is illustrated in Fig. In the next step, we derive EDiCOM based on a probabilistic graphical assumption.
Probabilistic graphical models: Since the augmented inputs are generated from the same sample, the deep views should be close each other. Motivated by previous work in kernel learning (Yul
3

---

Under review as a conference paper at ICLR 2022
11
Tmm
1]M
1/mm
FM
m
FM
f
FT1
FM
SpcCoM
PICOM
(8) DiCoM-N
(b) DiCoM-B
Figure 1: The DiCoM framework with two variants: (a) DiCoM-N (multi-network) and (b) DiCoM- (multi-branch). Solid black arrows denote input/output; double black arrows denote augmenta- tions; solid blue and dotted red arrows represent forward and backward passes, respectively.
et al. 2011) and linear regression (Nguyen et al. 2019). consider fi, f....... fas as random variables and introduce a consensus function fe as a latent variable that connects to each the deep views. This function enforces the mutual agreement among the views. We assume that the difference between the consensus function and each view follows a zero-mean Gaussian distribution
Sa fm x (0, (m))
Vm 1,.., M.
(1)
f
fm
fR
:
fe
fm
:
fM
fi
fM
(ʃ. y)²
w(G-y) exp
2a
(fe f
((fo exp
v(for = exp( -m (m - fo
(// - n)*
2n
Pff. exp
20/
(a)
(b)
(c)
Figure 2: Undirected probabilistic graphical models of DiCoM: (a) for unlaheled sample, (b) after marginalization of the views and (c) for labeled sample.
Considering the This probabilistic relation is known the consensus potential Yu et al. 201
whole graph, this potential implies that all views are random Gaussian variables with a shared mean fo and variance o2 As result, the views stay consistent w.r.t. each other by taking values not too far away from the shared consensus. This graphical model, shown Fig. 3 (a), is assumed for each unlabeled sample. The joint density associated with the graph is given by
1
II
T (Te, fm)
is the potential function of the edge connecting Ja and fm. From this model, derive important results. On side note,
where is normalizing constant and & (fer fm) = exp
20
4

---

Under review as a conference paper at ICLR 2022
derivation generalizes to vector-valued labels y, but here assume scalar labels for ease of exposition. The proofs of our results are provided in Appendix A
(I) Marginalization of the views: By integrating the latent consensus function fo out of the joint density, the marginal distribution of the views is
p(i....., & exp
ΣΣ -mm, (Jm Sx)²
where Am,k
20 ok
Em omm
This result implies that the marginal likelihood can be fac-
torized product of (2) terms. Each term an isotropic Gaussian distribution on the difference between pair of views (Jm: fa), with zero mean and variance (2)m, ) The equivalent graphical
model shown Fig. Bb)
(II) Conditional of the consensus function: By applying Bayes' theorem, the conditional distribu- tion of the consensus function fe given all the views a Gaussian
JM ~ N (ū,
(4)
and jr = Em fm. This result highlights that the conditional distri- bution of fa depends only the weighted average and the values of individual views are not required. Furthermore, fi can be treated as a view itself, with a variance that smaller than any of the variances of the views.
where 07; = (Em
o
Derivation of DiCoM loss function: For simplicity, we assume equal variance for different deep views, i.e., of of Vm. For an unlabeled sample (Ir,), we directly apply the first result (I) to obtain the following negative log likelihood function
M
Luni
2M (fm((n) fx(zn))
(5)
For labeled sample (Zn, Yn), since the ground-truth is given, assume graphical model that involves the final DiCoM prediction, i.e. the averaged output μ.. This graph shown in Fig. (jc).. Since assume shared variance oZe the weighted output now reduces an equal-weight average, following from result (II)
M Σ
fm((Fn) p(xn) of
of M
(T
(6)
M
Subsequently, we apply result (I) on this graph to get the negative log likelihood as follows M
Libb = 2Mo 20
In x(=n))
1
Σ (m)(a)) - -n -am3-(((mm) )3
2M + 20
m=1
M
Σ
(xm [m - ynl - - m(xn))
20
10
8 we have applied the ambiguity decomposition (Krogh et al. 1995) and in equation (9) we have assumed that the label accurate, i.e., o² < oZ
where in equation
Given training batch of labeled samples {(En, Vr Hi and unlabeled samples {(x,) Mr assum ing that the samples are independently generated, we can add the log likelihood functions across all training samples. This can be done via simply adding up two equations (5) and
L
M
EDICOM =
Σ
(Fm( - ynl? Kadiv (xm( - m(xn))]
L
==1
m=1
(10)
U
M
+
Σ
Σ
Kese [frn (xt - fa (2rw))
0
==1 m=1 kmm
5

---

Under review as a conference paper at ICLR 2022
where we introduce two hyperparameters Rdiv and Kese to absorb other constants and to enable trade-off between diversity and consistency components of the loss.
The DiCoM loss encourages diversity on labeled data, while enforcing consistency on unlabeled data. These two seemingly opposing components can both be derived from the same underlying graphical assumptions. Furthermore, they should not be weighted equally. In fact, have shown that it depends on the number of views: when M increases, diversity grows in O(M), while con- sistency grows in (M²)). It is worth noting that our method is fundamentally different from other extensions of NCL such as Semi-supervised NCL (Jean et al. 2018). which enforces diversity on both labeled and unlabeled data. Last but not least, since both diversity and consistency are incorpo- rated in the DiCoM objective function, the method is highly adaptable to different implementations such as multi-network or multi-branch, as long as the views are provided.
EXPERIMENTS
In this section, we study the proposed method in different settings, including regression tasks on eight tabular datasets and a crowd counting task on image data. We provide additional experiment results in Appendix and additional experiment on toy data in Appendix D
4.1 EXPERIMENTS ON UCI TABULAR DATA
Datasets: We evaluate DiCoM on eight datasets from the UCI repository (Dua & Graff 2017 1
skillcraft, parkinsons, elevators, protein, blog, etslice, buzz, and electric. These datasets are collected from real-worid regression scenarios, with varying sample sizes and input dimensions. For each dataset, we keep 1000 labeled samples as a hold-out test set; further retain N 300 samples for the labeled training set, and keep the rest as the unlabeled training set. We follow the realistic evaluation setup in (Oliver et al. 2018) and use a 90%-10% rrainvalidattin split, i.e., 270 samples are used for training, leaving only 30 for validation.
Experiment Setup: We implement both variants of DiCoM, Our DiCoM-N networks adopt the same architecture (Jean et al.) [018 Wilson et al., 2016), which is fully-connected multilayer perceptron with four hidden layers, containing 100, 50, 50 and 2 hidden nodes, respectively. Our DiCoM-B also utilizes this architecture, but branch out after the third hidden layer, i.e., the backbone contains hidden layers of 100, 50 and 50 nodes, while the branches each contains one hidden layers of 2 nodes. This model is trained end-to-end, the backbone is trained together with the branches. DiCoM hyperparameters (Kaiv, Kese) are chosen from a grid of values based on validation errors. Across 10 random seeds, we report the noot-mean-squared errors (RMSE) statisties on the test set. For simplicity, we append '-M" to the end of our method name to denote the number of views, e.g., DiCoM-B-4 represents the multi-brance DiCoM network with 4 branches.
we apply zero-mean Gaussian noise which is commonly used for tabular data. For the DiCoM-B model, Gaussian noise is applied on the input and on the features at the beginning of each branch, right after branching out. Since the independent Gaussian noise is added during the forward pass, it does not affect the gradient values during backpropagation. The exact amount of Gaussian noise can be found in Appendix B
Data Augmentation:
We compare DiCoM against six semi-supervised regression methods: SSDKL. (Jean et al. 2018), COREG (Zhou & Li 2005), LP (Zhur & Ghahramanirh) 2002). VAE (Jean et al.) 2018). MT (Tar- vainen & Valpola| 2017) and VAT (Miyato et al.) 2018). These methods span a wide range of approaches such as consistency regularization (MT, VAT), entropy minimization (SSDKL), multi- view learning (COREG), graph-base based (LP), or generative modeling (VAE). The detailed information about the datasets and experiment setup can be found in Appendix B
Fig. 3 shows the experiment results. We observe significant improvements compared to the state-of-the-art semi-supervised regression methods. The largest performance gains achieved on parkinsons and etslice, where DiCoM-N-2 improves upon the best competing method by 32.4% and 42.5%, respectively. DiCoM-N-2 also outperforms DiCoM-B-2 on all datasets, ex- cept for protein. This is expected since DiCoM-N-2 has almost twice as many learnable param- eters as DiCoM-B-2. For elevators, the most frequently selected hyperparameters across 10 ran-
Results:
https://archiver iss uci www.emddmmm//mee hindex

---

Under review as a conference paper at ICLR 2022
dom seeds (Kdiv Kesc) (1, 0.01), while for etslice, the most frequently selected values (Rdiv Fiese) = (0.1.1), This shows that different datasets require different trade-offs between con- sistency and diversity. We also notice that LP (a graph-based method) and COREG (a nearest- neighbors-based method) performs relatively well on blog, etslice and buzz. Meanwhile, MT and VAT, which based consistency regularization (without diversity regularization), did not per- form well on these regression datasets, even though they have been shown to be effective on classi- fication tasks.
0.30
0.8
0.25
0.6
0.4-
0.2
0.10
0.3
0.1
0.05
0.1
0.0
0.00
blog
obiice
buse
electric
1.2
1.75
1.0
15:
0.6
0.6
0.4
0.4
0.59
0.2
0.0
0.00
0.0
SSDKL
COREC
LP
VAE
MT
VAT
DICoM- N-2
DiCaM-B-2
Figure 3: Test RMSE on UCI datasets: each subplot shows the results for one dataset.
We analyse the effect of different components of the DiCoM- N-2 model by individually removing them from the model. For the first model, Ablation-1, remove data augmentation. the consistency loss labeled data is loss to both labeled and unlabeled training data for model Ablation-4. Using the results of DiCoM- N-2 as the baseline, we also report the percentage reduction test RMSE (% Redc.) for the other methods as follows:
Ablation study on the components:
Ablation-2, we remove the diversity loss on the unlabeled data. Next,
zero for Ablation-3 model. Lastly, apply diversity
Percentage Reduction = (baseline score - new score) x 100
(11)
baseline score
Table reports the ablation results. The average percentage reduction scores tell us the importance of each component. Augmentation has small impact on the performance of DiCoM-N-2, while diversity and consistency regularization are both important. DiCoM-N-2 outperforms Ablation-4 in all cases, which suggests that mere reliance on diversity is insufficient.
Table 1: Test RMSE from Ablation Study on UCI Datasets.
DICoM-N-2
Ablation-1
Ablation-2
Ablation-3
Ablation-4
Aug. Lab. Div.
Unlab. Csc.
Unlab. Div.
V
RMSE
RMSE
Rede.
RMSE
skillcraft
0,313 1: 0.005
0,330 1: 0.017
-5559
0.327
0.008
-4.610
0.333 2: 0.012
-6.446
0.330 1: 0.011
-5.326
parkinsons
2.285
0.208
2.437 0.280
-6.666
2.390
0.278
.45993
2.563 0.299
-12.159
2.438
4: 0.286
6.692
elevators
0.145
0.142
0.031
2.022
0.149
0.022
-2.328
0.155
0.027
-6.225
0.157
0.028
-7,936
protein
0.646 : 0.031
0.679 1 0.029
-5.153
0.654
0.025
-1.313
0.669
- 0.034
-3.546
0.661
4: 0.020
-2.327
blog
0.930 0.040
1.014 0.039
.8.98
1,021
0.051
.9.752
1.027
0.036
-10.455
1.007 0.045
.8.261
etslice
5.575 1: 0.606
6.976
0.984
25.120
7.154
1.176
-28.306
7.461
2 1.014
-33,813
7.398 1 0.662
-32.699
buzz
0.715 0.136
0.757
: 0.064
.5.825
0.938
0.489
-31.184
0.830
=: 0.131
-16.045
1.038 4 0.550
45.085
electric
0.114
0.097
0.124
-33.833
6.327
0.117
-31,543
Average
-5.073
-14.490
-10.295
-17.484
Varying the number of views: We further evaluate the impact of the number views M. Fig. shows the performance of three DiCoM-N models with increasing number of views M {2, 4,
The results show that a larger value of M leads to an improvement the performance. When M increases from 8, the average reduction in test RMSE is 3. 3.48%, larger than the average reduction

---

Under review as a conference paper at ICLR 2022
rate of 1.70% when M increases from 4. While varying the number of views, also monitor the changes in the model hyperparameters. Using the values that were selected to minimize the For most datasets,
validation eπror of DiCoM-N, Table 2 shows the log ratio of (Kdiv/ we see that this log ratio tends to increase for larger number of views. This is because in LDiCOM- the number of diversity terms grows in ((M) while the number of consistency terms grows in ( Thus, as M increases, larger (Kdiv/ (Keie) ratio is required to keep those terms balanced.
skillcraft
parkinsons
elevators
protein
6.7
.350
8:30
blog
ctslice
buzz
electric
1.0
04
100
Figure 4: Test RMSE of DiCoM-N UCI datasets with varying number of views M (2, 4,
The x-axis shows number of views M, the y-axis shows test RMSE.
Table 2: Median Values of log (Hgg((iai///)ee)) across 10 Seeds from DiCoM-N.
skilleraft
parkinsons
elevators
protein
blog
etslice
buzz
electric
.M
1.849
-0.151
1.151
0.500
1.000
-1.000
0.151
1.849
M = 4
2.000
-0.301
1.151
0.301
1.151
0.199
0.349
2.000
M 1.699
-0.500
1.500
1.151
1.849
0.349
0.500
2.000
Comparing multi-network and multi-branch: In this experiment, we compare the two variants of DiCoM against each other, by reporting both their performance and execution time (adding train and test time). Table 3 shows the percentage reduction computed using equation (TT) by treating DiCoM-N's results the baseline scores and DiCoM-B's corresponding results the new scores. It can be seen that DiCoM-N is consistently outperforming DiCoM-B in terms of test RMSE. The multi-network variant is also the faster option when M However, the number of views increases, the execution time of DiCoM-B is significantly faster. This shows that while DiCoM-N achieves better test performance, DiCoM-B demonstrates better scalability.
Table 3: From DiCoM-N to DiCoM-B: Percentage Reduction in Test RMSE and Execution Time.
Test RMSE
Execution Time
Dataset
M =
M
= =8
M = 2
M 4
M
skilleraft
-0.702
2.012
-2.945
-33.336
23.505
48.462
parkinsons
-43.933
-24.177
-57.821
36.845
$5.590
72.481
elevators
-0.436
-13.126
-21.811
-52.633
19.389
60.346
protein
2.798
-0.536
-0.520
-45.076
19.876
62.177
blog
-0.430
-6,908
-18.729
-393355
39.847
47.129
ctslice
-54,662
-34.924
-73.619
45.016
81.627
83.731
buzz
-15.885
-49844
-17.976
4.194
33.759
51.035
Average
-16.203
-11.806
-27.632
-12.049
39.085
60.766
4.2 EXPERIMENTS ON VISION DATA
In order show the versatility of DiCoM, further conduct experiments crowd counting task. Crowd counting is a fundamental question the vision community due to its far-reaching applica- tions in many scenarios, including video surveillance, metropolis security, human behavior analysis.
8

---

Under review as a conference paper at ICLR 2022
Crowd counting has been recently used benchmark for deep regression algorithms (hang et al. 2019); for this task, counting by regression has been perceived as the state-of ofate-of-the-art approach.
Dataset: We study the ShanghaiTech Part-A dataset (Zhang et al. 2016 This is new large- scale crowd counting dataset that contains extremely congested scenes, with varying perspective and unfixed resolution. The data are split into 300 training and 182 test samples. Among the 300 training samples, we randomly select N E {30, 120, 210} samples as the labeled set and use the remaining data as the unlabeled set. We follow the common practice to report both mean absolute errors (MAB) and noot-mean-squared errors (RMSE) on the test set. We note that this dataset inevitably contains personally identifiable information, which has been made public by the owner of the dataset.
Experiment Setup: In our experiments, we adopt the network architecture of CSRNet B (Li et al., 2018) and implement DiCoM-B-4, More specifically, we use pre-trained VGGI6 network as the encoder and append another decoder on top of it. In the penultimate layer of the decoder, we enlarge the number of hidden channels by M times. We then apply group-convolutional layer as the last layer, setting both the number of output channels and group size to M. Thus, the backbone of DiCoM-B-4 includes the pre-trained VGG16 and the decoder up to its penultimate layer. This architecture has been shown to be very effective in generating multiple predictions without much increase in computational cost over a single network (Shi et al. 2018 Zhang et al. 2019: Zhou
al. 202T
We compare DiCoM with the following competing methods: (i) the supervised baseline which uses only the labeled training samples and standard MSE loss; (ii) the DNCL model (Zhang al. 2019); and (iii) the Co-Regression model. Since we are not running for multiple random seeds, we remove all random data augmentations (e.g., cropping, flipping) to enable fair comparison between different methods.
From the results in Table 4 observe that both DNCL and Co-Regression outperform the supervised baseline by enforcing either diversity on labeled data or consistency on unlabeled data, and that overall, DiCoM-B-4 outperforms other methods by incorporating both diversity and consistency on the unlabeled data.
Results:
Table 4: Test Results ShanghaiTech.
Supervised Baseline
DNCL
Co-Regression
DiCoM-B-4
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
30
297.45
551.68
250.91
360.66
224.63
344.46
163.41
227 .42
N 120
132.05
210.82
119.48
186.80
128.57
206.90
114.31
181.91
N 210
107.76
165.91
104.00
165.97
103.20
165.04
101.90
155.49
5 CONCLUSION
In this paper, we proposed novel Diverse and Consistent Multi-view Networks for Semi-supervised Regression (DiCoM), that elegantly combines ensemble diversity with consistency regularization. DiCoM utilizes probabilistic graphical models to control the underlying dependencies among mul- tiple regression outputs and label. We also show that DiCoM is highly flexible, it can be adopted for multi-network or multi-branch implementations, the latter significantly improves the scalability of the method. Experiments on tabular UCI and visual ShanghaiTech datasets demonstrated the effectiveness the proposed method across diverse domains, while ablation studies validated the importance of both consistency and diversity. In the future, one may extend the DiCoM frame- work by introducing asymmetric views (of non-identical architectures). which naturally causes the final output y be an uneiuhel average. Another interesting direction is to explore the potential impact of data augmentation techniques, since multi-view learning often benefits from diversified inputs.
htteers//thhhlccc///eeeeee/////s wecphnggghhiiiihaaaaaaaaaaaa (BSD 2-clause license)

---

Under review as a conference paper at ICLR 2022
REFERENCES
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the eleventh annual conference on Computational learning theory, pp. 92-100. ACM, 1998.
Gavin Brown, Jeremy L Wyatt, Peter Tino, and Yoshua Bengio. Managing diversity in regression ensembles. Journal of machine learning research, 6(9), 2005.
Huanhuan Chen, Bingbing Jiang, and Xin Yao. Semisupervised negative correlation learning. IEEE transactions on neural networks and learning systems, (9(11)5366-5379 2018.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://arcciv..
ics.uci eemmmm
Chen Gong. Exploring commonality and individuality for multi-modal curriculum learning. In Thirty-first AAAI conference on artificial intelligence, 2017.
Neal Jean, Sang Michael Xie, and Stefano Ermon. Semi-supervised deep kernel learning: regression with unlabeled data by minimizing predictive variance. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 5327-5338, 2018.
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson WH Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 6728-6736, 2019.
Anders Krogh, Jesper Vedelsby, al, Neural network ensembles, cross validation, and active learn- ing. Advances in neural information processing systems, 7:231-238, 1995.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):4364444 2015.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csmet: Dilated convolutional neural networks for understanding the highly congested scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1091-1100, 2018.
Yong Liu and Xin Yao. Ensemble learning via negative correlation. Neural networks, 13919):1399- 1404, 1999.
Yong Liu, Xin Yao, and Tetsuya Higuchi. Evolutionary ensembles with negative correlation learn- ing, IEEE Transactions Evolutionary Computation, 4(4):380-387, 2000.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Cuong Manh Nguyen, Xin Li, Ronald De Shawn Blanton, and Xiang Li. Partial Bayesian co-training for virtual metrology. IEEE Transactions on Industrial Informatics, 16(5):2937-2945, 2019.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep supivsuper learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018.
Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi- supervised image recognition. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 135-152, 2018.
10

---

Under review as a conference paper at ICLR 2022
Zenglin Shi, Le Zhang, Yun Liu, Xiaofeng Cao, Yangdong Ye. Ming-Ming Cheng, and Guoyan Zheng. Crowd counting with deep negative correlation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5382-5390, 2018.
Ke Tang, Ponnuthurai N Suganthan, and Xin Yao. An analysis of diversity measures. Machine learning, 65(1):247-271, 2006.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve ceni-supervised deep learning results. In Advances neural information processing systems, pp. 1195-1204,201
Naonori Ueda and Ryohei Nakano. Generalization error of ensemble estimators. In Proceedings of International Conference on Neural Networks (ICNN"96) volume 1, pp. 90-95. IEEE, 1996.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn- ing, 109-2):373-440, 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial intelligence and statislics, pp. 370-378. PMLR, 2016.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks, 2017.
Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. arXiv preprint 1xXvv:304.5634 2013.
Shipeng Yu, Balaji Krishnapuram, Rómer Rosales, and Bharat Rao. Bayesian co-training. Journal of Machine Learning Research, 2(Sep))2649-268)) 2011.
Le Zhang, Zenglin Shi, Ming-Ming Cheng, Yun Liu, Jia- Wang Bian, Joey Tianyi Zhou, Guoyan Zheng, and Zeng Zeng. Nonlinear regression via deep negative correlation learning. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 2019.
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd count- ing via multi-column convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 589-597, 2016.
Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-vies view learning overview: Recent progress and new challenges. Information Fusion, 38:43-54 2017.
Joey Tianyi Zhou, Le Zhang, Du Jiawei, Xi Peng. Zhiwen Fang. Zhe Xiao, and Hongyuan Zhu. Locality-aware crowd counting. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2021.
Zhi-Hua Zhou and Ming Li. Semi-supervised regression with co-training. In IJCAI, volume 5, pp. 908 913, 2005.
Xiaojin Zhur and Zoubin Ghahramanirh. Learning from labeled and unlabeled data with label prop- agation. 2002.

---

Under review as a conference paper at ICLR 2022
A MATHEMATICAL PROOFS
Consider again the general model, where there are M deep views, i.e., {fm JM Graphically, these functions are represented by nodes that are connected not directly, but only via the consensus function f, using isotropic Gaussian potentials
f. fm x N (0,
(12)
We note that in this general case, each deep view is vector (instead of scalar) and is assigned separate variance oh which are not necessarily equal to each other. In terms of notation, we use italic letters for scalar variables and boldface letters for vectors and matrices.
A.1 MARGINAL DENSITY THE VIEWS
In this proof, derive the marginal distribution of the views. Given the DiCoM graphical model, it is necessary to integrate f. out of the joint density distribution of the graph, because fe is latent variable. The joint density distribution function of this graphical model is as follows
M
| fmll
(fe,, ft...,,..... .ar)
exp
(13)
Z1 m=1
20 m
f
f fm
21 exp
(14)
2
20
2
m
m
exp
1
ce 2
fe + Φ T fe + +x
(15)
2
where the normalizing factor 21 is constant w.r.t fe, fi...... far and
F C.
X
(16)
2
2x m
Notice that v/, X constants w.r.t. f.. By applying the following integration rule for multivari- ate Gaussian variable x Petersen et al. (008))
[ fexp -xxxxxx+cxx dx /dt (2TA -1) exp (NOTATTC)
(17)
we can integrate out of the joint distribution in [5) obtain the following marginal likelihood
(f, ... fas) = (Per
f....., (.r) df.
(18)
1
④ +
exp
(19)
Z2
2Φ
1
1
Tf
fh
fm
exp
+ 2
1
(20)
Zg
21b
kSm
1
T
2f fg
f.T fxz
exp
(21)
Z
of
k>m
1
fm 1
exp
(22)
Zg
2p
1
exp
ΣΣ
Am, le fl - f2ll?
(23)
Zn
m
kmm
12

---

Under review as a conference paper at ICLR 2022
where Z2 is another constant w.r.t. fe. fi...... ,MM and
M
Pm. = R = o
Σ 1=1 of
(24)
M
Am,k- = (2pm,))
20
(25)
A.2 CONDITIONAL DENSITY OF THE CONSENSUS FUNCTION
In this proof, we derive the conditional density distribution of the consensus function f. given the views. Consider again the general model with M views, i.e., {fmm JMM Each view is represented by a random variable connected only to the consensus function f. via an isotropic Gaussian potential
as defined in (1).. From
22
the conditional distribution of fo given the views fi...., fy is
p (f., ......a))
p (f. fL....... fas) =
(26)
p (fi,....,fad
1
M
llfe Fm
2
M
M
12
2g
exp
20
2
(27)
m=
m
m
k>m
1
f f. + 2F fc
F.T Lm
F&
Zg
exp
20
2pm
(28)
1
f fc
ū fe
cxp
+
N
(29)
Z
20
o
where the normalizing factor Z3 is a constant w.r.t. for fi,..., fas and
M
Σ
(30)
m==
1
fm
0
(31)
m
f.
M
M Fm
1211
(32)
20 m
2pt
m=1
==
k>m
Using the definitions
24
30
and
31
N can be rewritten as follows
fm
f.
12
N =
my
(33)
20 m oN
20
oh
2
m
m kmm
f fm
Σ 1
| fg 2
01
(34)
202
20 m
of
k>m
T fm
Σ
(35)
20 m
2
20 o.
o2 o
m
kmmm
m
k>m
f fm
f
ΣΣ
(36)
20 m
k>m
m
2
(37)
2
2
m
m
- T μ
(38)
20
13

---

Under review as a conference paper at ICLR 2022
Thus, can rewrite (29 in its Gaussian form p (felf
1
fo fe
fc
1
[1lffc lIfe
20
)
fas)
exp
+
exp
(39)
23
20 M
oH
20
23
20
Therefore,
fe
fM
N (i, (
(40)
14

---

Under review as a conference paper at ICLR 2022
B EXPERIMENT SETUP DETAILS
In this section, we provide the detailed setups for our benchmarking experiments. All experiments are rum on NVIDIA GeForce GTX 1080Ti GPUs, using an Anaconda virtual environment installed with CUDA version 10.1, Python version 3.7.10 and Pytorch version 1.7.0.
Experiment setup for DiCoM in UCI experiments:
Network architecture: for DiCoM-N, fully-connected multilayer perceptron with hidden layers of size [100, 50, 50, 2/. For DiCoM-B, the backhone includes the first 3 hidden layers of size [100,, 50, 50] and the branches include one hidden layer of size Note that we are not counting the input and output layers.
Parallelization: for DiCoM-B, we use group convolution in order to back-propagate through all branches simultancously.
Random seeds: 20, 40, ... 200 (10 seeds in total).
Training: 2000 epochs with 250 epoch patience for early stopping (stop if no improvement is observed on validation set for 250 consecutive epochs).
Optimizer: Stochastic Gradient Descent with momentum 0.95 and weight decay 10-9, Learning rate is 10 for etslice and is 10-3 for other datasets.
Augmentation: additive random Gaussian noise with mean and standard deviation 0.05 for DiCoM-N and 0.01 for DiCoM-B.
Diversity hyperparameter search range: Kdiv {0.01, 0.05, 0.1, 0.5, 1}. Consistency hyperparameter search range: Kese E (0.01, 0.05, 0.1, 0.5, 1).
Experiment setup for DiCoM in ShanghaiTech experiment:
Network architecture: CSRNet Bi et al. 2018
Random seeds: 9999 (only seed).
Training: 1000 epochs with no early stopping.
Optimizer: Adam with weight decay 10-5 and learning rate 10-5
Augmentation: None.
Diversity hyperparameter search range: Kiiv (10-5, 10-1, 10-3). Consistency hyperparameter search range: Kese {10 10 4, 10 10-3). 3}.
15

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL EXPERIMENT RESULTS ON UCI DATASETS
Information of UCT datasets: The detailed statistics of the eight UCI datasets are given the table below.
Table 5: UCI Regression Datasets
Dataset
No. of Samples
Input Dim.
No. of Unique Label Values
Prediction Target
skillcraft
3.325
18
Skill level gamers (ordinal classification)
parkinsons
5.875
20
1,129
Unified Parkinson's 's Discase Rating Scale (UPDRS) scores
clevators
16,599
18
61
Aileron control F16 aireraft
protein
45,730
9
15,903
Physioochemical properties protein tertiary structure
blog
52,397
280
438
Number of comments received within 24 hrs
cislice
53,500
384
53,347
Relutive location of the image on the uxial axis
buzz
583,250
77
8,123
Popularity a topic social media
2,049,280
4,186
Power consumption in one household per minute
Additional experiment results UCI datasets with labeling budget N
300: Please see the
below additional results from Section 4.1
Table 6: Test RMSE on UCI Datasets with Labeling Budget N
300.
Datuset
SSDKL
COREG
VAE
MT
VAT
DiCeM-N-2
DiCoM-B-2
skilleraft
0.346 1: 0.036
0.361 :: 0.009
0.342 =: 0.012
0.39 :: 0.014
0.418 a: 0.038
0.454 = 0.025
0.313 1: 0.005
0.315 0.013
parkinsons
3.379
0.387
4.285 + 0.200
5.371 - 0.994
10655 + 0.151
8.725
0.904
10612 0.198
2.285 0.208
3,290 0.481
elevntors
0.151
0.013
0.006
0.189 0.055
0.228
0.261 0.031
0.261 0.012
0.145
0.025
0.146 0.035
proteia
blog
1.146 0.072
L,076 1: 0.040
1017 1 0.048
1,033 0.045
L.117 0.036
1.087 0.063
0,930 0.040
0.24 0.06
etslice
9.691 + 1.135
10.023
1,235
102200
1440
16.301 + 0.777
16,845 + 1.996
16.992 = 4.839
5.575
0.606
8.64 0923
buzz
0.766
0.040
0.752
+ 0.035
0351 0.000
1,098
=: 0.026
1,702 0.160
L,743 0.186
0.715
0.136
0.299 0.05
electric
0.117
0.011
0,261
0.008
0448
0,010
0.977 0.017
0.696 0.025
0.966 0.127
0.114
0.025
0.799 0.021
Table 7: Test RMSE DiCoM-N on UCI Datasets with Varying Number of Views, N 300
DiCoM-N-2
DiCoM-N-4
DiCoM-N-8
Dataset
RMSE
RMSE % Rede.
RMSE % Redc.
2 -
-
skilleraft
0.313 + 0.005
0.319 + 0.025
-1.917
0.304
0.008
4.702
parkinsons
2.285 + 0.208
2.291 + 0.355
-0.263
2.200 +: 0.219
3.972
elevators
0.145 + 0.025
0.135 1 0.020
7.187
0.125 0.010
7.407
protein
0.646 + 0.031
0.636 + 0.027
1.548
0.635 0.029
0.157
blog
0.930 + 0.040 0.040
0.912 + 0.030
1.935
0.897 0.020
1.645
ctslice
5.575 1 0.606
6.233 1 0.524
-11.795
6.174 0.587
0.947
buzz
0.715 + 0.136
0.688 0.078
3.804
0.654
0.041
4.942
electric
0.114 1 0.025
0.099 + 0.008
13.132
0.095 0.012
4.058
Average
1.704
3.479
Additional experiment results on UCI datasets with labeling budget 100: Using the same
UCI datasets, conduct experiments similar the ones in Section 4.1 with smaller labeling budget of N 100 samples. The results are provided below.
Table 8: Test RMSE on UCI Datasets with Labeling Budget N 100.
SSDKI.
COREG
VAE
VAT
DiCoM-N-2
DiCoM-B-2
parkinsons
4.701
0.687
5.552
7.037 1.373
10.694 0.316
0.778
10.738
0.662
dlvatoors
0.182
0.019
0213 + 0.000
0.202 0.008
0.351 + 0.029
01269
0.039
0.261
0.015
0.179
0.030
0.194
0.000
protein
0.741
0.050
0.751 1 0.015
0.766 0.022
0.789 1 0.006
0.757
0.005
0.784 1 0.050
0.691
1 0.021
0.675
0.013
blog
1.129 0.081
1.067 + 0.078
1.115 0.001
1051 4 0.100
1.112 0.03
1.136 0.11
1,031 0.055
0.998 :: 0.052
etslice
12.646 1,059
12.911
13.726 1.178
09750.13.199
17,152 1.352
18,318
10.742 0.629
12.022 1.413
buzz
0.006
1.114 0.036
0.330
electric
0.043
16

---

Under review as a conference paper at ICLR 2022
Table 9: Test RMSE from ablation study UCI datasets, N 100
DiCoM-N-2
Ablation-1
Ablation-2
Abiation-3
Ablation-4
Aug.
Lab. Div.
Unlab,
Unlab Dis.
RMSE
RMSE
% Redc.
RMSE
% Redc.
RMSE
% Redc.
RMSE
% Redc.
skillcraft
0.342
0.011
0.354
0.030
-3.404
0.361
0.032
-5.372
0.355
0.031
-3.756
0.363
0.032
-..24
parkinsons
elevators
0.179
0.030
0.186
0.033
-3.844
0.192 0.032
-6.864
0.199 0.029
-7.574
0.197
0.026
-9.715
protein
0.691
0.021
0.705
0.023
-1.959
0.711
0.013
-2,818
0.723
0.340
-4.003
0.714 = 0.023
-3.268
blog
0,065
10.742
0.564
11.341
-5.575
10,673
buzz
0.836
0.093
0.918 0.085
-9.760
0.933
0.263
-11.599
1.019
0.183
-21.893
1.017 0.242
-21.613
electric
0.185
0.116
0.176 0,103
4.602
0.198
0.115
-6.798
0.158
0,023
14,693
0.193
0.104
-4460
4.037
Table 10: Test RMSE of DiCoM-N on UCI Datasets with Varying Number of Views, N
100
DiCoM-N-2
DiCoM-N-4
DiCoM-N-8
Dataset
RMSE
RMSE
% Rede.
RMSE
Redc.
skillcraft
0,342 1 0.011
0.349 1 0.022
-1.977
0.334 1 0.018
4.250
parkinsons
3,580 0.662
3.343 + 0.394
6.610
3.250
2.782
elevators
0.179 0.030
0.171 + 0.026
4.605
0.169
0.028
1.185
protein
0.691 0.021
0.681 + 0.021
1.456
0.679
0.294
blog
1.031 0.055
= 11.164 + 1.972
5.432
0.961
1.436
ctslice
10.742 0.629
-3.924
10.457
1.109
buzz
0,836
0.832 + 0.083
0.447
0.796
electric
0.185
0.171 + 0.059
7.650
0.137
20.096
Average
2.537
5.091
Table 11: From DiCoM-N to DiCoM-B: Percentage Reduction in Test RMSE and Execution Time,
N 100.
Test RMSE
Execution Time
Dataset
M
2
M
M
M
2
M
M
skilleraft
3.503
-1.323
-10.419
-72.882
18.200
59.470
parkinsons
-24.102
-30.558
-57.663
28.085
28.085
65.291
clevators
-8.069
-13.635
-20.037
-11.291
37.481
37.481
protein
2.260
-5.401
-15.822
-57.697
20.049
63.618
blog
3.219
-4,069
-11.128
23,903
23.812
56.539
ctslice
-7.157
-6.599
-45.748
22.144
42.888
77.162
-16.619
-9.784
-2,9556
0.524
46.706
68.124
Average
-6.709
-10.124
-26.690
-9,602
31.032
61.098
17

---

Under review as a conference paper at ICLR 2022
Sclerat
elexators
0.5
0.8
0.7
0.5
0:15
0.4
0.2
0.10
0:3
0.1
0.05
0.3
0.0
0.00
0.0
blog
<biice
eloctre
1.0
1.75
1.0
0.8
1.00
0.4
02
0.50
0.2
0.0
0.00
0.0
SSDKL
COREG
VAE
MT
VAT
N--
DiCoM-8-2
Figure 5: Test RMSE on UCI datasets: each subplot shows the results for one dataset, N 100.
D ADDITIONAL EXPERIMENT ON ToY DATA
We conduct an experiment on synthetic toy dataset to illustrate how multi-view diversity and consistency work together to affect the training and inference of DiCoM-N.
We synthesize regression dataset where inputs x E R230 and labels y E RR, The labels are related to the inputs by y Az+ where A is fixed 2 30 coefficient matrix and . NNO,, 0.3² Each coordinate of æ drawn from the standard normal distribution. We generate a training set of 100 labeled and 1000 unlabeled samples, and hold-out test set of 1000 labeled samples.
Dataset:
Experiment Setup: Our DiCoM-N model has = 5 views, each uses simple neural network with single hidden layer containing two hidden nodes. We train the model with SGD for 50 epochs with a learning rate of 5 x 10 then evaluate the mean-squared-erss (MSE) of the model on the test set.
Results: We keep Kdiv = 1 and vary Kese on a log scale: Rese E {0.01, 0.1, 1}. Both quantitative and qualitative results shown in Fig. [ We plot the training losses the top and visually show the predictions of each network on eight random test samples. On the left scenario (Fig 6 a)) when Kdiv Kesc, the diversity loss dominates the consistency component. Even though the total loss converges on the training set, the individual views' losses do not, resulting in their large bias on test samples. This can be an issue if the DiCoM-N model contains smaller number of views. On the other hand, the consistency enforcement is too strong on the right scenario (Fig. ((c)). The individual views and the averaged output seem to have all collapsed into a single point. Where the models collapse to is limited by the individual networks' capacity and may not necessarily be the global optimum for the averaged output. Finally, in the middle scenario (Fig. ((b)), the effects of diversity and consistency losses are balanced, yielding good trade-off. The averaged model output able to perform better than each individual view, and is also the best among three scenarios. These results also suggest that even though diversity and consistency contradicting forces, they still be applied simultaneously on the regression outputs to produce the desirable behaviours.
18

---

Under review as a conference paper at ICLR 2022
Ndiv
Ndiv Nov
Fidiv Nov
Training Locs (MSE)
15
10*
div
div
12
Training
total
100
div
6Sc
10
20
40
50
20
30
50
10
30
Egoch
Epoch
Epoch
5.0
2
1
0.0
(1.12e+01)
(1.40e
(1.25e
-.5
04)
34e-01)
(1.25e-01)
-5.0
f3 +
f3 (3.79a.01)
(1.25a-01)
(1.99e+01)
(1.33e-01)
(1.25e:01)
15 (1. (6e++0+0)
15 (2.77e-01)
f5 (1.25e-01)
(1.27e-01)
(1.20e-01)
(1.25e-01)
-10.0
label
label y
label y
-5.0
-25
0.0
2.5
7.5
-5.0
255
5.0
Figure 6: Experiment results on toy data. The top row shows training losses in symmetric log scale. The bottom row shows model predictions on eight random test samples. In the legend, next to model name, we report the MSE scores evaluated on 1000 test samples.
19


Reviews:
[{"review_id": "RjnylkcTvTB", "reviewer": "Reviewer_UrUc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThe consideration of diversity for semi-supervised learning is interesting, and the reviewer considers that it is helpful for improving performance.\n2.\tThe proposed method combines diversity with consistency based on underlying probabilistic graphical assumptions is a meaningful idea.\n\nweakness\n1.\tThe idea of this paper is direct. This paper still considers traditional semi-supervised learning, and the multi-view refers to the view generated from instance augmentation. Why only measure the diversity of labeled instances and not extend to unlabeled instances? Are there relevant experimental verifications?\n2.\tWhat is the advantage of combining diversity with consistency based on underlying probabilistic graphical assumptions? Several multi-view methods have already considered the diversity with consistency simultaneously, but the authors choose to ignore them, e.g., “Exclusivity-Consistency Regularized Multi-view Subspace Clustering”, “End-to-End Adversarial-Attention Network for Multi-Modal Clustering”. More analyses and comparisons are expected, the reviewer considers that these ideas can also be transformed into the semi-supervised framework.\n3.\tAlthough this paper considers semi-supervised regression, why can't the semi-supervised classification method extend to the regression problem by replacing the loss function? The SOTA comparison method is in 2018, more comparison methods recently are expected.\n\n", "summary_of_the_paper": "This paper concerns the semi-supervised problem. Different from SOTA deep semi-supervised methods, the proposed DiCom employs a diversity measure on the labeled multi-view data, and combines diversity with consistency based on underlying probabilistic graphical assumptions. Experiments verify the effectiveness.", "summary_of_the_review": "This paper concerns the semi-supervised problem. Different from SOTA deep semi-supervised methods, the proposed DiCom employs a diversity measure on the labeled multi-view data, and combines diversity with consistency based on underlying probabilistic graphical assumptions. Experiments verify the effectiveness. However, there exist several shortcomings for this paper, so I recommend weak rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QeTv1lC_K2Y", "reviewer": "Reviewer_6car", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The ablation study missed an interesting question:  would increasing the number of views with either consistency or diversity loss surpass the combined loss with fewer views?\n\nOn one hand, increasing the number of views certainly increases the chance of producing more diverse features or predictions, but on the other hand, having more views on the same data also leads to a higher probability of consistent predictions from multiple views. It would be good to see how the number of views plays in the game.\n\n2. The fundamental grounding of multi-view learning is that, the utility of all views is maximised if, given the label, all views are independent of each other. \n\nThe paper made an argument in the intro that negative correlations between or among individual views would be helpful, which doesn't make sense. Referred literature in this paper talked about decreasing correlations by including a penalty term, but they didn't mean that we would like to see negative correlations.\n\n3. In ensemble learning, at least in bagging, the improvement over individual learners is often more significant when these learners are more diverse (have larger variance), but again, the baseline model here is a trained base learner. It is highly likely that an ensemble of model A would still be worse than a single model B. There is always a tradeoff between the diversity (variance) of the base learner and its accuracy. I hope the authors could be more precise on their wording in the intro. \n\n4. I am not super positive about the derivations in Section 3. Let me elaborate a bit.\n\nProviding that $$f_m \\sim \\mathcal{N}(f_c, \\sigma_m^2), \\forall m=1,...,M$$ with the distribution of f_c unknown.\n\nThe joint distribution should be $$p(f_c, f_1, ..., f_M)=p(f_1, ..., f_M|f_c)p(f_c)=p(f_c)\\Pi_{m=1}^Mp(f_m|f_c)=p(f_c)\\Pi_{m=1}^M\\mathcal{N}(f_c, \\sigma_n^2)$$, which is different from Eq. 2.\n\nThe marginal distribution should be $$p( f_1, ..., f_M)=\\int p(f_1, ..., f_M|f_c)p(f_c) df_c=\\int p(f_c)\\Pi_{m=1}^M\\mathcal{N}(f_c, \\sigma_m^2)df_c$$, which doesn't give us Eq. 3 since we don't know  the distribution of f_c.\n\nLet's move on to the posterior distribution of the consensus function: \n\n$$p(f_c | f_1, …, f_M) = p(f_c, f_1, …, f_M) / p(f_1, …, f_M)= p(f_c) p(f_1, …, f_M | f_c) / \\int p(f_c) p(f_1, …, f_M | f_c) df_c$$\n\nwhich may not be a gaussian distribution. \n\nA quick fix is to also assume that p(f_c) is a gaussian distribution, although this assumption seems to be very strong. ", "summary_of_the_paper": "the submission proposed a semi-supervised learning framework that leverages the benefits of multi-view learning with neural networks. On labelled data, besides minimising the empirical risk, the objective function encourages diversity by driving the correlation between individual functions with the mean function; on the unlabelled data, the objective forces multiple views to produce consistent features or predictions. ", "summary_of_the_review": "Overall, this submission's empirical contribution is more appreciated than its unclear theoretical justification. However, the arguments made in the intro were too vague and also misleading. Therefore, I do not vote for accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6mygeKE0qGq", "reviewer": "Reviewer_C8H3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nThis paper studied the semi-supervised regression problem and tackled it with a technically sound approach.\nIt sheds light on the effects of diversity and consistency in multi-view learning.\nBesides, this paper is mostly clear written.\nIt is easy to follow the derivation and verify its correctness.\n\n---\n\n## Weaknesses\n\nMy biggest concern is its novelty.\nAs cited in the paper, most of the probabilistic graphical model definition, marginalization, and likelihood derivation is quite similar to those used in [Yu et al., 2011].\nAlthough [Yu et al., 2011] used the Gaussian process while this work mainly used neural network and gradient-based optimization, the techniques for deriving the learning objective are basically the same.\n\nConcretely, if I didn't misunderstand anything, the definition of the undirected graphical model is roughly as follows:\n\n- $p(y, f_c, f_1, \\dots, f_M, x) = p(y, f_c) \\prod_{m=1}^M p(f_c, f_m) p(f_m, x)$\n\n- $p(f_m, x) \\propto F_m(\\eta_m(x), \\theta_m)$ (multi-view neural networks)\n\n- $p(f_c, f_m) \\propto \\exp\\left(-\\frac{(f_c - f_m)^2}{2\\sigma^2_m}\\right)$ (consensus potential, \"within-view potential\" $p(f_m)$ in [Yu et al., 2011] does not appear here)\n\n- $p(y, f_c) \\propto \\exp\\left(-\\frac{(y - f_c)^2}{2\\sigma^2}\\right)$ (output potential, Gaussian noise model for regression)\n\nThen, both [Yu et al., 2011] and this work marginalize the potential function to obtain:\n\n- $p(f_1, \\dots, f_M) \\propto \\int \\prod_{m=1}^M p(f_c, f_m) \\,\\mathrm{d} f_c$ (similar to [Yu et al., 2011] Eq. (7) in Section 3.1 (Marginal 1) and Appendix A.1 (derivation) without the within-view potential)\n\nAnd the conditional probability is calculated via Bayes' rule:\n\n- $p(f_c | f_1, \\dots, f_M) = \\frac{p(f_c, f_1, \\dots, f_M)}{p(f_1, \\dots, f_M)} \\propto \\exp\\left(-\\sum_{m=1}^M \\frac{(f_c - f_m)^2}{2\\sigma^2_m}  + \\sum_{m<k} \\lambda_{m,k} (f_m - f_k)^2\\right)$\n\nIt seems that [Yu et al., 2011] can be used for semi-supervised multi-view regression as well as classification.\nI'd like to know the main challenge and unique contribution of this work compared with [Yu et al., 2011].\n\n---\n\n## Questions and comments\n\n### Regression\n\n- I might have missed something, but is there anything \"regression-specific\" in this work?\n  If we just change the form of likelihood, can we apply the same technique to classification as well?\n\n### Data augmentation\n\n- How do you perform data augmentation on tabular data?\n- How do you perform data augmentation after the backbone network in DiCoM-B?\n- The data augmentation seems non-differentiable, so the backbone is pre-trained and fixed?\n\n### Graphical model notation\n\n- Figure 2 does not seem to be standard probabilistic graphical model notation so it's a little confusing.\n  For example, edges should not be labeled and $\\sigma^2$ and $\\lambda$ should be fixed parameters.\n\n### Others\n\n- Eq. (1): Shouldn't it be $\\sigma^2_m$? The assumption of $\\sigma^2_m = \\sigma^2_v$ is not given here.\n- Eq. (2): $p(f_m, f_c)$ is not defined nor explained.\n\n---\n\n## References\n\n- Yu, Shipeng, et al. \"Bayesian co-training.\" The Journal of Machine Learning Research 12 (2011): 2649-2680.", "summary_of_the_paper": "This paper proposed a method for **semi-supervised multi-view regression** based on **data augmentation** and an **undirected graphical model**.\nThe author derived a \"**consistency**\" term for unlabeled data and a \"**diversity**\" term for labeled data based on their log-likelihood and combined them linearly with two hyperparameters.\nThe proposed method was evaluated on tabular and image data and analyzed via ablation study.", "summary_of_the_review": "This paper proposed a technically sound method based on an undirected graphical model for semi-supervised regression.\nHowever, compared to an existing formulation [Yu et al., 2011], its novelty and contribution are a little questionable and need to be further highlighted.\nHence, I think it's on the borderline for now.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-HTTkYf-7QV", "reviewer": "Reviewer_BTsB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I want to know how many branches is good and why don't you use multiple heads? e.g., GAT uses multi-head.\n2. In multiview learning, I want to know how to model diversity? The consistency is easy to model, but the diversity is not well addressed. Even CCA can be regarded as a view-consistency modeling.\n3. I want to know why you design the model? for what applications?", "summary_of_the_paper": "A view-consistent loss is designed, which is based on PGM. The PGM is quite simple to understand.", "summary_of_the_review": "The view-consistency modeling does not give a supervise or somehow it is not new enough. \nThe diversity loss is not well addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposed multi-view learning framework that combines diversity and consistency objectives for semi-supervised learning. While reviewers appreciated that simplicity of the proposed method, they raised concerns on the limited contribution on top of the original Bayesian Co-Training work. Although authors provided detailed rebuttals that addressed some of the reviewers' concerns, and one reviewer did raise their score, the other reviewers' scores remained unchanged. Given the work is closely based off the BCT work, I would like to see more detailed analyses on the importance of the changes brought in this work, such as changing the base learners and introduction of diversity objectives as pointed out by the authors.
Decision: Reject