Year: 2021

Abstract:
Label collection is costly in many applications, which poses the need for labelefficient learning. In this work, we present Diverse and Consistent Multi-view Networks (DiCoM) – a novel semi-supervised regression technique based on a multi-view learning framework. DiCoM combines diversity with consistency – two seemingly opposing yet complementary principles of multi-view learning based on underlying probabilistic graphical assumptions. Given multiple deep views of the same input, DiCoM encourages a negative correlation among the views’ predictions on labeled data, while simultaneously enforces their agreement on unlabeled data. DiCoM can utilize either multi-network or multi-branch architectures to make a trade-off between computational cost and modeling performance. Under realistic evaluation setups, DiCoM outperforms competing methods on tabular and image data. Our ablation studies confirm the importance of having both consistency and diversity.

heading: 1 INTRODUCTION
text: Deep neural networks have achieved tremendous success across several domains, ranging from computer vision, natural language processing, to audio analysis (LeCun et al., 2015). However, to train neural networks that perform well typically requires a large amount of labeled data. In many cases, this requirement for a large labeled dataset presents a challenge, because the annotation process can be labour-intensive and thus expensive, especially when specialized expertise is required. To address this challenge, semi-supervised learning methods (Van Engelen & Hoos, 2020) that can achieve similarly high performance with less labeled data by using unlabeled data have been developed.
We focus on semi-supervised learning in the regression setting. There are several approaches for semi-supervised regression, including graph-based methods (Zhur & Ghahramanirh, 2002), cotraining (Blum & Mitchell, 1998) and entropy minimization (Jean et al., 2018). Consistency-based approaches that have been popular in the classification setting, such as Mean Teacher (Tarvainen & Valpola, 2017) and Virtual Adversarial Training (Miyato et al., 2018), which reinforce the output consistency of the network under input perturbations, have also been adapted to the regression setting (Jean et al., 2018). However, enforcing consistency alone may not be sufficient for good performance, and may lead to model collapsing (Qiao et al., 2018) or confirmation bias issues (Ke et al., 2019).
To address these issues, we draw inspiration from ensemble learning with neural networks for regression. A necessary and sufficient condition for an ensemble of learners to be more accurate than any of its individual members is if the base learners are accurate and diverse (Dietterich, 2000). Therefore, the key component that can make or break an ensemble is the diversity (or disagreement) among its individual regressors. If this diversity is insufficient, the ensembling may not result in better performance. On the other hand, overemphasizing diversity can degrade the learnability of the ensemble members. So far, the most successful mechanism to leverage ensemble diversity in regression is Negative Correlation Learning (Liu & Yao, 1999; Zhang et al., 2019).
In this work, we propose Diverse and Consistent Multi-view Networks for Semi-supervised Regression (DiCoM) that elegantly unifies consistency and diversity in a multi-view learning framework. Based on probabilistic graphical assumptions, we derive a loss function that integrates both consistency and diversity components – diversity is encouraged on labeled data, while consistency is enforced on unlabeled data. Furthermore, we develop two variants of DiCoM: the first uses multiple
networks to achieve better performance, while the second employs a single network with multiple branches to help with scalability. We compare DiCoM against state-of-the-art methods on eight tabular datasets and a crowd-counting dataset, where we show that DiCoM outperforms existing methods. We further perform ablation studies to analyze the importance of diversity and consistency, and the effect of varying the number of views in the model.
While other works have leveraged related ideas of complementary and consensus in multi-view classification (Xu et al., 2013); or explored commonality and individuality in multi-modal curriculum learning (Gong, 2017), these methods were developed for classification or clustering tasks, and cannot be easily modified to suit semi-supervised regression.
In summary, the major contributions of this work are as follows:
• We derive a novel objective function from a probabilistic graphical perspective. Our objective function unifies multi-view diversity and consistency and provides theoretical insights into the relationship between diversity-consistency trade-off and the number of views.
• We show the high flexibility of DiCoM, which can be adaptively scaled up to larger number of views while maintaining competitive performance.
• We demonstrate the performance of DiCoM on both tabular and visual types of input data, where it outperforms competing methods. Our ablation studies validate the importance of having both diversity and consistency.

heading: 2 RELATED WORK
text: Semi-supervised Regression: Semi-supervised learning is a data-efficient learning paradigm that offers the ability to learn from unlabeled data. In recent years, much work has focused on semisupervised classification, and there have been far fewer studies on semi-supervised regression. For regression tasks, graph-based methods are among the first to be developed. One example is Label Propagation (LP) (Zhur & Ghahramanirh, 2002) which defines a graph of training data and propagates ground-truth labels through high density regions of the graph. Kernel methods have also been proposed, such as Semi-supervised Deep Kernel Learning (SSDKL) (Jean et al., 2018). This method minimizes the predictive variance in a posterior regularization framework to learn a more generalizable feature embedding on unlabeled data. Co-training regressors (COREG) (Zhou & Li, 2005) employs k-Nearest neighbor regressors, each of which generate pseudo-labels for the other during training; this helps to maximize their agreement on unlabeled data.
Apart from the aforementioned approaches, consistency-based methods are also gaining traction. Mean Teacher (MT) (Tarvainen & Valpola, 2017) enforces posterior consistency between two neural networks, a student and a teacher, the latter being an exponential moving average of the former in the parameter space. An orthogonal approach is to enforce consistency on adversarially augmented input, as implemented in Virtual Adversarial Training (VAT) (Miyato et al., 2018). These methods were originally developed for classification, and were subsequently adapted to regression tasks (Jean et al., 2018). However, both MT and VAT maintain only a single trainable network, which may lead to problems such as confirmation bias (Ke et al., 2019) and overly-sensitive hyperparameters. In this paper, we show that consistency-based methods can be further improved with ensemble diversity.
Ensemble Diversity: Ensembles of neural networks have been extensively studied and widely used in many applications. Their effectiveness largely depends on the level of diversity (or disagreement) among members of the ensemble. It is well-understood that a good ensemble must manage the tradeoff between the accuracy of the individual learners and the diversity among them (Brown et al., 2005; Tang et al., 2006). For regression tasks, a commonly-used ensemble technique is Negative Correlation Learning (NCL) (Liu & Yao, 1999; Liu et al., 2000), which formulates a diversity-promoting loss using an ambiguity decomposition of the squared ensemble loss (Krogh et al., 1995). In this formulation, a correlation penalty term (also refered to as an ambiguity term) measures how much each member’s prediction deviates from the ensemble output. When this penalty term is maximized, the errors of individual learners become negatively correlated. It was theoretically proven (Brown et al., 2005) that the strategy employed by NCL is equivalent to leveraging a bias-variance-covariance trade-off (Ueda & Nakano, 1996) of the ensemble error.
Recently, NCL has been extended to semi-supervised learning (Chen et al., 2018), where the correlation penalty term is extended to the unlabeled data. However, this method was demonstrated only on tabular data. Another variant of NCL is Deep Negative Correlation Learning (DNCL) (Shi et al., 2018; Zhang et al., 2019), which is designed for visual regression tasks in a purely supervised learning setting.
Multi-view Learning: A dataset is considered as having multiple views when its data samples are represented by more than one feature set, each of which is sufficient for the learning task. Although each view is supposed to be sufficient for learning the task, a model trained on only one single view often faces the risk of overfitting, especially when labeled data is limited (Xu et al., 2013). To address this problem, multi-view learning assigns a modeling function to each view and jointly optimize these functions to improve overall generalization performance (Zhao et al., 2017). By analyzing the development of various techniques, Xu et al. (Xu et al., 2013) summarized two significant principles that underpin multi-view learning: consensus and complementary. The consensus principle states that a multi-view technique must aim at maximizing the agreement on different views. This is similar to how consistency-based semi-supervised learning methods works: for instance, MT enforces agreement with its past self. The complementary principle states that in order to make improvement, each view must contain some information that the other views do not carry. In other words, the views should be sufficiently diverse. This is related to diversity regularization in ensemble learning, where individual learners are encouraged to give diverse predictions. Thus, multi-view learning offers a unifying perspective of both consistency and diversity.

heading: 3 PROPOSED METHOD
text: We start by describing how multiple deep views can be generated from input data. Then, we propose our multi-view learning framework for regression, in which multiple deep views can be simultaneously optimized via backpropagation. We then discuss the graphical models that govern the probabilistic dependencies among the ground-truth label and the deep views. Finally, we derive the DiCoM loss function using these graphical models and provide a few insights.
View creation: Consider a regression task where the goal is to estimate a label y ∈ R from an input x. To create multiple views, our first approach is to use M neural networks F1,F2, . . . ,FM , each parameterized by θ1, θ2, . . . , θM , respectively. By applying different data augmentations η1, η2, . . . , ηM on the original x, we generate M different augmented inputs xm = ηm(x) ∀m= 1, . . . ,M . With each augmented input, the corresponding neural network produces a regression output fm(x) = Fm(xm, θm) ∀ m = 1, . . . ,M . Due to the different augmentations and network parameters, each output fm can be treated as one deep view of the original input x. We call this multi-network setup DiCoM-N, where the N stands for ‘network’ (see Fig. 1 (a)).
The second approach is to utilize a single network with a shared backbone B and multiple branches F1,F2, . . . ,FM . We use θB to denote the learnable parameters from the backbone and θ1, θ2, . . . , θM to denote the parameters of the branches. The hidden features generated by the backbone serve as input to the branches. While the backbone still applies a random augmentation to the input x, each branch Fm applies its own random augmentation ηm as well. Thus, regression outputs f1, f2, . . . , fM from the branches can be considered as deep views of the original input. We name this setup DiCoM-B, where the B is short for ‘branch’ (see Fig. 1 (b)). Multi-branch technique was widely adopted for supervised classification (Xie et al., 2017). In this work, it allows us to harness the power of multi-view learning with a relatively lower number of trainable parameters.
Multi-view learning framework for regression: Regardless of how they are generated, the deep views are used together with the true label y to compute a semi-supervised loss function LDiCoM. During training phase, LDiCoM is back-propagated simultaneously through the deep views to optimize network parameters θ1, θ2, . . . , θM (including θB in the case of DiCoM-B). During inference, all augmentations are removed so that the forward pass is applied on the raw input x. The final prediction is computed as the average of all deep views: µ(x) = ∑M i=1 1 M fm(x). The general DiCoM framework is illustrated in Fig. 1. In the next step, we derive LDiCoM based on a probabilistic graphical assumption.
Probabilistic graphical models: Since the augmented inputs are generated from the same sample, the deep views should be close to each other. Motivated by previous work in kernel learning (Yu
et al., 2011) and linear regression (Nguyen et al., 2019), we consider f1, f2, . . . , fM as random variables and introduce a consensus function fc as a latent variable that connects to each of the deep views. This function enforces the mutual agreement among the views. We assume that the difference between the consensus function and each view follows a zero-mean Gaussian distribution
fc − fm ∝ N ( 0, σ2m ) ∀m = 1, . . . ,M. (1)
This probabilistic relation is known as the consensus potential (Yu et al., 2011). Considering the whole graph, this potential implies that all views are random Gaussian variables with a shared mean fc and variance σ2m. As a result, the views stay consistent w.r.t. each other by taking values not too far away from the shared consensus. This graphical model, shown in Fig. 3 (a), is assumed for each unlabeled sample. The joint density associated with the graph is given by
p (fc, f1, . . . , fM ) = 1
Z M∏ m=1 Ψ(fc, fm) (2)
where Z is a normalizing constant and Ψ(fc, fm) = exp [ − (fc−fm) 2
2σ2m
] is the potential function of
the edge connecting fc and fm. From this model, we derive two important results. On a side note,
our derivation generalizes to vector-valued labels y, but here we assume scalar labels for ease of exposition. The proofs of our results are provided in Appendix A.
(I) Marginalization of the views: By integrating the latent consensus function fc out of the joint density, the marginal distribution of the views is
p (f1, . . . , fM ) ∝ exp [ M∑ m=1 ∑ k>m −λm,k (fm − fk)2 ]
(3)
where λm,k = [ 2σ2mσ 2 k (∑ m 1 σ2m )]−1 . This result implies that the marginal likelihood can be fac-
torized as a product of ( M 2 ) terms. Each term is an isotropic Gaussian distribution on the difference between a pair of views (fm, fk), with zero mean and variance (2λm,k) −1. The equivalent graphical model is shown in Fig. 3(b).
(II) Conditional of the consensus function: By applying Bayes’ theorem, the conditional distribution of the consensus function fc given all the views is a Gaussian
fc|f1, . . . , fM ∼ N ( µ̃, σ2µ ) . (4)
where σ2µ = (∑
m 1 σ2m
)−1 and µ̃ = σ2µ ∑ m fm σ2m
. This result highlights that the conditional distribution of fc depends only on the weighted average µ̃, and the values of individual views are not required. Furthermore, µ̃ can be treated as a view itself, with a variance that is smaller than any of the variances of the views.
Derivation of DiCoM loss function: For simplicity, we assume equal variance for different deep views, i.e., σ2m = σ 2 v ∀m. For an unlabeled sample (xn), we directly apply the first result (I) to obtain the following negative log likelihood function
Lunl = M∑ m=1 ∑ k>m 1 2Mσ2v [fm(xn)− fk(xn)]2 (5)
For a labeled sample (xn, yn), since the ground-truth is given, we assume a graphical model that involves the final DiCoM prediction, i.e., the averaged output µ. This graph is shown in Fig. 3(c). Since we assume a shared variance σ2v, the weighted output now reduces to an equal-weight average, following from result (II)
µ̃(xn) = M∑ m=1 fm(xn) M = µ(xn) σ 2 µ = σ2v M
(6)
Subsequently, we apply result (I) on this graph to get the negative log likelihood as follows
Llab = M
2Mσ2y + 2σ 2 v
[yn − µ(xn)]2 (7)
= 1
2Mσ2y + 2σ 2 v M∑ m=1 { [fm(xn)− yn]2 − [fm(xn)− µ(xn)]2 } (8)
≈ 1 2σ2v M∑ m=1 { [fm(xn)− yn]2 − [fm(xn)− µ(xn)]2 } (9)
where in equation (8), we have applied the ambiguity decomposition (Krogh et al., 1995) and in equation (9), we have assumed that the label is accurate, i.e., σ2y σ2v.
Given a training batch of labeled samples {(xn, yn)}Ln=1 and unlabeled samples {(xn)}Un=1, assuming that the samples are independently generated, we can add the log-likelihood functions across all training samples. This can be done via simply adding up two equations (5) and (9)
LDiCoM = 1
L L∑ n=1 M∑ m=1 { [fm(xn)− yn]2 − κdiv [fm(xn)− µ(xn)]2 } + 1
U U∑ n=1 M∑ m=1 ∑ k>m κcsc [fm(xn)− fk(xn)]2 (10)
where we introduce two hyperparameters κdiv and κcsc to absorb other constants and to enable a trade-off between diversity and consistency components of the loss.
The DiCoM loss encourages diversity on labeled data, while enforcing consistency on unlabeled data. These two seemingly opposing components can both be derived from the same underlying graphical assumptions. Furthermore, they should not be weighted equally. In fact, we have shown that it depends on the number of views: when M increases, diversity grows in O(M), while consistency grows in O(M2). It is worth noting that our method is fundamentally different from other extensions of NCL such as Semi-supervised NCL (Jean et al., 2018), which enforces diversity on both labeled and unlabeled data. Last but not least, since both diversity and consistency are incorporated in the DiCoM objective function, the method is highly adaptable to different implementations such as multi-network or multi-branch, as long as the views are provided.

heading: 4 EXPERIMENTS
text: In this section, we study the proposed method in different settings, including regression tasks on eight tabular datasets and a crowd counting task on image data. We provide additional experiment results in Appendix C and an additional experiment on toy data in Appendix D.

heading: 4.1 EXPERIMENTS ON UCI TABULAR DATA
text: Datasets: We evaluate DiCoM on eight datasets from the UCI repository (Dua & Graff, 2017)1: skillcraft, parkinsons, elevators, protein, blog, ctslice, buzz, and electric. These datasets are collected from real-world regression scenarios, with varying sample sizes and input dimensions. For each dataset, we keep 1000 labeled samples as a hold-out test set; further retain N = 300 samples for the labeled training set, and keep the rest as the unlabeled training set. We follow the realistic evaluation setup in (Oliver et al., 2018) and use a 90%-10% train-validation split, i.e., 270 samples are used for training, leaving only 30 for validation.
Experiment Setup: We implement both variants of DiCoM. Our DiCoM-N networks adopt the same architecture as (Jean et al., 2018; Wilson et al., 2016), which is a fully-connected multilayer perceptron with four hidden layers, containing 100, 50, 50 and 2 hidden nodes, respectively. Our DiCoM-B also utilizes this architecture, but branch out after the third hidden layer, i.e., the backbone contains hidden layers of 100, 50 and 50 nodes, while the branches each contains one hidden layers of 2 nodes. This model is trained end-to-end, the backbone is trained together with the branches. DiCoM hyperparameters (κdiv, κcsc) are chosen from a grid of values based on validation errors. Across 10 random seeds, we report the root-mean-squared errors (RMSE) statistics on the test set. For simplicity, we append ‘-M ’ to the end of our method name to denote the number of views, e.g., DiCoM-B-4 represents the multi-branch DiCoM network with 4 branches.
Data Augmentation: we apply zero-mean Gaussian noise which is commonly used for tabular data. For the DiCoM-B model, Gaussian noise is applied on the input and on the features at the beginning of each branch, right after branching out. Since the independent Gaussian noise is added during the forward pass, it does not affect the gradient values during backpropagation. The exact amount of Gaussian noise can be found in Appendix B.
We compare DiCoM against six semi-supervised regression methods: SSDKL (Jean et al., 2018), COREG (Zhou & Li, 2005), LP (Zhur & Ghahramanirh, 2002), VAE (Jean et al., 2018), MT (Tarvainen & Valpola, 2017) and VAT (Miyato et al., 2018). These methods span a wide range of approaches such as consistency regularization (MT, VAT), entropy minimization (SSDKL), multiview learning (COREG), graph-based (LP), or generative modeling (VAE). The detailed information about the datasets and experiment setup can be found in Appendix B.
Results: Fig. 3 shows the experiment results. We observe significant improvements compared to the state-of-the-art semi-supervised regression methods. The largest performance gains are achieved on parkinsons and ctslice, where DiCoM-N-2 improves upon the best competing method by 32.4% and 42.5%, respectively. DiCoM-N-2 also outperforms DiCoM-B-2 on all datasets, except for protein. This is expected since DiCoM-N-2 has almost twice as many learnable parameters as DiCoM-B-2. For elevators, the most frequently selected hyperparameters across 10 ran-
1https://archive.ics.uci.edu/ml/index.php
dom seeds are (κdiv, κcsc) = (1, 0.01), while for ctslice, the most frequently selected values are (κdiv, κcsc) = (0.1, 1). This shows that different datasets require different trade-offs between consistency and diversity. We also notice that LP (a graph-based method) and COREG (a nearestneighbors-based method) performs relatively well on blog, ctslice and buzz. Meanwhile, MT and VAT, which are based on consistency regularization (without diversity regularization), did not perform well on these regression datasets, even though they have been shown to be effective on classification tasks.
Ablation study on the components: We analyse the effect of different components of the DiCoMN-2 model by individually removing them from the model. For the first model, Ablation-1, we remove data augmentation. In Ablation-2, we remove the diversity loss on the unlabeled data. Next, the consistency loss on labeled data is set to zero for Ablation-3 model. Lastly, we apply diversity loss to both labeled and unlabeled training data for model Ablation-4. Using the results of DiCoMN-2 as the baseline, we also report the percentage reduction in test RMSE (% Redc.) for the other methods as follows:
Percentage Reduction = (baseline score− new score)× 100
baseline score (11)
Table 1 reports the ablation results. The average percentage reduction scores tell us the importance of each component. Augmentation has a small impact on the performance of DiCoM-N-2, while diversity and consistency regularization are both important. DiCoM-N-2 outperforms Ablation-4 in all cases, which suggests that a mere reliance on diversity is insufficient.
Varying the number of views: We further evaluate the impact of the number of views M . Fig. 4 shows the performance of three DiCoM-N models with increasing number of views M ∈ {2, 4, 8}. The results show that a larger value of M leads to an improvement in the performance. When M increases from 4 to 8, the average reduction in test RMSE is 3.48%, larger than the average reduction
rate of 1.70% when M increases from 2 to 4. While varying the number of views, we also monitor the changes in the model hyperparameters. Using the values that were selected to minimize the validation error of DiCoM-N, Table 2 shows the log ratio of log10(κdiv/κcsc). For most datasets, we see that this log ratio tends to increase for larger number of views. This is because in LDiCoM, the number of diversity terms grows in O(M) while the number of consistency terms grows in O(M2). Thus, as M increases, a larger (κdiv/κcsc) ratio is required to keep those terms balanced.
Comparing multi-network and multi-branch: In this experiment, we compare the two variants of DiCoM against each other, by reporting both their performance and execution time (adding train and test time). Table 3 shows the percentage reduction computed using equation (11) by treating DiCoM-N’s results as the baseline scores and DiCoM-B’s corresponding results as the new scores. It can be seen that DiCoM-N is consistently outperforming DiCoM-B in terms of test RMSE. The multi-network variant is also the faster option when M = 2. However, as the number of views increases, the execution time of DiCoM-B is significantly faster. This shows that while DiCoM-N achieves better test performance, DiCoM-B demonstrates better scalability.

heading: 4.2 EXPERIMENTS ON VISION DATA
text: In order to show the versatility of DiCoM, we further conduct experiments on a crowd counting task. Crowd counting is a fundamental question in the vision community due to its far-reaching applications in many scenarios, including video surveillance, metropolis security, human behavior analysis.
Crowd counting has been recently used as a benchmark for deep regression algorithms (Zhang et al., 2019); for this task, counting by regression has been perceived as the state-of-the-art approach.
Dataset: We study the ShanghaiTech Part-A dataset (Zhang et al., 2016)2. This is a new largescale crowd counting dataset that contains extremely congested scenes, with varying perspective and unfixed resolution. The data are split into 300 training and 182 test samples. Among the 300 training samples, we randomly select N ∈ {30, 120, 210} samples as the labeled set and use the remaining data as the unlabeled set. We follow the common practice to report both mean absolute errors (MAE) and root-mean-squared errors (RMSE) on the test set. We note that this dataset inevitably contains personally identifiable information, which has been made public by the owner of the dataset.
Experiment Setup: In our experiments, we adopt the network architecture of CSRNet B (Li et al., 2018) and implement DiCoM-B-4. More specifically, we use a pre-trained VGG16 network as the encoder and append another decoder on top of it. In the penultimate layer of the decoder, we enlarge the number of hidden channels by M times. We then apply a group-convolutional layer as the last layer, setting both the number of output channels and group size to M . Thus, the backbone of DiCoM-B-4 includes the pre-trained VGG16 and the decoder up to its penultimate layer. This architecture has been shown to be very effective in generating multiple predictions without much increase in computational cost over a single network (Shi et al., 2018; Zhang et al., 2019; Zhou et al., 2021).
We compare DiCoM with the following competing methods: (i) the supervised baseline which uses only the labeled training samples and standard MSE loss; (ii) the DNCL model (Zhang et al., 2019); and (iii) the Co-Regression model. Since we are not running for multiple random seeds, we remove all random data augmentations (e.g., cropping, flipping) to enable fair comparison between different methods.
Results: From the results in Table 4, we observe that both DNCL and Co-Regression outperform the supervised baseline by enforcing either diversity on labeled data or consistency on unlabeled data, and that overall, DiCoM-B-4 outperforms other methods by incorporating both diversity and consistency on the unlabeled data.

heading: 5 CONCLUSION
text: In this paper, we proposed novel Diverse and Consistent Multi-view Networks for Semi-supervised Regression (DiCoM), that elegantly combines ensemble diversity with consistency regularization. DiCoM utilizes probabilistic graphical models to control the underlying dependencies among multiple regression outputs and label. We also show that DiCoM is highly flexible, it can be adopted for multi-network or multi-branch implementations, the latter significantly improves the scalability of the method. Experiments on tabular UCI and visual ShanghaiTech datasets demonstrated the effectiveness of the proposed method across diverse domains, while ablation studies validated the importance of both consistency and diversity. In the future, one may extend the DiCoM framework by introducing asymmetric views (of non-identical architectures), which naturally causes the final output µ to be an unequally-weighted average. Another interesting direction is to explore the potential impact of data augmentation techniques, since multi-view learning often benefits from diversified inputs.
2https://github.com/desenzhou/ShanghaiTechDataset (BSD 2-clause license)

heading: A MATHEMATICAL PROOFS
text: Consider again the general model, where there are M deep views, i.e., {fm}Mm=1. Graphically, these functions are represented by nodes that are connected not directly, but only via the consensus function fc using isotropic Gaussian potentials
fc − fm ∝ N ( 0, σ2mI ) . (12)
We note that in this general case, each deep view is a vector (instead of a scalar) and is assigned a separate variance σ2m, which are not necessarily equal to each other. In terms of notation, we use italic letters for scalar variables and boldface letters for vectors and matrices.
A.1 MARGINAL DENSITY OF THE VIEWS
In this proof, we derive the marginal distribution of the views. Given the DiCoM graphical model, it is necessary to integrate fc out of the joint density distribution of the graph, because fc is a latent variable. The joint density distribution function of this graphical model is as follows
p (fc, f1, . . . , fM ) = 1
Z1 M∏ m=1 exp ( −‖fc − fm‖ 2 2σ2m ) (13)
= 1
Z1 exp ( − ∑ m f>c fc 2σ2m + ∑ m f>c fm σ2m − ∑ m f>mfm 2σ2m ) (14)
= 1
Z1 exp ( −ψ 2 f>c fc + φ >fc + χ ) , (15)
where the normalizing factor Z1 is a constant w.r.t. fc, f1, . . . , fM and
ψ = M∑ m=1 1 σ2m φ = M∑ m=1 fm σ2m
χ = M∑ m=1 − f > mfm 2σ2m . (16)
Notice that ψ,φ, χ are constants w.r.t. fc. By applying the following integration rule for a multivariate Gaussian variable x Petersen et al. (2008)∫
exp ( −1 2 x>Ax + c>x ) dx = √ det (2πA−1) exp ( 1 2 c>A−>c ) , (17)
we can integrate fc out of the joint distribution in (15) to obtain the following marginal likelihood
p (f1, . . . , fM ) = ∫ p (fc, f1, . . . , fM ) dfc (18)
= 1
Z2 exp
( φ>φ
2ψ + χ
) (19)
= 1
Z2 exp
[ 1
2ψ (∑ m f>mfm σ4m + 2 ∑ m ∑ k>m f>mfk σ2mσ 2 k − ψ ∑ m f>mfm σ2m )] (20)
= 1
Z2 exp
[ 1
2ψ (∑ m ∑ k>m − f > mfm − 2f>mfk + f>k fk σ2mσ 2 k )] (21)
= 1
Z2 exp (∑ m ∑ k>m −‖fm − fk‖2 2ρ2m,k ) (22)
= 1
Z2 exp (∑ m ∑ k>m −λm,k‖fm − fk‖2 ) , (23)
where Z2 is another constant w.r.t. fc, f1, . . . , fM , and
ρm,k = σ 2 mσ 2 kψ = σ 2 mσ 2 k ( M∑ i=1 1 σ2i ) (24)
λm,k = (2ρm,k) −1 = [ 2σ2mσ 2 k ( M∑ i=1 1 σ2i )]−1 . (25)
A.2 CONDITIONAL DENSITY OF THE CONSENSUS FUNCTION
In this proof, we derive the conditional density distribution of the consensus function fc given the views. Consider again the general model withM views, i.e., {fm}Mm=1. Each view is represented by a random variable connected only to the consensus function fc via an isotropic Gaussian potential as defined in (12). From (13), (22), the conditional distribution of fc given the views f1, . . . , fM is
p (fc|f1, . . . , fM ) = p (fc, f1, . . . fM )
p (f1, . . . , fM ) (26)
= 1
Z3 exp ( M∑ m=1 −‖fc − fm‖2 2σ2m + M∑ m=1 M∑ k>m ‖fm − fk‖2 2ρ2m,k ) (27)
= 1
Z3 exp (∑ m −f>c fc + 2f>mfc − f>mfm 2σ2m + ∑ m ∑ k>m ‖fm − fk‖2 2ρ2m,k ) (28)
= 1
Z3 exp ( −f>c fc 2σ2µ + µ̃>fc σ2µ + ℵ ) , (29)
where the normalizing factor Z3 is a constant w.r.t. fc, f1, . . . , fM and
σ2µ = ( M∑ m=1 1 σ2m )−1 (30)
µ̃ = σ2µ M∑ m=1 fm σ2m
(31)
ℵ = M∑ m=1 −f>mfm 2σ2m + M∑ m=1 M∑ k>m ‖fm − fk‖2 2ρ2m,k . (32)
Using the definitions (24), (30) and (31), ℵ can be rewritten as follows
ℵ = σ2µ (∑ m −f>mfm 2σ2mσ 2 µ + ∑ m ∑ k>m ‖fm − fk‖2 2σ2mσ 2 k ) (33)
= σ2µ [(∑ m −f>mfm 2σ2m )(∑ k 1 σ2k ) + ∑ m ∑ k>m ‖fm − fk‖2 2σ2mσ 2 k ] (34)
= σ2µ ∑ m ∑ k −f>mfm 2σ2mσ 2 k + ∑ m ∑ k 6=m f>mfm 2σ2mσ 2 k + ∑ m ∑ k>m −f>mfk σ2mσ 2 k  (35) = σ2µ (∑ m −f>mfm 2σ4m + ∑ m ∑ k>m −f>mfk σ2mσ 2 k ) (36)
= − σ2µ 2 (∑ m f>m σ2m )(∑ m fm σ2m ) (37) = −µ̃>µ̃ 2σ2µ . (38)
Thus, we can rewrite (29) in its Gaussian form
p (fc|f1, . . . , fM ) = 1
Z3 exp ( −f>c fc 2σ2µ + µ̃>fc σ2µ − µ̃ >µ̃ 2σ2µ ) = 1 Z3 exp ( −‖fc − µ̃‖ 2 2σ2µ ) . (39)
Therefore, fc|f1, . . . , fM ∼ N ( µ̃, σ2µI ) . (40)

heading: B EXPERIMENT SETUP DETAILS
text: In this section, we provide the detailed setups for our benchmarking experiments. All experiments are run on NVIDIA GeForce GTX 1080Ti GPUs, using an Anaconda virtual environment installed with CUDA version 10.1, Python version 3.7.10 and Pytorch version 1.7.0.
Experiment setup for DiCoM in UCI experiments:
• Network architecture: for DiCoM-N, fully-connected multilayer perceptron with hidden layers of size [100, 50, 50, 2]. For DiCoM-B, the backbone includes the first 3 hidden layers of size [100, 50, 50] and the branches include one hidden layer of size 2. Note that we are not counting the input and output layers.
• Parallelization: for DiCoM-B, we use group convolution in order to back-propagate through all branches simultaneously.
• Random seeds: 20, 40, . . . , 200 (10 seeds in total). • Training: 2000 epochs with 250 epoch patience for early stopping (stop if no improvement
is observed on validation set for 250 consecutive epochs). • Optimizer: Stochastic Gradient Descent with momentum 0.95 and weight decay 10−9.
Learning rate is 10−4 for ctslice and is 10−3 for other datasets. • Augmentation: additive random Gaussian noise with mean 0 and standard deviation 0.05
for DiCoM-N and 0.01 for DiCoM-B. • Diversity hyperparameter search range: κdiv ∈ {0.01, 0.05, 0.1, 0.5, 1}. • Consistency hyperparameter search range: κcsc ∈ {0.01, 0.05, 0.1, 0.5, 1}.
Experiment setup for DiCoM in ShanghaiTech experiment:
• Network architecture: CSRNet B Li et al. (2018). • Random seeds: 9999 (only 1 seed). • Training: 1000 epochs with no early stopping. • Optimizer: Adam with weight decay 10−5 and learning rate 10−5. • Augmentation: None. • Diversity hyperparameter search range: κdiv ∈ {10−5, 10−4, 10−3}. • Consistency hyperparameter search range: κcsc ∈ {10−5, 10−4, 10−3}.

heading: C ADDITIONAL EXPERIMENT RESULTS ON UCI DATASETS
text: Information of UCI datasets: The detailed statistics of the eight UCI datasets are given in the table below.
Additional experiment results on UCI datasets with labeling budget N = 300: Please see the below additional results from Section 4.1.
Additional experiment results on UCI datasets with labeling budget N = 100: Using the same UCI datasets, we conduct experiments similar to the ones in Section 4.1 with a smaller labeling budget of N = 100 samples. The results are provided below.

heading: D ADDITIONAL EXPERIMENT ON TOY DATA
text: We conduct an experiment on a synthetic toy dataset to illustrate how multi-view diversity and consistency work together to affect the training and inference of DiCoM-N.
Dataset: We synthesize a regression dataset where inputs x ∈ R30 and labels y ∈ R2. The labels are related to the inputs by y = Ax+ , whereA is a fixed 2×30 coefficient matrix and ∼ N (0, 0.32I). Each coordinate of x is drawn from the standard normal distribution. We generate a training set of 100 labeled and 1000 unlabeled samples, and a hold-out test set of 1000 labeled samples.
Experiment Setup: Our DiCoM-N model has M = 5 views, each uses a simple neural network with a single hidden layer containing two hidden nodes. We train the model with SGD for 50 epochs with a learning rate of 5 × 10−2, then evaluate the mean-squared-error (MSE) of the model on the test set.
Results: We keep κdiv = 1 and vary κcsc on a log scale: κcsc ∈ {0.01, 0.1, 1}. Both quantitative and qualitative results are shown in Fig. 6. We plot the training losses on the top row and visually show the predictions of each network on eight random test samples. On the left scenario (Fig. 6(a)) when κdiv κcsc, the diversity loss dominates the consistency component. Even though the total loss converges on the training set, the individual views’ losses do not, resulting in their large bias on test samples. This can be an issue if the DiCoM-N model contains a smaller number of views. On the other hand, the consistency enforcement is too strong on the right scenario (Fig. 6(c)). The individual views and the averaged output seem to have all collapsed into a single point. Where the models collapse to is limited by the individual networks’ capacity and may not necessarily be the global optimum for the averaged output. Finally, in the middle scenario (Fig. 6(b)), the effects of diversity and consistency losses are balanced, yielding a good trade-off. The averaged model output is able to perform better than each individual view, and is also the best among three scenarios. These results also suggest that even though diversity and consistency are contradicting forces, they can still be applied simultaneously on the regression outputs to produce the desirable behaviours.

Review summaries:
This paper proposed a method for **semi-supervised multi-view regression** based on **data augmentation** and an **undirected graphical model**. The author derived a "**consistency**" term for unlabeled data and a "**diversity**" term for labeled data based on their log-likelihood and combined them linearly with two hyperparameters. The proposed method was evaluated on tabular and image data and analyzed via ablation study.

This paper concerns the semi-supervised problem. Different from SOTA deep semi-supervised methods, the proposed DiCom employs a diversity measure on the labeled multi-view data, and combines diversity with consistency based on underlying probabilistic graphical assumptions. Experiments verify the effectiveness.