Diverse and Consistent Multi-view Networks for Semi-supervised Regression

1 INTRODUCTION . Deep neural networks have achieved tremendous success across several domains , ranging from computer vision , natural language processing , to audio analysis ( LeCun et al. , 2015 ) . However , to train neural networks that perform well typically requires a large amount of labeled data . In many cases , this requirement for a large labeled dataset presents a challenge , because the annotation process can be labour-intensive and thus expensive , especially when specialized expertise is required . To address this challenge , semi-supervised learning methods ( Van Engelen & Hoos , 2020 ) that can achieve similarly high performance with less labeled data by using unlabeled data have been developed . We focus on semi-supervised learning in the regression setting . There are several approaches for semi-supervised regression , including graph-based methods ( Zhur & Ghahramanirh , 2002 ) , cotraining ( Blum & Mitchell , 1998 ) and entropy minimization ( Jean et al. , 2018 ) . Consistency-based approaches that have been popular in the classification setting , such as Mean Teacher ( Tarvainen & Valpola , 2017 ) and Virtual Adversarial Training ( Miyato et al. , 2018 ) , which reinforce the output consistency of the network under input perturbations , have also been adapted to the regression setting ( Jean et al. , 2018 ) . However , enforcing consistency alone may not be sufficient for good performance , and may lead to model collapsing ( Qiao et al. , 2018 ) or confirmation bias issues ( Ke et al. , 2019 ) . To address these issues , we draw inspiration from ensemble learning with neural networks for regression . A necessary and sufficient condition for an ensemble of learners to be more accurate than any of its individual members is if the base learners are accurate and diverse ( Dietterich , 2000 ) . Therefore , the key component that can make or break an ensemble is the diversity ( or disagreement ) among its individual regressors . If this diversity is insufficient , the ensembling may not result in better performance . On the other hand , overemphasizing diversity can degrade the learnability of the ensemble members . So far , the most successful mechanism to leverage ensemble diversity in regression is Negative Correlation Learning ( Liu & Yao , 1999 ; Zhang et al. , 2019 ) . In this work , we propose Diverse and Consistent Multi-view Networks for Semi-supervised Regression ( DiCoM ) that elegantly unifies consistency and diversity in a multi-view learning framework . Based on probabilistic graphical assumptions , we derive a loss function that integrates both consistency and diversity components – diversity is encouraged on labeled data , while consistency is enforced on unlabeled data . Furthermore , we develop two variants of DiCoM : the first uses multiple networks to achieve better performance , while the second employs a single network with multiple branches to help with scalability . We compare DiCoM against state-of-the-art methods on eight tabular datasets and a crowd-counting dataset , where we show that DiCoM outperforms existing methods . We further perform ablation studies to analyze the importance of diversity and consistency , and the effect of varying the number of views in the model . While other works have leveraged related ideas of complementary and consensus in multi-view classification ( Xu et al. , 2013 ) ; or explored commonality and individuality in multi-modal curriculum learning ( Gong , 2017 ) , these methods were developed for classification or clustering tasks , and can not be easily modified to suit semi-supervised regression . In summary , the major contributions of this work are as follows : • We derive a novel objective function from a probabilistic graphical perspective . Our objective function unifies multi-view diversity and consistency and provides theoretical insights into the relationship between diversity-consistency trade-off and the number of views . • We show the high flexibility of DiCoM , which can be adaptively scaled up to larger number of views while maintaining competitive performance . • We demonstrate the performance of DiCoM on both tabular and visual types of input data , where it outperforms competing methods . Our ablation studies validate the importance of having both diversity and consistency . 2 RELATED WORK . Semi-supervised Regression : Semi-supervised learning is a data-efficient learning paradigm that offers the ability to learn from unlabeled data . In recent years , much work has focused on semisupervised classification , and there have been far fewer studies on semi-supervised regression . For regression tasks , graph-based methods are among the first to be developed . One example is Label Propagation ( LP ) ( Zhur & Ghahramanirh , 2002 ) which defines a graph of training data and propagates ground-truth labels through high density regions of the graph . Kernel methods have also been proposed , such as Semi-supervised Deep Kernel Learning ( SSDKL ) ( Jean et al. , 2018 ) . This method minimizes the predictive variance in a posterior regularization framework to learn a more generalizable feature embedding on unlabeled data . Co-training regressors ( COREG ) ( Zhou & Li , 2005 ) employs k-Nearest neighbor regressors , each of which generate pseudo-labels for the other during training ; this helps to maximize their agreement on unlabeled data . Apart from the aforementioned approaches , consistency-based methods are also gaining traction . Mean Teacher ( MT ) ( Tarvainen & Valpola , 2017 ) enforces posterior consistency between two neural networks , a student and a teacher , the latter being an exponential moving average of the former in the parameter space . An orthogonal approach is to enforce consistency on adversarially augmented input , as implemented in Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2018 ) . These methods were originally developed for classification , and were subsequently adapted to regression tasks ( Jean et al. , 2018 ) . However , both MT and VAT maintain only a single trainable network , which may lead to problems such as confirmation bias ( Ke et al. , 2019 ) and overly-sensitive hyperparameters . In this paper , we show that consistency-based methods can be further improved with ensemble diversity . Ensemble Diversity : Ensembles of neural networks have been extensively studied and widely used in many applications . Their effectiveness largely depends on the level of diversity ( or disagreement ) among members of the ensemble . It is well-understood that a good ensemble must manage the tradeoff between the accuracy of the individual learners and the diversity among them ( Brown et al. , 2005 ; Tang et al. , 2006 ) . For regression tasks , a commonly-used ensemble technique is Negative Correlation Learning ( NCL ) ( Liu & Yao , 1999 ; Liu et al. , 2000 ) , which formulates a diversity-promoting loss using an ambiguity decomposition of the squared ensemble loss ( Krogh et al. , 1995 ) . In this formulation , a correlation penalty term ( also refered to as an ambiguity term ) measures how much each member ’ s prediction deviates from the ensemble output . When this penalty term is maximized , the errors of individual learners become negatively correlated . It was theoretically proven ( Brown et al. , 2005 ) that the strategy employed by NCL is equivalent to leveraging a bias-variance-covariance trade-off ( Ueda & Nakano , 1996 ) of the ensemble error . Recently , NCL has been extended to semi-supervised learning ( Chen et al. , 2018 ) , where the correlation penalty term is extended to the unlabeled data . However , this method was demonstrated only on tabular data . Another variant of NCL is Deep Negative Correlation Learning ( DNCL ) ( Shi et al. , 2018 ; Zhang et al. , 2019 ) , which is designed for visual regression tasks in a purely supervised learning setting . Multi-view Learning : A dataset is considered as having multiple views when its data samples are represented by more than one feature set , each of which is sufficient for the learning task . Although each view is supposed to be sufficient for learning the task , a model trained on only one single view often faces the risk of overfitting , especially when labeled data is limited ( Xu et al. , 2013 ) . To address this problem , multi-view learning assigns a modeling function to each view and jointly optimize these functions to improve overall generalization performance ( Zhao et al. , 2017 ) . By analyzing the development of various techniques , Xu et al . ( Xu et al. , 2013 ) summarized two significant principles that underpin multi-view learning : consensus and complementary . The consensus principle states that a multi-view technique must aim at maximizing the agreement on different views . This is similar to how consistency-based semi-supervised learning methods works : for instance , MT enforces agreement with its past self . The complementary principle states that in order to make improvement , each view must contain some information that the other views do not carry . In other words , the views should be sufficiently diverse . This is related to diversity regularization in ensemble learning , where individual learners are encouraged to give diverse predictions . Thus , multi-view learning offers a unifying perspective of both consistency and diversity . 3 PROPOSED METHOD . We start by describing how multiple deep views can be generated from input data . Then , we propose our multi-view learning framework for regression , in which multiple deep views can be simultaneously optimized via backpropagation . We then discuss the graphical models that govern the probabilistic dependencies among the ground-truth label and the deep views . Finally , we derive the DiCoM loss function using these graphical models and provide a few insights . View creation : Consider a regression task where the goal is to estimate a label y ∈ R from an input x . To create multiple views , our first approach is to use M neural networks F1 , F2 , . . . , FM , each parameterized by θ1 , θ2 , . . . , θM , respectively . By applying different data augmentations η1 , η2 , . . . , ηM on the original x , we generate M different augmented inputs xm = ηm ( x ) ∀m= 1 , . . . , M . With each augmented input , the corresponding neural network produces a regression output fm ( x ) = Fm ( xm , θm ) ∀ m = 1 , . . . , M . Due to the different augmentations and network parameters , each output fm can be treated as one deep view of the original input x . We call this multi-network setup DiCoM-N , where the N stands for ‘ network ’ ( see Fig . 1 ( a ) ) . The second approach is to utilize a single network with a shared backbone B and multiple branches F1 , F2 , . . . , FM . We use θB to denote the learnable parameters from the backbone and θ1 , θ2 , . . . , θM to denote the parameters of the branches . The hidden features generated by the backbone serve as input to the branches . While the backbone still applies a random augmentation to the input x , each branch Fm applies its own random augmentation ηm as well . Thus , regression outputs f1 , f2 , . . . , fM from the branches can be considered as deep views of the original input . We name this setup DiCoM-B , where the B is short for ‘ branch ’ ( see Fig . 1 ( b ) ) . Multi-branch technique was widely adopted for supervised classification ( Xie et al. , 2017 ) . In this work , it allows us to harness the power of multi-view learning with a relatively lower number of trainable parameters . Multi-view learning framework for regression : Regardless of how they are generated , the deep views are used together with the true label y to compute a semi-supervised loss function LDiCoM . During training phase , LDiCoM is back-propagated simultaneously through the deep views to optimize network parameters θ1 , θ2 , . . . , θM ( including θB in the case of DiCoM-B ) . During inference , all augmentations are removed so that the forward pass is applied on the raw input x . The final prediction is computed as the average of all deep views : µ ( x ) = ∑M i=1 1 M fm ( x ) . The general DiCoM framework is illustrated in Fig . 1 . In the next step , we derive LDiCoM based on a probabilistic graphical assumption . Probabilistic graphical models : Since the augmented inputs are generated from the same sample , the deep views should be close to each other . Motivated by previous work in kernel learning ( Yu et al. , 2011 ) and linear regression ( Nguyen et al. , 2019 ) , we consider f1 , f2 , . . . , fM as random variables and introduce a consensus function fc as a latent variable that connects to each of the deep views . This function enforces the mutual agreement among the views . We assume that the difference between the consensus function and each view follows a zero-mean Gaussian distribution fc − fm ∝ N ( 0 , σ2m ) ∀m = 1 , . . . , M. ( 1 ) This probabilistic relation is known as the consensus potential ( Yu et al. , 2011 ) . Considering the whole graph , this potential implies that all views are random Gaussian variables with a shared mean fc and variance σ2m . As a result , the views stay consistent w.r.t . each other by taking values not too far away from the shared consensus . This graphical model , shown in Fig . 3 ( a ) , is assumed for each unlabeled sample . The joint density associated with the graph is given by p ( fc , f1 , . . . , fM ) = 1 Z M∏ m=1 Ψ ( fc , fm ) ( 2 ) where Z is a normalizing constant and Ψ ( fc , fm ) = exp [ − ( fc−fm ) 2 2σ2m ] is the potential function of the edge connecting fc and fm . From this model , we derive two important results . On a side note , our derivation generalizes to vector-valued labels y , but here we assume scalar labels for ease of exposition . The proofs of our results are provided in Appendix A . ( I ) Marginalization of the views : By integrating the latent consensus function fc out of the joint density , the marginal distribution of the views is p ( f1 , . . . , fM ) ∝ exp [ M∑ m=1 ∑ k > m −λm , k ( fm − fk ) 2 ] ( 3 ) where λm , k = [ 2σ2mσ 2 k ( ∑ m 1 σ2m ) ] −1 . This result implies that the marginal likelihood can be fac- torized as a product of ( M 2 ) terms . Each term is an isotropic Gaussian distribution on the difference between a pair of views ( fm , fk ) , with zero mean and variance ( 2λm , k ) −1 . The equivalent graphical model is shown in Fig . 3 ( b ) . ( II ) Conditional of the consensus function : By applying Bayes ’ theorem , the conditional distribution of the consensus function fc given all the views is a Gaussian fc|f1 , . . . , fM ∼ N ( µ̃ , σ2µ ) . ( 4 ) where σ2µ = ( ∑ m 1 σ2m ) −1 and µ̃ = σ2µ ∑ m fm σ2m . This result highlights that the conditional distribution of fc depends only on the weighted average µ̃ , and the values of individual views are not required . Furthermore , µ̃ can be treated as a view itself , with a variance that is smaller than any of the variances of the views . Derivation of DiCoM loss function : For simplicity , we assume equal variance for different deep views , i.e. , σ2m = σ 2 v ∀m . For an unlabeled sample ( xn ) , we directly apply the first result ( I ) to obtain the following negative log likelihood function Lunl = M∑ m=1 ∑ k > m 1 2Mσ2v [ fm ( xn ) − fk ( xn ) ] 2 ( 5 ) For a labeled sample ( xn , yn ) , since the ground-truth is given , we assume a graphical model that involves the final DiCoM prediction , i.e. , the averaged output µ . This graph is shown in Fig . 3 ( c ) . Since we assume a shared variance σ2v , the weighted output now reduces to an equal-weight average , following from result ( II ) µ̃ ( xn ) = M∑ m=1 fm ( xn ) M = µ ( xn ) σ 2 µ = σ2v M ( 6 ) Subsequently , we apply result ( I ) on this graph to get the negative log likelihood as follows Llab = M 2Mσ2y + 2σ 2 v [ yn − µ ( xn ) ] 2 ( 7 ) = 1 2Mσ2y + 2σ 2 v M∑ m=1 { [ fm ( xn ) − yn ] 2 − [ fm ( xn ) − µ ( xn ) ] 2 } ( 8 ) ≈ 1 2σ2v M∑ m=1 { [ fm ( xn ) − yn ] 2 − [ fm ( xn ) − µ ( xn ) ] 2 } ( 9 ) where in equation ( 8 ) , we have applied the ambiguity decomposition ( Krogh et al. , 1995 ) and in equation ( 9 ) , we have assumed that the label is accurate , i.e. , σ2y σ2v . Given a training batch of labeled samples { ( xn , yn ) } Ln=1 and unlabeled samples { ( xn ) } Un=1 , assuming that the samples are independently generated , we can add the log-likelihood functions across all training samples . This can be done via simply adding up two equations ( 5 ) and ( 9 ) LDiCoM = 1 L L∑ n=1 M∑ m=1 { [ fm ( xn ) − yn ] 2 − κdiv [ fm ( xn ) − µ ( xn ) ] 2 } + 1 U U∑ n=1 M∑ m=1 ∑ k > m κcsc [ fm ( xn ) − fk ( xn ) ] 2 ( 10 ) where we introduce two hyperparameters κdiv and κcsc to absorb other constants and to enable a trade-off between diversity and consistency components of the loss . The DiCoM loss encourages diversity on labeled data , while enforcing consistency on unlabeled data . These two seemingly opposing components can both be derived from the same underlying graphical assumptions . Furthermore , they should not be weighted equally . In fact , we have shown that it depends on the number of views : when M increases , diversity grows in O ( M ) , while consistency grows in O ( M2 ) . It is worth noting that our method is fundamentally different from other extensions of NCL such as Semi-supervised NCL ( Jean et al. , 2018 ) , which enforces diversity on both labeled and unlabeled data . Last but not least , since both diversity and consistency are incorporated in the DiCoM objective function , the method is highly adaptable to different implementations such as multi-network or multi-branch , as long as the views are provided .

Summary: This paper concerns the semi-supervised problem. Different from SOTA deep semi-supervised methods, the proposed DiCom employs a diversity measure on the labeled multi-view data, and combines diversity with consistency based on underlying probabilistic graphical assumptions. Experiments verify the effectiveness.