A Survey on Semi-Supervised Learning Algorithms — part 1
By: Aardra Kannan Ambili
Tags: Data, Data Science, Supervised Learning, Machine Learning, Classification Algorithms

A beautiful picture of a manifold (taken from the internet)

I wrote this paper back in the Spring of 2013, when dealt with a problem in cancer detection through tumour images and was faced with the issue of little or no data. This is quite a common issue among most machine learning tasks (especially in industry)— and there are only two ways to go about it — (1) to artificially create the data (mimicking the results) or (2) to use semi-supervised learning algorithms and then to hope for the best. :)

Introduction

This paper is a Literature survey on Semi-supervised learning Algorithms, which are a class of Machine Learning algorithms that learn from both labeled and unlabeled data. This paper does not claim to cover the whole field of Semi-supervised learning algorithms, nor does it intend to go into the mathematical details involved in each algorithm. It is meant to be a practical overview of certain classes of Semi-Supervised learning algorithms, some of which have often outdone their counterparts.

These algorithms are in contrast with Supervised Learning algorithms (that learn only from labeled data) and Unsupervised Learning algorithms (that learn only from unlabeled data). The growth of Semi-supervised Learning algorithms is due to the fact that in most real-world applications, labeled data is often found in small amounts, while unlabeled data is found in abundance. Converting unlabeled data to labeled data is an expensive and laborious process. Semi-supervised learning tries to use the labeled and the abundant unlabeled data to achieve good results in the Machine Learning world. However, the process of trying to learn the structure of the whole data space from a small amount of labeled data is not easy. We have to use strong assumptions in trying to learn the joint probability P(x, y), for the relation between P(x) and P (y|x), where x is the instance and y is the label of the instance. (Zhu, 2008) Before going any deeper into the classification of the algorithms, I think it’s necessary to give an informative account of the assumptions some of the algorithms use.

A brief note should be mentioned about Transductive learning. Transductive learning is learning from labeled data and unlabeled data like Semi-supervised learning, but the difference is that it cannot be applied on unseen examples. Inductive learning can handle unseen examples. Under this definition, Tranductive SVMs belong to the class of Semi-Supervised Learning Algorithms.

2. Assumptions

2. A. The Cluster Assumption

According to Chapelle & Zien (2005), almost all Semi-supervised learning algorithms make use of the cluster assumption either directly or indirectly. The cluster assumption says that the decision boundary which is technically the region between well-formed clusters should reside in a low-density region, and should never cross high-density regions. To elucidate, the path connecting high density regions would contain datapoints that have the same data label or would belong to one class. Consider for instance an example where the cluster assumption does not hold: a uniform distribution of input points. Then the unlabeled data would convey almost no information and in fact produces miscalculations, and maximizing the margin on those instances is useless. Such a situation can in fact be disastrous to the algorithm and ruin the model learned by the classifier from the labeled data. For example, the Transductive SVM (Support Vector Machine), which is inductive by nature as they are defined over the whole instance space, attempts to find a hyperplane as far away from the unlabeled points as possible, thus implementing the cluster assumption indirectly. An example of a Semi-Supervised learning algorithm that indirectly uses the Cluster assumption is the class of the Manifold learning algorithms. A manifold is a n-dimensional topological space, in which the neighborhood of any point on the topological space would be identical to n-dimensional Euclidean space. Intuitively, a manifold is some space such that if you zoom in enough, it looks like flat Euclidean space. In Manifold Learning, the algorithm tries to learn decision functions that vary little along the manifold on which the data resides.

B. The Manifold Assumption

Continuing our discussion on manifolds, the manifold assumption basically states that different classes lie on different manifolds. The manifold assumption can be considered as a natural extension to the cluster assumption as the manifold assumption merely extends the latter by stating that the data points on the path connecting the high density regions on the data manifold are likely to have the same label. However most often we have the data (both labeled and unlabeled) but no information about the density or the manifolds. One way to solve this problem would be to use a graph to approximate density.

The manifold folds can represent data in different dimensions. (Vivodtzey et al, 2005)

C. The Smoothness Assumption

Points which are close to each other are more likely to share a label. This is also generally assumed in supervised learning and these would be simpler to solve as this would yield simpler decision surfaces. In the case of semi-supervised learning, the smoothness assumption produces a preference for decision boundaries in regions, where there are fewer points close to each other.

3. Semi-Supervised Classification algorithms

3. A. Low Density Separations

3. A.1. Transductive SVM (Semi-Supervised SVM or S3SVM)

According to Chapelle et al (2005), “TSVM might seem to be the perfect semi-supervised algorithm, since it combines the powerful regularization of SVMs with a direct implementation of the cluster assumption.” The paper explains that the maximization of the margin whether in labeled data (through regularization) and unlabeled data that implements the cluster assumption but does not exactly use regularization, through a view by Vapnik(1998).

Given two jointly distributed random variables X and Y, the conditional probability distribution of Y given X is the probability distribution of Y when X is known to be a particular value. The discriminative approach to the learning problem where x is a vector of instances and c is the vector of corresponding class labels, is to represent the conditional distribution using a parametric model, and then to determine the parameters using a training set consisting of pairs {x, c}. (Bishop & Lasserre, 2007).However they have the unfortunate blind spot that they leave out P(x), outside of the parameter estimation loop, if P(x) and P(c|x) do not share parameters. P(x) is a critical component as it is all we get from unlabeled data. It is believed that if P(x) and P(y|x) do not share parameters, the discriminative approach to semi-supervised learning cannot help. (Seeger, 2001).

However, TSVMs go an additional step ahead and build the connection between p(x) and the discriminative decision boundary by not putting the boundary in high density regions. The decision boundary should have the smallest generalization error bound on unlabeled data (Vapnik, 1998). So we can infer that the TSVM lets the unlabeled data guides the process of producing a decision boundary such that it has the smallest generalization error bound on the unlabeled data. However finding the ideal decision boundary using TSVM is an NP-hard problem. Efficient approximation algorithms (Bennett & Demiriz, 1999) (Bennett & Demiriz, 2000) (Fung & Mangasarian, 1999) to solve the problem either could not handle more than a few hundred unlabeled examples, or did not do so in experiments. The SVM-light TSVM implementation (Joachims, 1999) is the first widely used software that implemented these algorithms.

De Bie and Cristianini (De Bie & Cristianini, 2004; De Bie & Cristianini, 2006) remodel the Transductive learning problem into a semi-definite programming problem (SDP). SDP is a subfield of convex optimization concerned with the optimization of a linear objective function. Xu and Schuurmans (2005) also present a similar multi-class version of the SDP formulation. The computational cost of SDP was considered expensive but with today’s multi-processor GPUs, it poses an open research question.

Chapelle and Zien (2005) propose ∇SVM which solves the learning problem, by transforming the hat loss function (the hat loss is a sum of a convex function and a concave function) with a Gaussian function, and perform gradient search. Sindhwani et al. (2006) uses an intuitive deterministic annealing approach and involves a sequence of convex optimization problems, which starts from an ‘easy’ problem, and gradually deforms it to the TSVM objective. Chapelle, Chi & Zien (2006) use a similar continuation approach, using the solution of previous iterations to initialize the next ones. Collobert et al. (2006) optimizes the hard TSVM directly,through an optimization procedure known as the concave-convex procedure (CCCP). CCCP can be applied to any optimization problem (after fulfilling certain conditions) and many existing algorithms can be interpreted in terms of CCCP. (Yuille & Rangarajan, 2003) One can perform convex minimization to produce an upper bound of the loss function, after replacing the concave part of the hat loss function. The algorithm is iterated until a local minima is reached. Although this algorithm does not reach global optima, there is significant speed up of TSVM training with CCCP. However Chapelle, Sindhwani & Keerthi (2006) is able to find global optimal solution for small datasets using the Branch and bound method. Although Branch and Bound shows exciting promise and cannot be implemented for large datasets, the results provide some relief and points to the potentials of TSVMs in finding better solutions faster.

3. B. Self-Training

A commonly used wrapper algorithm in Semi-supervised Learning, it is also known as bootstrapping. This is probably one of my favorite go-to methods. Self-training has been applied to several natural language processing tasks. Yarowsky (1995) used self-training for word sense disambiguation. It has also been used for subjective nouns. Initially, a classifier is first trained by labeled data and then used to estimate labels for the unlabeled data. A selection metric is then used to decide which of the weakly labeled examples were labeled correctly. The use of the selection metric is crucial. The mostly confidently classified unlabeled instances together with their predicted labels are added to the training set. The classifier in then re-trained on this newly formed labeled dataset. Then again, the re-trained classifier is used to estimate labels for the data points from the pool of unlabeled data. This process is repeated until the unlabeled data set runs empty or when until the maximum no. of iterations is reached. The algorithm is straightforward and intuitive. Usually it produces outstanding results. However, it will fail to converge, if labeled data does not represent patterns in the data and does not give good results with discriminative classiﬁers (Gan et al., 2013).

Rosenberg et al. (2005) performs an empirical study to show that a model trained with a large set of unlabeled data and small set of labeled data is able to perform as well, if not better than a model trained with a much larger labeled dataset .They also show that it is important to define a training data selection metric which is independent of the detector as this greatly outperforms a selection metric based on the detection confidence generated by the detector.

(Wang et al., 2008) investigates the use of Semi-supervised learning specifically Self-training for subjectivity analysis at the sentence level. Naive Bayes (NB) is often selected as the base classifier as its class membership probability estimates have good ranking performance. The first contribution of this paper is to study the performance of self-training using different base classifiers. They experiments are performed on the decision tree models, C4.5, C4.4, and naïve Bayes tree (NBTree). They also propose an adapted Value Difference Metric (VDM) as the selection metric in self-training, which does not depend on class membership probabilities. The experimental results were impressive in showing two important results: One that the performance of self-training is improved by using VDM instead of the confidence degree. Second, that self-training with NB Tree and VDM outperforms self-training with other combinations of underlying classifiers and selection metrics. Guo, Niu and Zhang (2010) performed an extensive empirical study on 26 UCI datasets to show that the confidence selection metric was not certainly superior to randomly selecting unlabeled data instances. Li and Zhou (2005) demonstrate a method named SETRED that utilizes the information of the neighbors of each self-labeled instance (The instances that are selected using the selection metric are called “self-labeled” instances) to identify and remove the mislabeled examples from the self-labeled data.

Guo et al. (2011) propose a selection method called the Instance Selection Method Based On The Original Labeled Data (ISBOLD) in place of the confidence selection metric to improve the performance of self-training and co-training when using Naive Bayes (NB) as the base classifier. ISBOLD considers two factors: One is the prediction confidence of the current classifier on the self-labeled data and the second factor is the accuracy on the original labeled data. Experiments on 26 UCI datasets demonstrate that, ISBOLD significantly improves the accuracy of self-training and co-training compared to our previous experimental results in 2010. (Guo et al, 2010) Semi-supervised fuzzy c-means algorithm, a semi-supervised clustering algorithm is integrated into Self-training classification to help train a better classifier. (Gan et al., 2013)

3. B. 1. Help-Training

An improved version of Self-Training called Help-Training has been proposed. (Adankon et al, 2011).The new algorithm outperforms Self-Training by using a generative classifier which does not focus only on the boundary between classes but focuses on their joint probability, P(x, y) of the instances and the labels, that may help to train the main discriminative classifier to label the unlabeled data. The authors report that the new algorithm performs better than the Self-Training strategy.

C. Co-training and Multi-view Learning

C. 1. Co-training

Blum & Mitchell (1998) propose the Co-training algorithm. The Co-training algorithm requires certain assumptions to be fulfilled, although certain algorithms have been proposed that relax these conditional assumptions. (Blum et al, 1998). The conditional independence assumption states that the data should be expressed by two separates sets of features that can define the solution. To elucidate consider the problem of classification of web-pages. The possible description of a web-page in terms of available features can be partitioned into two views: One view is through words occurring on that page. A second view is through words which are hyperlinks to that webpage. So either view would be sufficient for learning if there was sufficient labeled data, but since we don’t, the goal is to incorporate both the views together to allow inexpensive unlabeled data to augment a small set of labeled data to attain superior classification accuracy. According to the paper, as long as conditional independence is maintained the below-mentioned stringent assumption can be relaxed.

f1(x1 ) ≠ f2(x2 ), where f1 and f2 are target functions for the two separate views and x1, x2 are examples.

This assumption would have been hard to fulfill for all the training examples. As long as these assumptions are maintained, Blum et al, (1989) sets up the algorithm as follows: Initially two base supervised classifiers are trained with labeled data. In the paper, they use Naïve Bayes as the base classifier. Here the condition that both classifiers are completely capable of defining the target function using two separate sub-feature subsets is still maintained. Each classifier then classifies the unlabeled data, and then provides the other classifier with a few unlabeled examples (and their corresponding predicted labels) which it feels feel most confident about (this selection is made use of a selection metric). Then finally, each classifier is retrained with newly formed labeled dataset given by the other classifier, and the process repeats. The co-training classifier outperforms the other classifiers, formed by supervised training in such scenarios.

Nigam et al. (2000) performs an empirical comparison of co-training with generative mixture models and EM. Their result shows co-training performs well if the conditional independence assumption indeed holds. Surprisingly they are able to show that co-training with artificial feature split is a possible solution, and that it might still outperform algorithms that do not use a split, although the performance is not as superior as the ones which leverage the conditional independence assumption. The artificial feature split is done by randomly breaking the set into sets. This artificial split can also be performed by developing a splitting algorithm that reduces the conditional mutual information between the feature sets to zero. This can be achieved by using efficient approximate min-cut graph partitioning algorithms. This hasn’t been done yet and it is a very interesting research question. Additionally, one of the reasons why co-training performs so well is because it is more robust to the assumptions made by its underlying classifier than the classifiers itself. Balcan, Blum & Yang, (2005) propose a variation on Co-training which relaxes the conditional assumption.

In addition, it is better to probabilistically label the entire U, instead of just considering the most confident weakly labeled data points. They name this paradigm Co-EM. The Co-EM algorithm does exactly this using one learner to assign labels to all the unlabeled data, from which the second classifier learns from. In contrast, the co-training algorithm learns from only a single example at a time. According to Nigam et al. (2000), this is more in agreement with the concept proposed by Blum et al. (1998) than co-training could be.

Collins et al. (1999) and Jones (2005) implemented co-training, Co-EM and other related methods for information extraction from text. Co-testing (Muslea et al., 2000), is an active learning approach that works like Co-training, uses two classifiers with redundant views and compares their outputs for an unlabeled example. If they disagree, then the example is considered as a inflection from the regular data, and therefore it should be handed over to the domain expert for manual labeling. However, this might be a problem in some real-world applications where unlabeled data is left unlabeled because their labels cannot be estimated even by experts.

Zhang (2007) further relaxes one of the major assumptions of the Co-training model. They propose a two-view feature generation model that does not assume that the two views can independently predict labels for the unknown examples. In this model, the two view feature model is similar to the one used in Co-training, but here a generative model is used to learn eﬀective parametric feature representations for discriminative learning. Therefore, one can construct a small set of features from unlabeled data so that the optimum predictor can be represented as a linear combination of these features.

Sun et al. (2011) propose an entity-based co-training (EcT) algorithm, which considers into account the entity semantics hidden in web pages and requires no prior knowledge about the underlying class distribution which is crucial in standard co-training algorithms used in web classification. Jeon et al. (2012) proposes a novel labeling and selection method under the co-training framework for prosodic event detection tasks, which does not consider the assumptions.

To be continued..

References A. K. Jain, M. N. Murty, and P. J. Flynn. (1999) Data clustering: A review. ACM Computing Surveys, 31(3):264–323 Adankon, M. M., Cheriet M., (2011) Help-training for semi-supervised support vector machines, Pattern Recognition 44 (9) 2220–2230. Aixin Sun, Ying Liu, Ee-Peng Lim(2011), Web classification of conceptual entities using co-training, Expert Systems with Applications 38(12)14367–14375 Balcan, M.-F., Blum, A., & Yang, K. (2005). Co-training and expansion: Towards bridging theory and practice. In L. K. Saul, Y. Weiss and L. Bottou (Eds.), Advances in neural information processing systems 17. Cambridge, MA: MIT Press. Baluja, S. (1998). Probabilistic modeling for face orientation discrimination: Learning from labeled and unlabeled data. Neural Information Processing Systems. Bar-Hillel, Aaron, Hertz, T., Shental, Noam, Weinshall, Daphna, 2003. Learning distance functions using equivalence relations. In: Proc. 20th Internat. Conf. on Machine Learning, pp. 11–18. Basu, S.; Banerjee, A.; Mooney, R. (2002) Semi-Supervised Clustering by Seeding Proceedings of the 19th International Conference on Machine Learning (ICML-2002), pp. 19–26, Sydney, Australia, July 2002 Basu, S.; Bilenko, M. ;, Mooney, R. J. (2004). A probabilistic framework for semi-supervised clustering. In: Proc. 10th KDD, pp. 59–68. Bauer, E. and Kohavi, R. (2004) An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants, Machine Learning, 36, 105–139 Bennett, K., & Demiriz, A. (1999). Semi-supervised support vector machines. Advances in Neural Information Processing Systems, 11, 368–374. Bilenko, M., Basu, S., Mooney, R.J. (2004) Integrating constraints and metric learning in semi-supervised clustering. In: Proceedings of 21st International Conference on Machine Learning, ICML Bishop, C. M. and Lasserre, J. (2007). Generative or Discriminative? Getting the best of both worlds, In Bayesian Statistics 8, Bernardo, J. M. et al. (Eds), Oxford University Press. 3–23. Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. COLT: Proceedings of the Workshop on Computational Learning Theory. Brefeld, U., & Scheffer, T. (2006). Semi-supervised learning for structured output variables. ICML06, 23rd International Conference on Machine Learning. Pittsburgh, USA. Brefeld, U., B¨uscher, C., & Scheffer, T. (2005). Multiview discriminative sequential learning. European Conference on Machine Learning (ECML). Brefeld, U., Gaertner, T., Scheffer, T., & Wrobel, S. (2006). Efficient coregularized least squares regression. ICML06, 23rd International Conference on Machine Learning. Pittsburgh, USA. Castelli, V., & Cover, T. (1995). The exponential value of labeled samples. Pattern Recognition Letters, 16, 105–111. Castelli, V., & Cover, T. (1996). The relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter. IEEE Transactions on Information Theory, 42, 2101–2117. Chapelle O., Zien A., (2005). Semi-Supervised Classification by Low Density Separation, 10th International Workshop on Artificial Intelligence and Statistics, AI STATS 2005, Barbados Chapelle, O., Chi, M., & Zien, A. (2006). A continuation method for semi supervised SVMs. ICML06, 23rd International Conference on Machine Learning. Pittsburgh, USA. Chapelle, O., Sindhwani, V., & Keerthi, S. S. (2006). Branch and bound for semi supervised support vector machines. Advances in Neural Information Processing Systems (NIPS). Chapelle, O.; Schölkopf, B.; Zien, A. (2006) (Eds.), Semi-Supervised Learning, MIT Press, Cambridge, MA Collins, M., & Singer, Y. (1999). Unsupervised models for named entity classification. Collobert, R., Weston, J., & Bottou, L. (2006). Trading convexity for scalability. ICML06, 23rd International Conference on Machine Learning. Pittsburgh, USA. Cozman, F.G., Cohen, I. (2002): Unlabeled data can degrade classification performance of generative classifiers. In: Proceedings of the Fifteenth International Florida Artificial Intelligence Research Society Conference De Bie, T., & Cristianini, N. (2004). Convex methods for transduction. In S. Thrun, L. Saul and B. Sch¨olkopf (Eds.), Advances in neural information processing systems 16. Cambridge, MA: MIT Press. De Bie, T., & Cristianini, N. (2006). Semi-supervised learning using semi definite programming. In O. Chapelle, B. Scho¨elkopf and A. Zien (Eds.), Semi supervised learning. Cambridge-Massachusetts: MIT Press. de Sa, V. R. (1993). Learning classification with unlabeled data. Advances in Neural Information Processing Systems (NIPS). Demir, C.; Saraclar, M.; Cemgil, A.T.,(2013) “Single-Channel Speech-Music Separation for Robust ASR With Mixture Models,” Audio, Speech, and Language Processing, IEEE Transactions on 21(4),725,736. Demirez, A., & Bennett, K. (2000). Optimization approaches to semi supervised learning. In M. Ferris, O. Mangasarian and J. Pang (Eds.), Applications and algorithms of complementarity. Boston: Kluwer Academic Publishers. Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B. Elworthy, D. (1994). Does Baum-Welch re-estimation help taggers? Proceedings of the 4th Conference on Applied Natural Language Processing.

EMNLP/VLC-99.