# A Semi-Supervised Framework for MMMs-Induced Fuzzy Co-Clustering with Virtual Samples

Authors: Daiji Tanaka; Katsuhiro Honda; Seiki Ubukata; Akira Notsu
Journal: Advances in Fuzzy Systems
 (2016)

---

## Abstract

Although the goal of clustering is to reveal structural information from unlabeled datasets, in cases with partial structural supervisions, semi-supervised clustering is expected to improve partition quality. However, in many real applications, it may cause additional costs to provide an enough amount of supervised objects with class labels. A virtual sample approach is a practical technique for improving classification quality in semi-supervised learning, in which additional virtual samples are generated from supervised objects. In this research, the virtual sample approach is adopted in semi-supervised fuzzy co-clustering, where the goal is to reveal object-item pairwise cluster structures from cooccurrence information among them. Several experimental results demonstrate the characteristics of the proposed approach.


---

## Body

## 1. Introduction

Clustering or cluster analysis is a basic technique for unsupervised classification, whose goal is to reveal intrinsic substructures varied in large scale unlabeled datasets. In some applications, however, it is possible to utilize partial knowledge on the substructures [1], such asmust-link andcannot-link among some objects [2–4], class labels on a part of objects [5–7], or predefined fuzzy membership degrees [8, 9], and it is expected that partition quality can be significantly improved by utilizing such partial knowledge. In this study, a situation of having semi-supervision is considered, in which we have some supervising objects in conjunction with their class labels. Semi-supervised clustering [5, 10] is a practical approach for utilizing partial supervised information with the goal of improving the partition quality in unsupervised classification. Such partial knowledge can be utilized in two phases ofsupervised initialization andsupervised membership assignment [6, 7].Although partition quality is expected to be improved with sufficient semi-supervisions, in many real world applications, it is often difficult to utilize an enough amount of supervised objects. For example, in many web data analyses, various open data are available while most of them are unlabeled and their class labels cannot be provided without heavy costs. A promising approach to improve classification quality in semi-supervised learning is the virtual sample approach, in which additional virtual samples are artificially generated from several supervised objects [11–13].In this paper, the virtual sample approach is adopted in semi-supervised fuzzy co-clustering while the conventional model [13] was designed for semi-supervised classification in multidimensional data space. Fuzzy co-clustering is a fundamental technique for summarizing mutual cooccurrence information among objects and items such as document-keyword frequencies in document analysis [14] and customer-products preferences in purchase history analysis [15]. The task of co-clustering is achieved by simultaneously estimating memberships of both objects and items from a cooccurrence information matrix. Fuzzy co-clustering induced by MMMs concept (FCCMM) [16] is a fuzzy co-clustering algorithm induced by multinomial mixture models (MMMs) [17], where statistical mixture models are interpreted as k-means-type classification models with regularized objective functions. The iterative algorithm is composed of updating two types of memberships of items and objects. In semi-supervised fuzzy co-clustering [18], partial knowledge was utilized in two phases ofsupervised initialization andsupervised object membership assignment in a similar manner to the framework of [6, 7]. Insupervised initialization, initial item memberships are generated using labeled objects only. Then, insupervised object membership assignment, memberships of supervised objects are prefixed while others are updated in the iterative algorithm.The remaining parts of this paper are organized as follows: Section2 provides a brief review of the MMMs-induced fuzzy co-clustering, which is then combined with a semi-supervised framework. An artificial process for generating virtual samples is introduced in Section 3. Section 4 presents experimental results and Section 5 gives summary conclusions.


## 2. MMMs-Induced Fuzzy Co-Clustering and Its Variant for Semi-Supervised Clustering

### 2.1. Fuzzy Co-Clustering Model Induced by MMMs Concept

Assume that we have a cooccurrence matrixR
=
{
r
i
j
} on objects i
=
1
,
…
,
n and items j
=
1
,
…
,
m, in which r
i
j represents the cooccurrence degree of object i and item j. For example, r
i
j can be the frequency of keyword (item) j in document (object) i in document analysis. In many traditional results, it has been shown that co-cluster structure analysis is useful for summarizing contents such as intrinsic chapter or category information of many documents with their representative keywords. In fuzzy co-clustering context, the task is reduced to the problem of finding the pairwise clusters of mutually familiar objects and items, in which the goal is to estimate the fuzzy memberships of both objects and items such that mutually familiar object and item pairs have large memberships in the same cluster.MMMs [17] is a probabilistic mixture model for co-clustering, in which each component density is multinomial distribution. Multinomial distribution is a multicategory extension of binomial distribution, where the probability of an object with item cooccurrence vector r
i
=
(
r
i
1
,
…
,
r
i
m
)
⊤ is defined as the joint probability of all items with their frequencies of r
i
j. MMMs construct a mixture distribution by iteratively estimating the item occurrence probability and the a priori probability of generative distributions in conjunction with estimating object a posteriori probability to each generative model.Following the soft clustering interpretation of probabilistic mixture models [19], Honda et al. [16] introduced a fuzzy co-clustering-based interpretation of MMMs, where the pseudo-log-likelihood function was decomposed into an object-item aggregation measure and K-L information-based fuzzification penalty. FCCMM [16] is an MMMs-induced fuzzy co-clustering model, in which the degree of object partition fuzziness can be tuned with an adjustable penalty weight on the K-L information-based penalty. For extracting C fuzzy co-clusters, the objective function to be maximized is defined as(1)LFCCMM=∑c=1C∑i=1n∑j=1mucirijlog⁡wcj+λu∑c=1C∑i=1nucilog⁡πcuci,where u
c
i and w
c
j are the fuzzy memberships of object i and item j to cluster c, respectively. π
c is the volume of cluster c such that ∑
c
=
1
C
‍
π
c
=
1. From the MMMs viewpoint, u
c
i is the a posteriori probability of class c given object i and should be constrained to ∑
c
=
1
C
‍
u
c
i
=
1. On the other hand, w
c
j corresponds to the generative probability of item j in component c and should be ∑
j
=
1
m
‍
w
c
j
=
1. Then, u
c
i is mainly responsible for exclusive object partition while w
c
j just represents the typicality of item j within a generative model.∑
c
=
1
C
‍
∑
i
=
1
n
‍
∑
j
=
1
m
‍
u
c
i
r
i
j
log
⁡
w
c
j is the aggregation criterion of objects and items, which measures the degree of aggregation of familiar objects and items, and becomes larger when mutually familiar object-item pairs with large r
i
j have large memberships u
c
i and w
c
j in a same cluster c. From the viewpoint of k-means-type clustering with respect to object memberships u
c
i, the aggregation criterion is essentially a (linear) hard clustering measure, and the K-L information term [20, 21] is responsible for estimating soft partition of objects in the same manner with the soft partition nature of Gaussian mixture models [19]. λ
u tunes the degree of fuzziness of object partition. When λ
u
=
1, the objective function is reduced to the pseudo-log-likelihood function of MMMs with C component distributions. Then, in the case of λ
u
>
1, FCCMM brings a fuzzier co-cluster partition than MMMs while object partition becomes more crisp with λ
u
<
1. As an extremal case, λ
u
→
0 implies a crisp co-clustering model. It was shown that a careful tuning of fuzziness degree can contribute to improvement of partition quality of MMMs by reducing initialization and noise sensitivities [16].The updating rules for these model parameters are given as(2)uci=πc∏j=1mwcjrij/λu∑l=1Cπl∏j=1mwljrij/λu,(3)πc=1n∑i=1nuci,(4)wcj=∑i=1nrijuci∑l=1m∑i=1nriluci.The clustering algorithm is the 3-step iterative process composed of these updating rules.Algorithm of FCCMM
(1)
LetC be the number of clusters. Choose the fuzzification weight λ
u.
(2)
Initialization: initialize object memberships u
c
i, ∀
c
,
i with random values such that ∑
c
=
1
C
‍
u
c
i
=
1.
(3)
Iterative Process: iterate the following process until convergence of all u
c
i.
(a)
Update cluster volumesπ
c using (3).
(b)
Update item membershipsw
c
j using (4).
(c)
Update object membershipsu
c
i using (2).


### 2.2. Semi-Supervised Fuzzy Co-Clustering and Inductive Classification

When some objects have their intrinsic class labels, the partition quality of unsupervised clustering is expected to be improved with their supports [5, 10]. A possible semi-supervised framework for fuzzy co-clustering [18] utilized such partial supervision in two levels:supervised initialization andsupervised membership assignment. Assume that we have an n
×
m cooccurrence matrix, where only a part of objects, such that i
∈
O
s
O
s
<
n, has class labels but others do not. A framework of semi-supervised fuzzy co-clustering is given as follows [18].Semi-Supervised Fuzzy Co-Clustering Framework
(1)
LetC be the number of clusters, which is usually equivalent to the class number of the supervised objects. Choose the fuzzification weight λ
u.
(2)
Initialization: initialize the memberships of supervised objects i
∈
O
s such that u
c
i
=
1 in the labeled class c and u
c
i
=
0 in others. For unsupervised objects i
∉
O
s, u
c
i
=
ε
,
∀
c
,
i, where ε is a small positive. (The initial cluster volumes and item memberships are estimated by mainly reflecting supervised objects only.)
(3)
Iterative Process: iterate the following process until convergence of all u
c
i.
(a)
Update cluster volumesπ
c using (3).
(b)
Update item membershipsw
c
j using (4).
(c)
Update membershipsu
c
i of unsupervised object i
∉
O
s using (2). (The memberships of supervised objects are prefixed and unchanged.)In the initialization level, a plausible initial co-clusters are estimated considering only the supervised objects. Here, this approach is available only if the amount of supervised objects is enough for estimating cluster-wise item preferences, and each cluster index strictly corresponds to the intrinsic class index. In case with insufficient supervision, this initialization step should be performed with the conventional procedure of random assignment for avoiding overfitting to a few supervised objects.Next, in the iterative optimization level, a fixed crisp object membership is assigned to each supervised object reflecting its class label. The partial supervision can contribute to guiding other unlabeled objects to plausible co-clusters. This second level is expected to be useful even if we have only a few supervised objects.Once we got a co-cluster structure, we can perform inductive classification of new (unlabeled) objects supported by the co-cluster information [18].Inductive Classification Considering Fuzzy Co-Cluster Structure
(1)
Assume that we have estimated co-cluster model parameters ofπ
c and w
c
j
,
c
=
1
,
…
,
C
,
j
=
1
,
…
,
m. The goal is to predict the class of a new test object i
∗, which is associated with its cooccurrence information r
i
∗
=
(
r
i
∗
1
,
…
,
r
i
∗
m
).
(2)
Membership Calculation: calculate fuzzy memberships u
c
i
∗
,
c
=
1
,
…
,
C by (2).
(3)
Maximum Membership Assignment: search for the largest u
c
i
∗ and output its class label.This inductive classification approach was shown to outperform the supervised classification with a small set of supervised objects only [18]. This implies that unlabeled objects can contribute to effectively estimating class-wise distributions rather than supervised learning with insufficient supervisions.


## 2.1. Fuzzy Co-Clustering Model Induced by MMMs Concept

Assume that we have a cooccurrence matrixR
=
{
r
i
j
} on objects i
=
1
,
…
,
n and items j
=
1
,
…
,
m, in which r
i
j represents the cooccurrence degree of object i and item j. For example, r
i
j can be the frequency of keyword (item) j in document (object) i in document analysis. In many traditional results, it has been shown that co-cluster structure analysis is useful for summarizing contents such as intrinsic chapter or category information of many documents with their representative keywords. In fuzzy co-clustering context, the task is reduced to the problem of finding the pairwise clusters of mutually familiar objects and items, in which the goal is to estimate the fuzzy memberships of both objects and items such that mutually familiar object and item pairs have large memberships in the same cluster.MMMs [17] is a probabilistic mixture model for co-clustering, in which each component density is multinomial distribution. Multinomial distribution is a multicategory extension of binomial distribution, where the probability of an object with item cooccurrence vector r
i
=
(
r
i
1
,
…
,
r
i
m
)
⊤ is defined as the joint probability of all items with their frequencies of r
i
j. MMMs construct a mixture distribution by iteratively estimating the item occurrence probability and the a priori probability of generative distributions in conjunction with estimating object a posteriori probability to each generative model.Following the soft clustering interpretation of probabilistic mixture models [19], Honda et al. [16] introduced a fuzzy co-clustering-based interpretation of MMMs, where the pseudo-log-likelihood function was decomposed into an object-item aggregation measure and K-L information-based fuzzification penalty. FCCMM [16] is an MMMs-induced fuzzy co-clustering model, in which the degree of object partition fuzziness can be tuned with an adjustable penalty weight on the K-L information-based penalty. For extracting C fuzzy co-clusters, the objective function to be maximized is defined as(1)LFCCMM=∑c=1C∑i=1n∑j=1mucirijlog⁡wcj+λu∑c=1C∑i=1nucilog⁡πcuci,where u
c
i and w
c
j are the fuzzy memberships of object i and item j to cluster c, respectively. π
c is the volume of cluster c such that ∑
c
=
1
C
‍
π
c
=
1. From the MMMs viewpoint, u
c
i is the a posteriori probability of class c given object i and should be constrained to ∑
c
=
1
C
‍
u
c
i
=
1. On the other hand, w
c
j corresponds to the generative probability of item j in component c and should be ∑
j
=
1
m
‍
w
c
j
=
1. Then, u
c
i is mainly responsible for exclusive object partition while w
c
j just represents the typicality of item j within a generative model.∑
c
=
1
C
‍
∑
i
=
1
n
‍
∑
j
=
1
m
‍
u
c
i
r
i
j
log
⁡
w
c
j is the aggregation criterion of objects and items, which measures the degree of aggregation of familiar objects and items, and becomes larger when mutually familiar object-item pairs with large r
i
j have large memberships u
c
i and w
c
j in a same cluster c. From the viewpoint of k-means-type clustering with respect to object memberships u
c
i, the aggregation criterion is essentially a (linear) hard clustering measure, and the K-L information term [20, 21] is responsible for estimating soft partition of objects in the same manner with the soft partition nature of Gaussian mixture models [19]. λ
u tunes the degree of fuzziness of object partition. When λ
u
=
1, the objective function is reduced to the pseudo-log-likelihood function of MMMs with C component distributions. Then, in the case of λ
u
>
1, FCCMM brings a fuzzier co-cluster partition than MMMs while object partition becomes more crisp with λ
u
<
1. As an extremal case, λ
u
→
0 implies a crisp co-clustering model. It was shown that a careful tuning of fuzziness degree can contribute to improvement of partition quality of MMMs by reducing initialization and noise sensitivities [16].The updating rules for these model parameters are given as(2)uci=πc∏j=1mwcjrij/λu∑l=1Cπl∏j=1mwljrij/λu,(3)πc=1n∑i=1nuci,(4)wcj=∑i=1nrijuci∑l=1m∑i=1nriluci.The clustering algorithm is the 3-step iterative process composed of these updating rules.Algorithm of FCCMM
(1)
LetC be the number of clusters. Choose the fuzzification weight λ
u.
(2)
Initialization: initialize object memberships u
c
i, ∀
c
,
i with random values such that ∑
c
=
1
C
‍
u
c
i
=
1.
(3)
Iterative Process: iterate the following process until convergence of all u
c
i.
(a)
Update cluster volumesπ
c using (3).
(b)
Update item membershipsw
c
j using (4).
(c)
Update object membershipsu
c
i using (2).


## 2.2. Semi-Supervised Fuzzy Co-Clustering and Inductive Classification

When some objects have their intrinsic class labels, the partition quality of unsupervised clustering is expected to be improved with their supports [5, 10]. A possible semi-supervised framework for fuzzy co-clustering [18] utilized such partial supervision in two levels:supervised initialization andsupervised membership assignment. Assume that we have an n
×
m cooccurrence matrix, where only a part of objects, such that i
∈
O
s
O
s
<
n, has class labels but others do not. A framework of semi-supervised fuzzy co-clustering is given as follows [18].Semi-Supervised Fuzzy Co-Clustering Framework
(1)
LetC be the number of clusters, which is usually equivalent to the class number of the supervised objects. Choose the fuzzification weight λ
u.
(2)
Initialization: initialize the memberships of supervised objects i
∈
O
s such that u
c
i
=
1 in the labeled class c and u
c
i
=
0 in others. For unsupervised objects i
∉
O
s, u
c
i
=
ε
,
∀
c
,
i, where ε is a small positive. (The initial cluster volumes and item memberships are estimated by mainly reflecting supervised objects only.)
(3)
Iterative Process: iterate the following process until convergence of all u
c
i.
(a)
Update cluster volumesπ
c using (3).
(b)
Update item membershipsw
c
j using (4).
(c)
Update membershipsu
c
i of unsupervised object i
∉
O
s using (2). (The memberships of supervised objects are prefixed and unchanged.)In the initialization level, a plausible initial co-clusters are estimated considering only the supervised objects. Here, this approach is available only if the amount of supervised objects is enough for estimating cluster-wise item preferences, and each cluster index strictly corresponds to the intrinsic class index. In case with insufficient supervision, this initialization step should be performed with the conventional procedure of random assignment for avoiding overfitting to a few supervised objects.Next, in the iterative optimization level, a fixed crisp object membership is assigned to each supervised object reflecting its class label. The partial supervision can contribute to guiding other unlabeled objects to plausible co-clusters. This second level is expected to be useful even if we have only a few supervised objects.Once we got a co-cluster structure, we can perform inductive classification of new (unlabeled) objects supported by the co-cluster information [18].Inductive Classification Considering Fuzzy Co-Cluster Structure
(1)
Assume that we have estimated co-cluster model parameters ofπ
c and w
c
j
,
c
=
1
,
…
,
C
,
j
=
1
,
…
,
m. The goal is to predict the class of a new test object i
∗, which is associated with its cooccurrence information r
i
∗
=
(
r
i
∗
1
,
…
,
r
i
∗
m
).
(2)
Membership Calculation: calculate fuzzy memberships u
c
i
∗
,
c
=
1
,
…
,
C by (2).
(3)
Maximum Membership Assignment: search for the largest u
c
i
∗ and output its class label.This inductive classification approach was shown to outperform the supervised classification with a small set of supervised objects only [18]. This implies that unlabeled objects can contribute to effectively estimating class-wise distributions rather than supervised learning with insufficient supervisions.


## 3. Generation of Virtual Samples for Semi-Supervised Fuzzy Co-Clustering

Although the performance of semi-supervised clustering is expected to be improved as the number of labeled objects becomes larger, it may cause a high cost to generate an enough amount of supervised objects in real applications. The virtual sample approach is a practical strategy for improving classification quality in semi-supervised learning without additional costs. In the remaining parts of this paper, the virtual sample approach is adopted in semi-supervised fuzzy co-clustering.Sassano [13] proposed two methods for generating virtual samples for text classification based on the following assumption:The category of a document is unchanged even if a small number of words are added or deleted.Because the documents belonging to the same category usually contain several common keywords, deletion or addition of a small number of words is expected not to have a severe impact on classification quality. In [13], two strategies ofDeletion andAddition were considered and were utilized in semi-supervised support vector machine learning of text documents. InGenerateByDeletion, virtual samples were generated by deleting some portions of the original supervised documents and were added to the class of the original ones. On the other hand, inGenerateByAddition, virtual samples were generated by adding a small number of words into the original supervised documents. The words to be added are taken from documents, the label of which is the same as that of the original document.In this paper, the two generative strategies are introduced to semi-supervised fuzzy co-clustering tasks. Assume thatr
i
=
(
r
i
1
,
…
,
r
i
m
) is the cooccurrence information vector on a supervised object i
∈
O
s, where r
i
j
∈
{
0,1
} just represents the appearance/absence of item j in object i. Its virtual copy r
i
′
=
r
i
1
′
,
…
,
r
i
m
′ is given as an additional virtual (supervised) sample to be added to partial supervision after slight revisions based on the two strategies.GenerateByDeletion
(1)
Copyr
i to r
i
′.
(2)
For each itemj with r
i
j
=
1, if r
a
n
d
≤
t then set r
i
j
′
=
0, where r
a
n
d is a function which generates a random number from 0 to 1, and t is a parameter to decide how many elements are deleted.This procedure implies that each virtual object is a virtual copy of the original supervised object having a smaller number of appearances. Virtual objectr
i
′ is almost equivalent to the original r
i with t
→
0 while r
i
′ becomes sparse as t is larger.Here, the detailed process is demonstrated with a toy example using a set of 6 supervised objects (O
s
=
6) shown in Table 1, where each cooccurrence information vector r
i is composed of ten items (m
=
10) and the objects belong to one of two supervised classes (C
=
2). Be noted that class 1 is mainly related to the first 5 items while class 2 is related to others.Table 1
A toy example of supervised objects set.
Object ID:i
Cooccurrence information vector
r
i
=
(
r
i
1
,
…
,
r
i
10
)
Class label
1
(1, 1, 0, 1, 1, 0, 0, 0, 0, 1)
1
2
(1, 0, 1, 1, 0, 1, 0, 0, 0, 0)
1
3
(1, 1, 1, 0, 1, 0, 0, 0, 0, 1)
1
4
(0, 0, 0, 0, 0, 1, 1, 0, 1, 1)
2
5
(0, 0, 0, 0, 1, 0, 1, 1, 0, 0)
2
6
(0, 0, 0, 0, 0, 1, 0, 1, 1, 1)
2UsingGenerateByDeletion, class 1 object r
1 can generate such virtual objects as 1,1
,
0,1
,
0,0
,
0,0
,
0,1, 1,1
,
0,0
,
1,0
,
0,0
,
0,1, or 1,0
,
0,1
,
1,0
,
0,0
,
0,0 with label “1”. Another class 2 object r
5 can generate such as 0,0
,
0,0
,
0,0
,
1,1
,
0,0, 0,0
,
0,0
,
1,0
,
1,0
,
0,0, or 0,0
,
0,0
,
0,0
,
0,1
,
0,0 with label “2”.GenerateByAddition
(1)
Construct the set of all supervised objects, whose class labels are the same as that ofr
i, and generate a temporal item set composed of all items that appeared in the supervised object set.
(2)
Copyr
i to r
i
′.
(3)
For each itemj with r
i
j
=
1, if r
a
n
d
≤
t then randomly select item a from the temporal item set and set as r
i
a
′
=
1.This procedure implies that each virtual object is a virtual copy of the original supervised object having a larger number of appearances. Virtual objectr
i
′ is almost equivalent to the original r
i with t
→
0 while r
i
′ becomes dense as t is larger.In the toy example of Table1,GenerateByAddition first constructs temporal item sets as 1,2
,
4,5
,
10,1
,
3,4
,
6,1
,
2,3
,
5,10 for class 1 and 6,7
,
9,10,5
,
7,8
,
6,8
,
9,10 for class 2. Then, a class 1 object r
1 can generate such virtual objects as 1,1
,
1,1
,
1,0
,
0,0
,
0,1, 1,1
,
0,1
,
1,1
,
0,0
,
0,1, or 1,1
,
1,1
,
1,1
,
0,0
,
0,1 with label “1”. Another class 2 object r
5 can generate such as 0,0
,
0,0
,
1,1
,
1,1
,
0,0, 0,0
,
0,0
,
1,0
,
1,1
,
1,0, or 0,0
,
0,0
,
1,1
,
1,1
,
0,1 with label “2”. Here, it should be noted that r
i
j
′ is always r
i
j
′
=
0 if item j does not appear in all supervised objects of the class.The generated virtual objects are added to the set of supervised objects and are utilized as semi-supervision in semi-supervised learning.


## 4. Numerical Experiment

The classification quality of semi-supervised fuzzy co-clustering with virtual objects is investigated through numerical experiments in this section.Two benchmark datasetsCiteSeer dataset andCora dataset were used, which are available from LINQS webpage of Statistical Relational Learning Group UMD (http://linqs.cs.umd.edu/projects/index.shtml).CiteSeer dataset consists of 3312 scientific publications (objects, n
=
3312), each of which is described by a 0/1-valued word vector (r
i
j
∈
0,1) indicating the absence/presence of 3703 unique words (items, m
=
3703) and is classified into one of six classes (C
=
6).Cora dataset also consists of 2708 scientific publications (objects, n
=
2708) and 1433 unique words (items, m
=
1433) with r
i
j
∈
{
0,1
}, each of which is classified into one of seven classes (C
=
7).The classification quality was investigated through 5-fold cross-validation scheme. In applying semi-supervised fuzzy co-clustering, the dataset was first partitioned into 5 disjoint subsets. Four subsets were utilized for the training set to be used in semi-supervised fuzzy co-clustering and the remaining one subset was used for the test set to validate the classification ability. This training/test trials were iterated 5 times rotating different test subsets. The number of clusters was set as the actual class number; that is,C
=
6 inCiteSeer dataset and C
=
7 inCora dataset. The fuzzification weight was set as λ
u
=
5.0 forCiteSeer dataset and λ
u
=
3.0 forCora dataset, respectively, such that a slightly fuzzier model rather than MMMs with λ
u
=
1.0 can contribute to better performances [16].


### 4.1. Preliminary Experiment

First, a preliminary experiment was performed with the goal of investigating the effects of the amount of supervised objects, where the semi-supervised framework was implementedwithout virtual objects. The amount of supervised objects, which are available in training data, was varied with various ratios of 0
%
,
1
%
,
5
%
,
10
%
,
20
%
,
50
%
,
100
% for two benchmark datasets but the class labels of the remaining objects were withheld to be unknown. “0
%” corresponds to the conventional unsupervised model, which utilizes no supervisions. “100
%” corresponds to the fully supervised model, where all training objects have their class information and C co-cluster models (cluster-wise item memberships) were independently estimated in each class. After co-cluster estimation, the classes of the unsupervised training objects were predicted with the largest memberships. The classes of the test objects were also predicted by the inductive classification scheme of Section 2.The recognition rates are compared in Table2 forCiteSeer dataset and Table 3 forCora dataset. “100
%” for training data are missing as shown with “—” because no unsupervised objects remained in training sets. In bothCiteSeer dataset andCora dataset, the recognition rates of unsupervised co-clustering (0
% supervision) are inferior to those of semi-supervised co-clustering. These results imply that semi-supervised co-clustering is more effective than nonsupervised co-clustering when partial supervisions are available, and the partition quality becomes better as the amount of available supervisions is larger. And the recognition rates in 5
% supervision or larger are almost comparative while the 1
% supervision is significantly inferior to others. Then, if we have 1
% supervision only, it is quite difficult to achieve sufficient recognition rates and we need additional information for improving the partition quality.Table 2
Comparison of recognition rates in preliminary experiment (CiteSeer).
Proportion of supervision
0%
1%
5%
10%
20%
50%
100%
Training
0.528
0.542
0.677
0.694
0.701
0.719
—
Test
0.480
0.488
0.635
0.648
0.646
0.633
0.662Table 3
Comparison of recognition rates in preliminary experiment (Cora).
Proportion of supervision
0%
1%
5%
10%
20%
50%
100%
Training
0.465
0.528
0.664
0.675
0.703
0.735
—
Test
0.416
0.457
0.590
0.592
0.603
0.612
0.619By the way, in these datasets, too many supervisions, such as more than10
%, could not contribute to further improvement of test evaluation. It may be because too many supervisions can bring the overfitting to training samples and cause poor generalization capability. Then, semi-supervised learning is expected to contribute to improving generalization capability rather than fully supervised learning.In the following experiments, a situation with1
% partial supervision was designed for simulating an insufficient partial knowledge case, where the goal is to demonstrate the advantage of additional virtual partial supervision.


### 4.2. Investigation of Effects of Virtual Objects

Next, the effect of virtual objects in semi-supervised fuzzy co-clustering was investigated. In this experiment, the classification abilities of semi-supervised fuzzy co-clustering of the cases with/without virtual objects were compared. Following the result of the above preliminary experiment,1
% supervised objects were included in training sets for semi-supervised co-clustering as a simulation of the situation where an enough amount of supervised objects is not available. The virtual objects were generated from each of these 1
% supervised objects byGenerateByDeletion andGenerateByAddition, and then they were added to training sets and utilized as additional supervision. Supported by the additional information of virtual objects, the partition quality is expected to be improved rather than the original 1
% supervision case.Additionally, the influences of the number of virtual objects were also investigated, where the number of virtual objects was increased by iterating the generative procedures. Furthermore, a hybrid method was also applied using bothGenerateByDeletion andGenerateByAddition, simultaneously. In this hybrid method, at least 2 virtual objects are generated from one supervised object.Both the recognition rates of training and test objects were estimated in the same manner with the preliminary experiment. Tables4 and 5 show the results withCiteSeer dataset. Tables 6 and 7 show the results withCora dataset. Bold types indicate the improved quality rather than the original 1
% supervision case (without virtual objects). “GBD” and “GBA” meanGenerateByDeletion andGenerateByAddition, respectively, and “GBD+GBA” is their hybrid method. In this experiment, the number of virtual objects per supervised object was varied in 1,2
,
5 adopting different random seeds inGenerateByDeletion andGenerateByAddition and varied in 2,4 adopting different random seeds in the hybrid method. Parameter t of addition/deletion rate was varied with t
=
0.01,0.05,0.1,0.2.Table 4
Comparison of recognition rates with and without virtual objects (CiteSeer-training).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.542
1 (GBD)
0.551
0.551
0.548
0.546
1 (GBA)
0.552
0.545
0.549
0.543
2 (GBD)
0.554
0.550
0.554
0.550
2 (GBA)
0.553
0.550
0.548
0.546
2 (GBD+GBA)
0.554
0.554
0.547
0.549
4 (GBD+GBA)
0.541
0.542
0.534
0.533
5 (GBD)
0.534
0.536
0.532
0.544
5 (GBA)
0.504
0.530
0.527
0.532Table 5
Comparison of recognition rates with and without virtual objects (CiteSeer-test).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.488
1 (GBD)
0.498
0.505
0.502
0.493
1 (GBA)
0.501
0.496
0.493
0.488
2 (GBD)
0.502
0.494
0.504
0.492
2 (GBA)
0.503
0.497
0.495
0.494
2 (GBD+GBA)
0.503
0.503
0.494
0.499
4 (GBD+GBA)
0.487
0.486
0.475
0.479
5 (GBD)
0.475
0.477
0.470
0.491
5 (GBA)
0.474
0.469
0.464
0.472Table 6
Comparison of recognition rates with and without virtual objects (Cora-training).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.528
1 (GBD)
0.539
0.540
0.548
0.536
1 (GBA)
0.539
0.546
0.536
0.537
2 (GBD)
0.540
0.536
0.537
0.544
2 (GBA)
0.539
0.541
0.543
0.542
2 (GBD+GBA)
0.542
0.541
0.530
0.541
4 (GBD+GBA)
0.528
0.529
0.531
0.530
5 (GBD)
0.525
0.512
0.529
0.536
5 (GBA)
0.525
0.521
0.503
0.504Table 7
Comparison of recognition rates with and without virtual objects (Cora-test).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.457
1 (GBD)
0.478
0.479
0.489
0.476
1 (GBA)
0.483
0.487
0.474
0.474
2 (GBD)
0.475
0.475
0.475
0.487
2 (GBA)
0.476
0.482
0.476
0.480
2 (GBD+GBA)
0.479
0.479
0.472
0.484
4 (GBD+GBA)
0.462
0.463
0.465
0.467
5 (GBD)
0.440
0.429
0.458
0.460
5 (GBA)
0.441
0.438
0.411
0.409In both training and test cases, the recognition rates of semi-supervised co-clustering with virtual objects are superior to those without virtual objects when the number of virtual objects generated from a supervised objects is relatively small. These results indicate that virtual objects can contribute to improving the classification ability of semi-supervised fuzzy co-clustering when we do not have enough semi-supervision in datasets.However, it seems that too many virtual objects degrade the partition quality. In this experiment, the quality was not improved when 5 or more virtual objects per supervised objects were generated. This indicates that a small number of virtual objects can contribute to improving the quality of semi-supervision while too many virtual objects bring a negative impact on the effect of semi-supervision. It may be because too many virtual objects can bring the overfitting to the virtual objects and cause poor generalization capability. Then, it will be important to generate a proper volume of virtual objects.


### 4.3. Validation of Class-Wise Partition Quality

Finally, the validity of class-wise partition quality is studied, where the best results of the virtual sample approach are compared with the original one given without virtual objects. The best results were given with1
G
B
D
,
t
=
0.05 for the test set ofCiteSeer dataset and 1
G
B
D
,
t
=
0.1 for the test set ofCora dataset, respectively.The quality of result of each class was validated byF-measure. Assume that the classification result in class c is shown as in Table 8. F-measure is defined as follows:(5)F=2PRP+R,where P and R are precision and recall, respectively, as follows:(6)P=αα+β,R=αα+γ.Precision P means the proportion of objects with actual class label c out of objects classified to class c. Recall R means the proportion of objects classified to class c out of objects with actual class label c. F-measure is the harmonic mean of precision P and recall R, and larger F-measure implies better classification ability. Supported by the 5-fold cross-validation scheme, the average of 5 trials is compared.Table 8
Example of classification result of a classc.
Output class label
Positive
Negative
Actual class label
True
α
γ
False
β
δTables9 and 10 show F-measure of each class for test objects. “—” denotes the failure of F-measure calculation caused by missing of P or R, where the clustering model failed to find the corresponding class objects in a trial after maximum memberships classification with very fuzzy boundaries. These tables indicate that the results with virtual objects are superior to those without virtual objects for most classes except forAgents andHCI inCiteSeer dataset andReinforcement_learning inCora dataset. Therefore, virtual objects can mostly contribute to improving partition quality from the viewpoint of class-wise recognition.Table 9
Comparison of classification ability of each class,CiteSeer.
Class label
No virtual objects
{
1
(
G
B
D
), t
=
0.05
}
Agents
0.655
0.651
AI
0.103
0.107
DB
0.346
0.374
IR
0.484
0.512
ML
0.464
0.465
HCI
0.629
0.627Table 10
Comparison of classification ability of each class,Cora.
Class label
No virtual objects
{
1
(
G
B
D
), t
=
0.1
}
Case_based
0.533
0.584
Genetic_algorithms
0.400
0.406
Neural_networks
0.632
0.662
Probabilistic_methods
—
0.416
Reinforcement_learning
—
—
Rule_learning
0.170
0.196
Theory
0.248
0.278


## 4.1. Preliminary Experiment

First, a preliminary experiment was performed with the goal of investigating the effects of the amount of supervised objects, where the semi-supervised framework was implementedwithout virtual objects. The amount of supervised objects, which are available in training data, was varied with various ratios of 0
%
,
1
%
,
5
%
,
10
%
,
20
%
,
50
%
,
100
% for two benchmark datasets but the class labels of the remaining objects were withheld to be unknown. “0
%” corresponds to the conventional unsupervised model, which utilizes no supervisions. “100
%” corresponds to the fully supervised model, where all training objects have their class information and C co-cluster models (cluster-wise item memberships) were independently estimated in each class. After co-cluster estimation, the classes of the unsupervised training objects were predicted with the largest memberships. The classes of the test objects were also predicted by the inductive classification scheme of Section 2.The recognition rates are compared in Table2 forCiteSeer dataset and Table 3 forCora dataset. “100
%” for training data are missing as shown with “—” because no unsupervised objects remained in training sets. In bothCiteSeer dataset andCora dataset, the recognition rates of unsupervised co-clustering (0
% supervision) are inferior to those of semi-supervised co-clustering. These results imply that semi-supervised co-clustering is more effective than nonsupervised co-clustering when partial supervisions are available, and the partition quality becomes better as the amount of available supervisions is larger. And the recognition rates in 5
% supervision or larger are almost comparative while the 1
% supervision is significantly inferior to others. Then, if we have 1
% supervision only, it is quite difficult to achieve sufficient recognition rates and we need additional information for improving the partition quality.Table 2
Comparison of recognition rates in preliminary experiment (CiteSeer).
Proportion of supervision
0%
1%
5%
10%
20%
50%
100%
Training
0.528
0.542
0.677
0.694
0.701
0.719
—
Test
0.480
0.488
0.635
0.648
0.646
0.633
0.662Table 3
Comparison of recognition rates in preliminary experiment (Cora).
Proportion of supervision
0%
1%
5%
10%
20%
50%
100%
Training
0.465
0.528
0.664
0.675
0.703
0.735
—
Test
0.416
0.457
0.590
0.592
0.603
0.612
0.619By the way, in these datasets, too many supervisions, such as more than10
%, could not contribute to further improvement of test evaluation. It may be because too many supervisions can bring the overfitting to training samples and cause poor generalization capability. Then, semi-supervised learning is expected to contribute to improving generalization capability rather than fully supervised learning.In the following experiments, a situation with1
% partial supervision was designed for simulating an insufficient partial knowledge case, where the goal is to demonstrate the advantage of additional virtual partial supervision.


## 4.2. Investigation of Effects of Virtual Objects

Next, the effect of virtual objects in semi-supervised fuzzy co-clustering was investigated. In this experiment, the classification abilities of semi-supervised fuzzy co-clustering of the cases with/without virtual objects were compared. Following the result of the above preliminary experiment,1
% supervised objects were included in training sets for semi-supervised co-clustering as a simulation of the situation where an enough amount of supervised objects is not available. The virtual objects were generated from each of these 1
% supervised objects byGenerateByDeletion andGenerateByAddition, and then they were added to training sets and utilized as additional supervision. Supported by the additional information of virtual objects, the partition quality is expected to be improved rather than the original 1
% supervision case.Additionally, the influences of the number of virtual objects were also investigated, where the number of virtual objects was increased by iterating the generative procedures. Furthermore, a hybrid method was also applied using bothGenerateByDeletion andGenerateByAddition, simultaneously. In this hybrid method, at least 2 virtual objects are generated from one supervised object.Both the recognition rates of training and test objects were estimated in the same manner with the preliminary experiment. Tables4 and 5 show the results withCiteSeer dataset. Tables 6 and 7 show the results withCora dataset. Bold types indicate the improved quality rather than the original 1
% supervision case (without virtual objects). “GBD” and “GBA” meanGenerateByDeletion andGenerateByAddition, respectively, and “GBD+GBA” is their hybrid method. In this experiment, the number of virtual objects per supervised object was varied in 1,2
,
5 adopting different random seeds inGenerateByDeletion andGenerateByAddition and varied in 2,4 adopting different random seeds in the hybrid method. Parameter t of addition/deletion rate was varied with t
=
0.01,0.05,0.1,0.2.Table 4
Comparison of recognition rates with and without virtual objects (CiteSeer-training).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.542
1 (GBD)
0.551
0.551
0.548
0.546
1 (GBA)
0.552
0.545
0.549
0.543
2 (GBD)
0.554
0.550
0.554
0.550
2 (GBA)
0.553
0.550
0.548
0.546
2 (GBD+GBA)
0.554
0.554
0.547
0.549
4 (GBD+GBA)
0.541
0.542
0.534
0.533
5 (GBD)
0.534
0.536
0.532
0.544
5 (GBA)
0.504
0.530
0.527
0.532Table 5
Comparison of recognition rates with and without virtual objects (CiteSeer-test).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.488
1 (GBD)
0.498
0.505
0.502
0.493
1 (GBA)
0.501
0.496
0.493
0.488
2 (GBD)
0.502
0.494
0.504
0.492
2 (GBA)
0.503
0.497
0.495
0.494
2 (GBD+GBA)
0.503
0.503
0.494
0.499
4 (GBD+GBA)
0.487
0.486
0.475
0.479
5 (GBD)
0.475
0.477
0.470
0.491
5 (GBA)
0.474
0.469
0.464
0.472Table 6
Comparison of recognition rates with and without virtual objects (Cora-training).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.528
1 (GBD)
0.539
0.540
0.548
0.536
1 (GBA)
0.539
0.546
0.536
0.537
2 (GBD)
0.540
0.536
0.537
0.544
2 (GBA)
0.539
0.541
0.543
0.542
2 (GBD+GBA)
0.542
0.541
0.530
0.541
4 (GBD+GBA)
0.528
0.529
0.531
0.530
5 (GBD)
0.525
0.512
0.529
0.536
5 (GBA)
0.525
0.521
0.503
0.504Table 7
Comparison of recognition rates with and without virtual objects (Cora-test).
Number of virtual objects per supervised object
t
0.01
0.05
0.1
0.2
No virtual objects
0.457
1 (GBD)
0.478
0.479
0.489
0.476
1 (GBA)
0.483
0.487
0.474
0.474
2 (GBD)
0.475
0.475
0.475
0.487
2 (GBA)
0.476
0.482
0.476
0.480
2 (GBD+GBA)
0.479
0.479
0.472
0.484
4 (GBD+GBA)
0.462
0.463
0.465
0.467
5 (GBD)
0.440
0.429
0.458
0.460
5 (GBA)
0.441
0.438
0.411
0.409In both training and test cases, the recognition rates of semi-supervised co-clustering with virtual objects are superior to those without virtual objects when the number of virtual objects generated from a supervised objects is relatively small. These results indicate that virtual objects can contribute to improving the classification ability of semi-supervised fuzzy co-clustering when we do not have enough semi-supervision in datasets.However, it seems that too many virtual objects degrade the partition quality. In this experiment, the quality was not improved when 5 or more virtual objects per supervised objects were generated. This indicates that a small number of virtual objects can contribute to improving the quality of semi-supervision while too many virtual objects bring a negative impact on the effect of semi-supervision. It may be because too many virtual objects can bring the overfitting to the virtual objects and cause poor generalization capability. Then, it will be important to generate a proper volume of virtual objects.


## 4.3. Validation of Class-Wise Partition Quality

Finally, the validity of class-wise partition quality is studied, where the best results of the virtual sample approach are compared with the original one given without virtual objects. The best results were given with1
G
B
D
,
t
=
0.05 for the test set ofCiteSeer dataset and 1
G
B
D
,
t
=
0.1 for the test set ofCora dataset, respectively.The quality of result of each class was validated byF-measure. Assume that the classification result in class c is shown as in Table 8. F-measure is defined as follows:(5)F=2PRP+R,where P and R are precision and recall, respectively, as follows:(6)P=αα+β,R=αα+γ.Precision P means the proportion of objects with actual class label c out of objects classified to class c. Recall R means the proportion of objects classified to class c out of objects with actual class label c. F-measure is the harmonic mean of precision P and recall R, and larger F-measure implies better classification ability. Supported by the 5-fold cross-validation scheme, the average of 5 trials is compared.Table 8
Example of classification result of a classc.
Output class label
Positive
Negative
Actual class label
True
α
γ
False
β
δTables9 and 10 show F-measure of each class for test objects. “—” denotes the failure of F-measure calculation caused by missing of P or R, where the clustering model failed to find the corresponding class objects in a trial after maximum memberships classification with very fuzzy boundaries. These tables indicate that the results with virtual objects are superior to those without virtual objects for most classes except forAgents andHCI inCiteSeer dataset andReinforcement_learning inCora dataset. Therefore, virtual objects can mostly contribute to improving partition quality from the viewpoint of class-wise recognition.Table 9
Comparison of classification ability of each class,CiteSeer.
Class label
No virtual objects
{
1
(
G
B
D
), t
=
0.05
}
Agents
0.655
0.651
AI
0.103
0.107
DB
0.346
0.374
IR
0.484
0.512
ML
0.464
0.465
HCI
0.629
0.627Table 10
Comparison of classification ability of each class,Cora.
Class label
No virtual objects
{
1
(
G
B
D
), t
=
0.1
}
Case_based
0.533
0.584
Genetic_algorithms
0.400
0.406
Neural_networks
0.632
0.662
Probabilistic_methods
—
0.416
Reinforcement_learning
—
—
Rule_learning
0.170
0.196
Theory
0.248
0.278


## 5. Conclusion

In this paper, the effect of virtual objects in semi-supervised fuzzy co-clustering was demonstrated. Following the previous study [13], two novel procedures for generating virtual objects for cooccurrence data analysis were proposed and their utility was investigated through numerical experiments. In the numerical experiments with two benchmark datasets, the effects of virtual objects in conjunction with the effects of the number of virtual objects per original supervised object were compared and it was indicated that classification quality of semi-supervised fuzzy co-clustering can be improved without additional cost for generating supervised objects by adding several virtual objects while the classification quality can be degraded with too many virtual objects.Future work includes improving the quality of virtual samples. For example, it may be possible to improve the quality of the virtual samples by evaluating the plausibility of the additional samples with some cluster validity measures for fuzzy co-clustering [22]. Furthermore, besides the simple virtual copy of objects, virtual copy of items may be another possible direction.



---

*Source: 5206048-2016-06-23.xml*