ARTICLE IN PRESS
Information Processing and Management xxx (2009) xxx–xxx

Contents lists available at ScienceDirect

Information Processing and Management
journal homepage: www.elsevier.com/locate/infoproman

Field independent probabilistic model for clustering multi-ﬁeld documents
Shanfeng Zhu a,b,*, Ichigaku Takigawa c, Jia Zeng d, Hiroshi Mamitsuka caShanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai 200433, China
School of Computer Science, Fudan University, 220 Handan Road, Shanghai 200433, China
Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji, Kyoto 611-0011, Japand
Department of Computer Science, Hong Kong Baptist University, Kowloon Tong, Hong Kongbca r t i c l e

i n f oArticle history:
Received 13 May 2008
Received in revised form 28 December 2008
Accepted 21 March 2009
Available online xxxx

Keywords:
Document clustering
Finite mixture modelMultivariate Bernoulli modelMultinomial model
Field independent clustering modela b s t r a c t
We propose a new ﬁnite mixture model for clustering multiple-ﬁeld documents, such as
scientiﬁc literature with distinct ﬁelds: title, abstract, keywords, main text and references.
This probabilistic model, which we call ﬁeld independent clustering model (FICM), incorporates the distinct word distributions of each ﬁeld to integrate the discriminative abilitiesof each ﬁeld as well as to select the most suitable component probabilistic model for eachﬁeld. We evaluated the performance of FICM by applying it to the problem of clustering
three-ﬁeld (title, abstract and MeSH) biomedical documents from TREC 2004 and 2005
Genomics tracks, and two-ﬁeld (title and abstract) news reports from Reuters-21578.
Experimental results showed that FICM outperformed the classical multinomial modeland the multivariate Bernoulli model, being at a statistically signiﬁcant level for all thethree collections. These results indicate that FICM outperformed widely-used probabilistic
models for document clustering by considering the characteristics of each ﬁeld. We further
showed that the component model, which is consistent with the nature of the corresponding ﬁeld, achieved a better performance and considering the diversity of model setting also
gave a further performance improvement. An extended abstract of parts of the work presented in this paper has appeared in Zhu et al. [Zhu, S., Takigawa, I., Zhang, S., & Mamitsuka,
H. (2007). A probabilistic model for clustering text documents with multiple ﬁelds. In Proceedings of the 29th European conference on information retrieval, ECIR 2007. Lecture notes incomputer science (Vol. 4425, pp. 331–342)].
Ó 2009 Elsevier Ltd. All rights reserved.

1. Introduction
Document clustering uses unsupervised learning algorithms to group documents into different topics, and performs
exploratory analysis on text collections (Ribeiro-Neto, 1999). Based on the structure of the ﬁnal solution, clustering methods
are divided into partitional (ﬂat) clustering and hierarchical clustering (Jain, Murty, & Flynn, 1999). Similarity-based (discriminative) methods and model-based (generative) methods have been two major strategies for learning clusters (Zhong &Ghosh, 2003). The similarity-based methods, such as the classical k-means algorithm, measure the similarity between document pairs, and group similar documents into the same cluster. In document clustering, each document is usually represented by a vector of weighted selected words according to vector space model, and the similarity between two
documents is calculated by Euclidean distance or the cosine of the angle between two corresponding vectors. On the other* Corresponding author. Address: School of Computer Science, Fudan University, 220 Handan Road, Shanghai 200433, China. Tel.: +86 21 65643786; fax:
+86 21 65654253.
E-mail addresses: zhushanfeng@gmail.com (S. Zhu), takigawa@kuicr.kyoto-u.ac.jp (I. Takigawa), j.zeng@ieee.org (J. Zeng), mami@kuicr.kyoto-u.ac.jp (H.
Mamitsuka).
0306-4573/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ipm.2009.03.005

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
2

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

hand, model-based methods can generate the documents in the same cluster by an identical model, which is usually speciﬁed a priori according to the characteristic of the data set. Without explicitly computing similarity between each document
pair, the model-based methods usually have a less computational complexity compared to the similarity-based methods.
Additionally they provide an intuitive explanation for each cluster through the corresponding model (Zhong & Ghosh, 2003).
Various model-based clustering algorithms have already been proposed to tackle the problem of clustering high dimensional and very sparse text documents. Multivariate Bernoulli model (BM) (McCallum & Nigam, 1998) and multinomial model (MM) (McCallum & Nigam, 1998) are the two most popular models. Zhong and Ghosh (2003) proposed a uniﬁed
framework for probabilistic model-based clustering, which divides a model-based approach into, a model re-estimation stepand data re-assignment step. In their recent study (Zhong & Ghosh, 2005), they compared the performance of three different
model-based clustering algorithms (BM, MM and von Miser–Fisher model Banerjee, Dhillon, Ghosh, & Sra, 2003) with different strategies of assigning documents to clusters. They found that MM and von Miser–Fisher outperformed BM in clustering
15 text datasets. In addition, soft and deterministic annealing (DA) based assignments achieved better performance than
hard assignment in most of datasets. Meila and Heckerman (2001) also compared soft assignment with hard assignment
in multinomial model clustering text documents, and obtained similar results. Rigouste, Cappé, and Yvon (2007) haveproposed multinomial mixture models to cluster documents, where each document is generated by a mixture of several
MMs.
In this paper, we are particularly interested in probabilistic model-based partitional clustering algorithms. To the best of
our knowledge, existing model-based algorithms treat each document as an integrated object. However, the document isusually composed of several distinct ﬁelds in reality. One typical example is the multiple-ﬁeld scientiﬁc document constituted by the title, abstract, keywords, main text, and references. Although each ﬁeld has a common objective of presenting
the document’s topic, it plays a different role, and thus has different distributions over various vocabularies. For example, thetitle is usually very short with around ten words summarizing the topic of the document, while the abstract is much longer
with about 100–200 words brieﬂy describing the motivation of the work, the proposed solution and the experimental result.
Another typical example is the news report, which includes two important ﬁelds: title and body. Similar to the scientiﬁc document, the title in the news report summarizes the topic, and the body gives a more detailed description. Moreover, the ﬁrst
paragraph of the body most likely outlines what has happened in the news.
Here we propose a new probabilistic model referred to as ﬁeld independent clustering model (FICM) to explicitly handle
each ﬁeld in a document separately. FICM is an extension of a ﬁnite mixture model to consider the distinct word distribution
in each ﬁeld. It then allows us to integrate all of them together. Given the cluster that a document belongs to, FICM assumes
that each ﬁeld of this document is conditionally independent. The generative model for each ﬁeld can be different according
to characteristics of data. FICM outperforms other classical model-based methods for the following reasons. First, it integrates the discriminative ability of each ﬁeld by modeling each ﬁeld separately. Second, we can further select the most suitable model for each ﬁeld, and thus strengthen the performance of FICM. The basis of FICM is the conditional independence
assumption of each ﬁeld. This type of conditional independence has been widely employed and has achieved great successes
in information retrieval and machine learning (Domingos & Pazzani, 1996; Lewis, 1998). For instance, the classical MM andBMs are also based on this kind of assumption, where, given the cluster the document belongs to, each word in the document
is generated, being conditionally independent. We stress that the corresponding models work very well in practice, although
this kind of assumption may not follow reality exactly. This is because that the conditional independence assumption may
change the posterior probabilities of each cluster, but the cluster with the maximum posterior probability is often
unchanged.
We conducted extensive and thorough experiments by applying FICM to clustering both scientiﬁc documents and news
reports. We focused on clustering biomedical literature (Jensen, Saric, & Bork, 2006), which has been gaining more and more
attention. MEDLINE (Wheeler et al., 2005) is the largest biomedical literature database for biomedical text mining. For eachdocument (record) in MEDLINE, many distinct ﬁelds are provided, such as title, authors, institution, source, MeSH (Medical
Subject Headings) and abstract, among which title, abstract and MeSH are the most informative. MeSH (Nelson, Schopen,
Savage, Schulman, & Arluk, 2004) is a controlled vocabulary thesaurus deﬁned by the National Library of Medicine for indexing documents in the MEDLINE database. It includes a set of description terms organized in a hierarchical structure. To makea reliable evaluation, we have built 100 datasets randomly generated from TREC 2004 and 2005 Genomics track, respectively, which makes use of 10-years MEDLINE records as the corpus. In addition, we also created 100 news report datasets
based on Reuters-21578 to evaluate the performance of FICM.
We ﬁrst compared the performance of FICM with those of BM and MM in the experiment because of their wide usage. In
the simplest case, the direct extension of BM and MM by applying the same model (BM or MM) for all ﬁelds in FICM has
made a signiﬁcant improvement over the original models in the majority of cases, being statistically signiﬁcant in somecases. For example, over the TREC 2004 Genomics data, the direct extension of MM by FICM outperformed MM in 63 outof total 100 datasets, and 15 of them were statistically signiﬁcant at the 95% conﬁdence level. From this result, we foundthat the signiﬁcant improvement in performance comes from the integration of the discriminative ability of each ﬁeld.
We can then improve the performance of FICM further by assigning a suitable component model to each ﬁeld. For example,
over the TREC 2004 Genomics data, FICM outperformed MM in 98 out of total 100 datasets, and 78 of them were statisticallysigniﬁcant at the 95% conﬁdence level by assigning MM to the abstract ﬁeld and BM to the title and the MeSH ﬁelds. We
further investigated the component model assignment problem to achieve the best clustering performance, by exploring
all eight possible combinations of component assignments in FICM for clustering TREC 2004 and TREC 2005 Genomics data.
Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
3

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

We propose a basic strategy for this problem:‘‘to assign the best model to each ﬁeld, keeping the diversity of each component
model”, and experimental results demonstrated the effectiveness of this strategy. Finally, we explore the effect of a method,which we call ‘Field Weighting’, where a ﬁeld is weighted more by assuming that the words in the ﬁeld appear more often.
Experimental results showed that weighting the title ﬁeld moderately further improves the clustering performance of FICM.
The remainder of the paper is organized as follows. In Section 2, we propose the FICM for clustering multiple-ﬁeld documents. We also discuss the superiority and optimal setting of component model of FICM. Section 3 brieﬂy describes thethree data collections, TREC Genomics track 2004, 2005 and Reuters-21578, as well as the procedure of generating 300 test
datasets from these three collections. We summarize the evaluation criteria and experimental procedures in Section 4. Section 5 presents the detailed experimental results, which demonstrates the superiority of FICM and the effectiveness of component model selection strategy. Section 6 draws conclusions and envisions future works.

2. Field independent clustering model (FICM)
In this section, we ﬁrst brieﬂy introduce the classical BM and MM in turn, and then describe the proposed FICM in detail.
In particular, we discuss the superiority as well as the time and space complexities of FICM.
To ﬁx notation, let D be a set of documents, and d be a document in D. Let Z be the set of classes (topics) of D; z be a class in
Z, and K be the number of the clusters in the dataset. C is the set of ﬁelds, and c is a speciﬁc ﬁeld in C, which could be a title,
abstract or MeSH in this work. Let Dc be a set of documents only considering ﬁeld c, and dc be ﬁeld c (e.g. title) of a documentd. We denote W as the set of words appearing in D; W c as the set of words appearing in Dc , and w as a word. Let N w;d be thefrequency of word w appearing in document d, and N w;dc be the frequency of w appearing in dc . Let Bw;d be 1 if N w;d > 0, otherwise 0. Let Bw;dc be 1 if N w;dc > 0, otherwise 0.
2.1. Mixture model for document clustering
All three models, BM, MM and the proposed FICM, belong to one important category of generative models: mixture models. The basic assumption for generative models for document clustering is that each document d in D is independently andidentically generated by the model (Duda, Hart, & Stork, 2001). That is, the probability of generating document dataset D is
the product of the probability of generating every document d in D:pðDÞ ¼

Y

pðdÞ

d2D

Moreover, the mixture model assumes that, each document is generated by a ﬁnite mixture distribution of the formPpðdÞ ¼ Ki¼1 pi pðd; hi Þ, where pi is the prior probability of cluster i; K is the number of components and pðd; hi Þ is the probability of generating d in cluster i, which depends on a parameter vector hi . Using the notation deﬁned above, the probabilityof generating document d can also be written aspðdÞ ¼

XpðzÞpðdjzÞ

z2Z

The speciﬁc model structure of pðdjzÞ could be further hypothesized, for example, as a multivariate Bernoulli distribution or
multinomial distribution. Then the basic goal of the mixture model is to use all documents in the dataset to estimate themodel parameters, which are usually learned by maximum-likelihood (ML) estimation and the Expectation-Maximization(EM) algorithm (Dempster, Laird, & Rubin, 1977). With these model parameters, we can easily calculate the cluster membership of each document pðzjdÞ. The differences between the BM, MM and FICM models lie in the detailed hypothesis employed
to generate a document, i.e., the speciﬁc probability structure of pðdjzÞ.
2.2. Multivariate Bernoulli model (BM)In BM, each document is represented by a binary vector, which denotes the presence or absence of each word in the document. The basic assumption is that, given the cluster that a document belongs to, the occurrence of each word in the document is assumed to be conditionally independent. For a document in class z, the probability of word w appearing in the
document is pðwjzÞ, while the probability for the absence of w in the document is 1  pðwjzÞ. We then try to maximize
the log-likelihood of generating the whole set of documents D. The ML estimators of this model can be obtained by thefollowing EM algorithm, which repeats the E- and M-steps alternatively until some stopping condition is satisﬁed. In the
M-step, we employ a Laplacian prior to avoid zero probabilities, which is a form of maximum a posteriori (MAP) parameter
estimation (DeGroot, 1970).
Probabilistic structure:LðDÞ ¼X
d2D

log pðdÞ ¼

X
d2D

log

X
z2Z

!
pðzÞpðdjzÞ

¼

X
d2D

log

X
z2Z

pðzÞ

Y

!!
Bw;d

pðwjzÞ

ð1  pðwjzÞÞ

1Bw;dw2dPlease cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
4

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

E-step:pðzjdÞ / pðzÞpðdjzÞ ¼ pðzÞYpðwjzÞBw;d ð1  pðwjzÞÞ1Bw;dw2dM-step: (with Laplacian smoothing)pðzÞ /

XpðzjdÞ

d2DpðwjzÞ ¼

P
1 þ d2D pðzjdÞ  Bw;dP
2 þ d2D pðzjdÞ

2.3. Multinomial model (MM)In contrast to BM, MM assumes that, given the cluster that a document belongs to, every occurrence of each word in the
document is assumed to be conditionally independent. Given a cluster z, it generates each word in a document indepenP
dently with constraint w2W pðwjzÞ ¼ 1. The probabilistic structure of multinomial model and its corresponding E- and Msteps are shown below.
Probabilistic structure:LðDÞ ¼X

log pðdÞ ¼

d2D

X

X

log

!

pðzÞpðdjzÞ

¼

z2Z

d2D

X
d2D

log

X
z2Z

pðzÞ

Y

!!

pðwjzÞNw;dw2d

E-step:pðzjdÞ / pðzÞpðdjzÞ ¼ pðzÞYpðwjzÞNw;dw2dM-step: (with Laplacian smoothing)pðzÞ /

XpðzjdÞ

d2DpðwjzÞ ¼

P
1 þ d2D pðzjdÞ  Nw;dPP
jWj þ w0 2W d2D pðzjdÞ  Nw0 ;d2.4. Field independent clustering model (FICM)
In spite of different assumptions for document representation and generation, both BM and MM treat the occurrence ofeach word at the document level rather than the ﬁeld level. Conversely, the basic idea of FICM is that, given the cluster towhich a document belongs, each component ﬁeld of a document is conditionally independently generated. Although in practice a document may not fully obey this rule, this kind of independence assumption has been widely used successfully inmachine learning and information retrieval (Domingos & Pazzani, 1996; Lewis, 1998). In fact, this kind of assumption is also
employed in the classical BM and MMs, which assume that, given the cluster to which a document belongs, each word in the
document is conditionally independently generated by the underlying probabilistic models. Under the assumption that eachﬁeld is generated independently, the probability of generating a document d is given as follows:pðdÞ ¼

X
z2Z

pðzÞpðdjzÞ ¼

X
z2Z

pðzÞ

Y

pðdc jzÞ

c2C

Here pðdc jzÞ is the probability of generating the ﬁeld c of document d given the underlying cluster z. In the simplest case of
having only two clusters z1 ; z2 , the likelihood-ratio LR of assigning d into z1 rather than z2 can be calculated by the following
formula:LR ¼

pðz1 jdÞ pðz1 Þ pðdjz1 Þ pðz1 Þ Y pðdc jz1 Þ¼

¼
pðz2 jdÞ pðz2 Þ pðdjz2 Þ pðz2 Þ c2C pðdc jz2 Þð1Þ

QSince pðz1 Þ=pðz2 Þ is the ratio of the prior distribution of clusters z1 ; z2 in the dataset, LR is determined by c2C ðpðdc jz1 Þ=pðdc jz2 ÞÞ, which is the product of the discriminative ability of each ﬁeld. If we assume each cluster has the same prior distribution, LR will be a direct integration of the discrimination value of each ﬁeld. Thus the strength of FICM relies on the integration of the clustering ability of each ﬁeld, which can be further improved by choosing a good probabilistic model for eachﬁeld. We can see that FICM is more like a framework, whose implementation depends on the probabilistic model used foreach ﬁeld. In this work, the component models are constrained into BM and MM. However, it can be easily extended to otherprobabilistic models. Let C b be the set of ﬁelds modeled by the BM, and C m be the set of ﬁelds modeled by the MM. We canderive the probabilistic structure of FICM as below, and show the E- and M-steps of the EM algorithm to estimate the parameters of this model.
Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
5

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

Probabilistic structure:LðDÞ ¼X

log pðdÞ ¼

d2D

X

X

log

!
pðzÞpðdjzÞ

¼

z2Z

d2D

X

log

X

pðzÞ

z2Z

d2D

Y

pðdc jzÞ Y!!
pðdc0 jzÞc0 2C m

c2C b

0
0
11
X
Y Y
Y
Y
X
N
@pðzÞ
log @
pðwjz; cÞBw;dc ð1  pðwjz; cÞÞ1Bw;dc ðpðwjz; c0 Þ w;dc0 ÞAA
¼
z2Z

d2D

c0 2C m w2dc0

c2C b w2dc

E-step:pðzjdÞ / pðzÞpðdjzÞ ¼ pðzÞ
¼ pðzÞY Y

Ypðdc jzÞ 

c0 2C

c2C b

pðwjz; cÞ

Y

Bw;dc

0

pðdc jzÞ

m

ð1  pðwjz; cÞÞ1Bw;dc

 Y Y

N

ðpðwjz; c0 Þ w;dc0 Þ

c0 2C m w2dc0

c2C b w2dc

M-step: (with Laplacian smoothing)pðzÞ /

XpðzjdÞ

d2DP
1 þ d2D pðzjdÞ  Bw;dcP
if c 2 C b
2 þ d2D pðzjdÞ
P1 þ d2D pðzjdÞ  Nw;dc0
PPpðwjz; c0 Þ ¼
jW c0 j þ w0 2W d2D pðzjdÞ  Nw0 ;dc0pðwjz; cÞ ¼

if c0 2 C m

In the simplest case, we can use the same probabilistic model for all ﬁelds in FICM, such as C b ¼ C or C m ¼ C, which is called
as Field Independent Clustering Bernoulli Model (FICBM) or Field Independent Clustering Multinomial Model (FICMM),
respectively.
2.5. Superiority of FICM
As shown in Eq. (1), FICM assumes that each ﬁeld has a certain degree of discriminative ability, and the overall clustering
performance could be improved by integrating each ﬁeld’s discriminative information. It is analogous to ensemble learning,which combines a set of individually trained classiﬁers for improving the overall performance on novel examples (Polikar,2006). Ensemble learning has been successfully applied in many classiﬁcation problems, and recently has also been used forclustering problems (Ghosh, 2002). Although both FICM and ensemble learning rely on the component learner (model) toachieve a better performance, the fundamental difference is that FICM directly integrates the discriminative ability of eachcomponent in the framework of the generative model for clustering, while ensemble learning computes the ﬁnal clusteringresult by aggregating a set of different clustering results, which are obtained by each component learner, respectively.
Dietterich (2000) analyzed the success of ensemble learning from the statistical, computational and representational pointof view, and Hansen and Salamon (1990) summarized that the necessary and sufﬁcient condition for the superiority of an
ensemble of classiﬁers over any single classiﬁer is that these component classiﬁers are accurate and diverse. Similarly, we
think that these two conditions for the component model also hold for FICM for achieving the better clustering performance.
Each component model should accurately describe document topic in order to bring certain beneﬁts to the ﬁnal clusteringresult. Additionally, diversity of component models can reduce the bias caused by a single component model, because different component models may assign the same document to different clusters. Therefore, we have the basic principle of thecomponent model design in FICM: ‘‘Assign the best model to each ﬁeld, and in the meanwhile maintain the diversity of eachcomponent model”.
2.6. A ‘Field Weighting’ extension: Weighting ﬁelds differently
To further improve the performance of FICM, we weigh (scale) each ﬁeld by assuming that the words in each ﬁeld appear
more often. This kind of approach has also been employed by other researchers. Recently Li, Xu, and Zhang (2007) have studied the problem of clustering blog documents of the World Wide Web (WWW). Each blog document consists of three components: the title, body and comments of the blog pages. They used the k-means clustering algorithm to cluster blog
documents with assigning different weights to different components. Their experimental results indicated that assigninga large weight value to the blog comments produced a better clustering solution. In their method, each blog document
was still deemed as a vector, and was a weighted sum of all three vectors, representing the title, body and comments, respectively. On the other hand, FICM models keeps ﬁelds being conditionally independent, and integrates them as a whole in aprobabilistic framework. In spite of signiﬁcant difference, the idea of assigning different weights to different ﬁelds can bealso applied in FICM.
For a ﬁeld dc and a cluster z, if the occurrence of each word in dc is multiplied by kc , which can be deemed as weighting
this ﬁeld by kc , then the probability of generating this weighted ﬁeld by MM will be pðdc jzÞkc . In terms of BM, we can assume
Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
6

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

Table 1Time and space complexities.
Model

Time

Space

BM
MM
FICM

OðjZj  jDj  jWjÞOðjZj  jDj  jWjÞOðjCj  jZj  jDj  jWjÞOðjDj  jWjÞOðjDj  jWjÞOðjCj  jDj  jWjÞthat ﬁeld dc happens kc times, instead of only once, then the probability of generating these kc ﬁelds by BM will be pðdc jzÞkc ,too. With this approach, the probability of generating d in FICM will be as follows:pðdÞ ¼

X
z2Z

pðzÞpðdjzÞ ¼

X
z2Z

pðzÞ

Y

pðdc jzÞkc

c2C

Then we can explore the effect of assigning different kc to dc , which we call ‘Field Weighting’.
2.7. Time and space complexities
Table 1 summarizes the time and space complexities of BM, MM and FICM. The most time-consuming part of BM is computing pðzjdÞ and pðwjzÞ, which is OðjZj  jDj  jWjÞ. As for the space complexities, besides the space for input dataset D, weonly need to keep space for pðzÞ; pðzjdÞ and pðwjzÞ. The space complexity is then upper-bounded by
jDj  jWj þ jZj  jDj þ jZj  jWj. Normally, jZj is far smaller than jDj and jWj, and so it can be simpliﬁed as OðjDj  jWjÞ. We caneasily see that MM has the same time and space complexities as BM. For FICM, the most time-consuming part is computingpðzjdÞ and pðwjz; cÞ. Since jW c j 6 jWj, the time complexity of FICM is upper-bounded by OðjCj  jZj  jDj  jWjÞ. Regarding thespace complexities of FICM, we need to keep each ﬁeld of the D and the space for saving pðzÞ; pðzjdÞ, as well as pðwjz; cÞ, which
is upper-bounded by OðjCj  jDj  jWjÞ. Although FICM has slightly higher time and space complexities than BM and MM withconsidering each ﬁeld explicitly, it can take advantage of integrating discrimination ability of each ﬁeld to improve the clustering performance.
3. Datasets
To examine the performance of the proposed FICM, we have built evaluation datasets from three collections:(1) TREC Genomics track 2004 and 20051 (Hersh et al., 2004; Hersh, Cohen, Bhupatiraju, Johnson, & Hearst, 2005);(2) Reuters-21578 news collection.2
The Genomics track of the TREC3 conference provides a testbed and benchmark for comparing different information retrieval systems for biomedical documents. There are totally 4,591,008 documents (MEDLINE records from year 1994 to 2003) in the
TREC Genomics track corpus for 2004 and 2005. In the 2004 track, biologists deﬁned 50 relatively independent topics, while inthe 2005 track, biologists must additionally follow a semantic template to deﬁne 50 topics. In both tracks, biologists manuallyassessed the relevance of retrieved records, and obtained a set of reliably relevant documents for each topic. The Reuters-21578
news collection contains 21,578 news stories which appeared on the Reuters newswire in 1987. There are totally 135 topics, andone document may belong to multiple topics.
In the Genomics track 2004 and Genomics track 2005, we extracted not only the whole text of each record, but also three
distinct ﬁelds (title, abstract and MeSH) of each record. In Reuters-21578, we used the ﬁrst paragraph of the news body as a
distinct ﬁeld, because it usually gives the summary of the news analogous to the abstract in a scientiﬁc document. Thus we
extracted two distinct ﬁelds for clustering, title and abstract (the ﬁrst paragraph of the news body). In all three sources, we
did not consider those topics associated with only nine or fewer documents, and removed those documents with empty
ﬁelds or relevant to more than one topic, and ﬁnally retained 39 topics (4400 documents) in the Genomics track 2004, 24
topics (2317 documents) in the Genomics track 2005 and 42 topics (8574 documents) in the Reuters-21578, respectively.
In the Reuters-21578, the two largest topics, ‘‘earn” and ‘‘acq”, have respectively 3735 and 2125 documents, which are muchlarger than the other 40 topics which have at most 355 relevant documents. To avoid the dominance of these two topics inthe dataset, we only retained all other 40 topics (2714 documents) for evaluation.
From these ﬁltered topics, we have built two sets of datasets based on the TREC Genomics tracks of 2004 and 2005, andone set of datasets based on the Reuters-21578, which are called as the Genomics2004, Genomics2005 and Reuters1987 collections, respectively. For a robust comparison, each collection includes 100 datasets, which were generated from the corresponding source by randomly choosing three or more (no more than 12) topics. With a speciﬁc number of topics, webuilt 10 different datasets. Compared with other randomly generated document datasets from MEDLINE (Yoo & Hu,1

http://ir.ohsu.edu/genomics/.
http://www.daviddlewis.com/resources/testcollections/reuters21578/.
3
http://trec.nist.gov/.
2

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
7

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

Table 2
Summary of statistical characteristics of Genomics2004, Genomics2005 and Reuters1987. The ‘min of all 100’, ‘mean of all 100’ and ‘max of all 100’ refer to theminimum, average and maximum value of each feature, respectively, out of all 100 datasets in each collection.
Collection

Data

Nd

W

K

Nl

Balance

Nt

Na

Nm

Wt

Wa

Wm

Genomics2004
Genomics2004
Genomics2004
Genomics2004

T200412amin of all 100
max of all 100
mean of all 100

1687
133
1960
860.2

4288
879
4630
2812.3

12
3
12
7.5

165.3
138.3
175.9
161

0.0281
0.02160.56250.1056

9.1
6
9.8
8.3

113.9
93.4
120.3
109.3

40.4
33.2
48
40.9

942
97
1044
543.7

3951
774
4290
2597.1

1230
251
1381
790.1

Genomics2005
Genomics2005
Genomics2005
Genomics2005

T200512amin of all 100
max of all 100
mean of all 100

1163
71
1469
690.7

3153
570
3442
2214.4

12
3
12
7.5

155
132.8
165.1
148.9

0.029
0.02110.66670.0711

8.8
5.6
8.9
8.0

106.2
92.8
111.3
103

38.4
28.3
41.1
35.3

662
60
727
400.6

2906
485
3168
2025.3

890
158
950
608.8

Reuters1987
Reuters1987
Reuters1987
Reuters1987

R198712amin of all 100
max of all 100
mean of all 100

714
52
1232
500.8

915
116
1280
622.3

12
3
12
7.5

19
12
19.6
18.0

0.03380.02820.619
0.1002

4
2.3
4.4
3.8

14.3
8.6
14.6
13.4

–
–
–
–

322
2.6
529
223.8

858
107
1198
575

–
–
–
–

2006), Genomics2004 and Genomics2005 are of high quality, where the topic of each document was assessed manually by
the biologists. In the pre-processing step, we removed stop words, carried out case folding and tokenized the documents
using the Porter’s stemming algorithm (Porter, 1980). Similar to Zhong and Ghosh (2005), we eliminated any (stemmed)
word that appears in less than three documents. The same procedure was also applied to the words in each ﬁeld.
Table 2 summarizes the statistical characteristics of Genomics2004, Genomics2005, and Reuters1987, where N d is thenumber of documents, W is the number of distinct words (tokens), K is the number of classes, N l is the average numberof words in each document, balance is the size ratio of the smallest class to the largest class, N t ðN a ; N m Þ is the average numberof words in the title (abstract, MeSH) ﬁeld, and W t ðW a ; W m Þ is the number of distinct words in the title (abstract, MeSH) ﬁeld.
Each dataset in Genomics2004 and Genomics2005 is named by combining an initial alphabet ‘‘T”, the year, the number of
topics, and the order of the dataset. For example, ‘‘T200412a” represents the ﬁrst dataset with twelve topics generated fromthe TREC Genomics track 2004. Similarly, each dataset in Reuters1987 is named by combining the initial alphabet ‘‘R”, theyear, the number of topics, and the order of this dataset. Table 2 shows that Genomics2004 varies greatly in some importantcharacteristics: the number of documents varies from 133 to 1960, the number of classes from 3 to 12, the balance from
0.022 to 0.563 and the number of distinct words in each dataset from 879 to 4630. Compared with Genomics2004, Genomics2005 is slightly smaller but also has large variations. In contrast, Reuters1987 is the smallest in terms of the average
length and the average number of documents in each dataset. The number of documents ranges from 52 to 1232, the numberof distinct words from 116 to 1280, and the balance from 0.0282 to 0.619. Overall, the diversity of Genomics2004, Genomics2005 and Reuters1987 makes them very suitable for comparing different clustering algorithms.
4. Evaluation criteria and experiment design
In this section, we describe the criterion based on the normalized mutual information to evaluate the proposed FICM. Furthermore, we discuss how to compare the performance of different models using our new performance representation form,
S-Pair.
4.1. Normalized mutual information (NMI)
We used external measures as evaluation criteria in document clustering. External measures evaluate the clustering result using the correct (true) class labels of the dataset, which is not provided during the clustering processes. Well-known
external measures include purity, average entropy, F-measure and mutual information. Ghosh (2003) compared these external measures, and showed that mutual information is a superior measure over other external measures. Both purity andaverage entropy favor large number of clusters, while F-measure is biased towards coarser clusterings. Therefore, we used
the normalized mutual information (NMI) to compare the performance of FICM with the other models. Normalized mutual
information (NMI) is calculated by the following formula:IðX; YÞ
NMI ¼ pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃHðXÞ  HðYÞwhere X and Y are the predicted clusters and the correct class labels, respectively, IðX; YÞ is the mutual information between
X and Y, and HðXÞ and HðYÞ are the entropy of X and Y, respectively. In practice, Zhong and Ghosh (2003, 2005) proposed asample estimate to calculate the NMI,P

h;l nh;l log

nnh;lnh nl



NMI ¼ qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ

ﬃ
P
nh P
nl
h nh log n
l nl log nð2ÞPlease cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
8

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

where n is the total number of documents in the whole collection, nh is the number of documents in class h (standard), nl is
the number of documents in cluster l (predicted), and nh;l is the number of documents in both class h and cluster l. The NMI
value ranges from zero to one, where an NMI value of zero means that the result is equal to random partitioning, and an NMI
value close to one means that the result is almost identical to the true class labeling.
4.2. Model comparison: S-Pair
In Genomics2004 and Genomics2005, each document consists of three ﬁelds, title, abstract and MeSH, while in Reuters1987, each document consists of two ﬁelds, title and abstract. When the BM is applied on the documents with title ﬁeld
only, abstract ﬁeld only, MeSH ﬁeld only and all ﬁelds, we denote these models as B-title, B-abstract, B-mesh and B-whole,respectively. When the MM is used, the corresponding models are denoted as M-title, M-abstract, M-mesh and M-whole,respectively. As described in Section 2, FICM is a framework where the speciﬁc implementation depends on the component
models assigned to each ﬁeld. Since we focus on two popular probabilistic models, BM and MM, there are 23 ¼ 8 possible
implementations of FICM for clustering datasets in Genomics2004 and Genomics2005, and there are 22 ¼ 4 possible implementations of FICM for clustering datasets in Reuters1987. To make a distinction, each speciﬁc implementation of FICM isdenoted by a combination of an initial alphabet F, the character ‘-’ and a number of alphabets, where each representing amodel used in this ﬁeld. The character ‘b’ and ‘m’, represent the BM and the MM, respectively. For example, ‘F-mmm’ stands
for a speciﬁc implementation of FICM, FICMM, where all three ﬁelds are modeled by MM. Table 3 lists the abbreviations ofsome models and their corresponding meanings.
To make a fair comparison, the same stop criterion for the EM algorithm was adopted for all the models: the relativechange of the maximum log-likelihood in two consecutive iteration is less than 0.001% or the number of iterations of the
EM algorithms exceeds 30. In addition, the number of classes, K, was given as a priori parameter in all the experiments. Moreover, to reduce the possible bias caused by a random initial partition, each experiment was carried out 100 times, and themeans, standard deviations and the paired t-test were used to compare different models. The comparison of two different
models, h1 and h2 , is carried on clustering 100 datasets of Genomics2004, Genomics2005, and Reuters1987. The numberof the datasets that h1 outperforms h2 is denoted by N h1 >h2 , in which the number of those datasets with statistically significant improvement at the 95% conﬁdence level is denoted by N h1 >h2 ;þ . Similarly, the number of datasets that h2 outperforms
h1 is denoted by N h2 >h1 , in which the number of datasets with statistically signiﬁcant improvement at the 95% conﬁdence
level is denoted by N h2 >h1 ;þ . Thus for comparing h1 with h2 on each collection, we can obtain a pair of two numbers
 N
Nh >h ;þh >h ;þ
þ N1h >h2 ;  N2h >h1 . Here we call it Signiﬁcant Pair, for short S-Pair, of comparing h1 with h2 . Please note
1

2

2

1

N h1 >h2 þ N h2 >h1 ¼ 100 because there are 100 datasets in each collection. We can say that h1 outperforms h2 (h1 > h2 ) if andonly if N h1 >h2 > N h2 >h1 and N h1 >h2 ;þ P N h2 >h1 ;þ .
To show the integrated power of FICM, we divided the datasets in two types in the following manner: We ﬁrst call thedatasets on which FICM outperforms the classical model (BM or MM on the entire text) superior datasets and the other datasets are called neutral datsets. If a component model can also obtain a better clustering result over superior datasets than overneutral datasets, we may say that the performance of FICM can be achieved by the good performance of this component
model. To check this hypothesis, we focused on two simplest implementations of FICM: FICBM and FICMM. For example,
for FICBM, we used the performance of B-whole (BM over the entire text) as the baseline, and the power of each component
model was represented by a relative measure, the ratio of the clustering performance of the component model to the baseline, which we call the relative discriminative ability (RDA) of this ﬁeld hereafter. That is, for the title ﬁeld, RDAtitle can be comNMIB-title. Similarly, for this example, we can compute RDAabstract and RDAMeSH for the abstract ﬁeld and MeSH. We canputed as NMI
B-wholeTable 3Model abbreviations.
Abbreviation

Meaning

BM
MM
FICM
FICBM
FICMM
B-titleB-abstract
B-mesh
B-wholeM-titleM-abstract
M-meshM-whole
F-bmb
F-bbb
F-mb

Multivariate Bernoulli modelMultinomial model
Field independent clustering model
Applying BM to all ﬁelds in FICM
Applying MM to all ﬁelds in FICM
Using BM on title only
Using BM on abstract only
Using BM on MeSH only
Using BM on whole text
Using MM on title only
Using MM on abstract only
Using MM on MeSH only
Using MM on whole text
Applying BM to the ﬁrst and third ﬁeld, and MM to the second ﬁeld in FICM
Applying BM to all three ﬁelds in FICM
Applying MM to the ﬁrst ﬁeld, and BM to the second ﬁeld in FICM

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
9S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxxþRDAMeSH
then compute the total RDA over all ﬁelds as RDA ¼ RDAtitle þRDAabstract
for the case of three ﬁelds: title, abstract and MeSH.
3
This is possible for FICMM by using M-whole as the baseline. Similarly, for the case of only two ﬁelds, title and abstract, thisabstract. We denote the RDA for superior datasets and neutral datasets by RDAsup and RDAneu ,RDA is given as RDA ¼ RDAtitle þRDA2
respectively.

5. Experimental results
Experimental results include two parts. In the ﬁrst part, we compared BM and MM with their direct extension, FICBM and
FICMM. We found that FICBM (FICMM) outperformed B-whole (M-whole) consistently in all three collections, which demonstrated the effectiveness of FICM, even in the simplest conﬁguration. In the second part, we further examined different combinations of component models in FICM, and obtained some insights into the component ﬁeld conﬁgurations.
5.1. FICBM and FICMM
5.1.1. Comparison of B-title, B-abstract, B-mesh, B-whole and FICBM
We compared the performance of the BM-based clustering models. In Genomics2004 and Genomics2005, FICBM is denoted by F-bbb since it consists of three different ﬁelds, while in Reuters1987 with two different ﬁelds, FICBM is denotedby F-bb. Table 4 illustrates the performance of different models in terms of NMI in Genomics2004, Genomics2005, and Reuters1987. For each collection, it presented the result on one example dataset in the collection, as well as the average result
over all 100 datasets. For the example dataset (or the average of all datasets), we used bold face to highlight the model withthe highest NMI. Experimental results showed that FICBM achieved the highest average NMI (0.756, 0.723 and 0.497) in all
three collections, respectively. Table 5 presented a comparison of these models by the paired t-test for indicating the statistical signiﬁcance of improvement. The experimental results showed that FICBM outperformed B-whole consistently for all
three collections, with S-Pair (+41/80, 0/20) in Genomics2004, S-Pair (+63/95, 1/5) in Genomics2005 and S-Pair (+73/89, 1/11) in Reuters1987.
5.1.2. Comparison of M-title, M-abstract, M-mesh, M-whole and FICMM
We compared the performance of MM-based clustering models and FICMM: F-mmm (or F-mm for Reuters1987). Table 6
presents the experimental result of these models on one example dataset and the average of all 100 datasets for Genomics2004, Genomics2005 and Reuters1987. We can see that FICMM achieved higher NMI than M-whole in all three collections.
For example, with Genomics2004, FICMM achieved the highest NMI of 0.740, which was followed by M-whole of 0.736. In
addition, interestingly, for Genomics2005 and Reuters1987, M-title (using MM on the title only) obtained the highest average
NMI out of all ﬁve models, which suggests that only a few distinguished words in the title are effective for document clustering. We further compared these models in terms of the signiﬁcance of improvement by paired t-test. Table 7 gives theresults for each collection. We found that FICMM outperformed M-whole consistently in all three collections, with S-Pair(+15/63, 0/37) in Genomics2004, S-Pair (+12/67, 3/33) in Genomics2005 and S-Pair (+69/94, 0/6) in Reuters1987.
5.1.3. The superiority of FICBM over B-whole and FICMM over M-whole relies on the integration of discriminative ability of eachcomponent model
In the above two rounds of experiments, the direct extension of BM and MM by FICM (FICBM and FICMM) outperformed
the corresponding original models (B-whole and M-whole) signiﬁcantly, and the improvement of FICBM over B-whole wasTable 4
Performance of B-title, B-abstract, B-mesh, B-whole and FICBM (F-bbb for Genomics2004 and Genomics2005 or F-bb for Reuters1987) in terms of NMI(mean ± standard deviation).
Collection

Data

B-titleB-abstract

B-mesh

B-wholeF-bbb (F-bb)

Genomics2004
Genomics2004
Genomics2005
Genomics2005
Reuters1987
Reuters1987

T200412a
Mean of all 100
T200512a
Mean of all 100
R198712a
Mean of all 100.630 ± .03.628 ± .06.671 ± .05.671 ± .06.300 ± .07.446 ± .07.734 ± .04.723 ± .07.697 ± .05.693 ± .07.359 ± .06.466 ± .07.739 ± .03.715 ± .05.634 ± .03.656 ± .05––.756 ± .04.743 ± .06.706 ± .04.697 ± .07.357 ± .05.460 ± .07.782 ± .04.756 ± .06.745 ± .04.723 ± .07.369 ± .07.497 ± .08

Table 5The comparison of B-title, B-abstract, B-mesh, B-whole and FICBM(F-bbb in Genomics2004 and Genomics2005 or F-bb in Reuters1987) in terms of S-Pair.
Collection

B-whole > B-title

B-whole > B-abstract

B-whole > B-mesh

F-bbb (F-bb) > B-whole

Genomics2004
Genomics2005
Reuters1987(+92/95, 2/ 5)(+53/68, 25/32)(+57/70, 25/30)(+53/80, 6/20)(+22/56, 13/44)(+ 2/36, 11/64)(+61/74, 13/26)(+75/84, 9/16)–(+41/80, 0/20)(+63/95, 1/ 5)(+73/89, 1/11)

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
10

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

Table 6
Performance of M-title, M-abstract, M-mesh, M-whole and FICMM (F-mmm in Genomics2004 and Genomics2005 or F-mm in Reuters1987) in terms of NMI(mean ± standard deviation).
Collection

Data

M-titleM-abstract

M-meshM-wholeF-mmm (F-mm)

Genomics2004
Genomics2004

T200412a
Mean of all 100.740 ± .03.709 ± .05.781 ± .03.719 ± .06.716 ± .03.676 ± .05.790 ± .03.736 ± .06.795 ± .04.740 ± .06

Genomics2005
Genomics2005

T200512a
Mean of all 100.755 ± .02.723 ± .05.754 ± .03.702 ± .06.603 ± .03.588 ± .05.758 ± .03.702 ± .06.758 ± .03.705 ± .06

Reuters1987
Reuters1987

R198712a
Mean of all 100.413 ± .04.493 ± .05.423 ± .03.482 ± .06––.407 ± .03.473 ± .06.430 ± .03.492 ± .06

Table 7The comparison of M-title, M-abstract, M-mesh, M-whole and FICMM(F-mmm in Genomics2004 and Genomics2005 or F-mm in Reuters1987) in terms of S-Pair.
Collection

M-whole > M-title

M-whole > M-abstract

M-whole > M-mesh

F-mmm (F-mm) > M-whole

Genomics2004
Genomics2005
Reuters1987(+64/76, 15/24)(+17/26, 57/74)(+23/33, 50/67)(+61/89, 0/11)(+13/50, 10/50)(+ 8/32, 35/68)(+89/94, 1/6)(+95/97, 2/3)–(+15/63, 0/37)(+12/67, 3/33)(+69/94, 0/ 6)

especially remarkable. As discussed in Section 4, we can compare RDAsup of FICBM (and FICMM) with RDAneu of that. Throughthis comparison, we can check if there is a signiﬁcant difference between the superior and neutral by paired t-test. Note that
we used RDA of the three ﬁeld case for Genomics2004 and Genomics2005, while that of the only two ﬁeld case for Reuters1987. As shown in Table 8, the upper three rows are by FICBM and the lower three rows are by FICMM. In all six cases,
the RDAsup was higher than RDAneu . Moreover, in four of them, the difference was statistically signiﬁcant at the 95 % conﬁdence level (p-value less than 0.05), and one of them obtained a p-value of 0.081, which was close to 0.05. The only exception
was the case of FICMM on Genomics2005 with a p-value of 0.43. The reason may be due to the dependence of different ﬁelds
and the unbalanced contribution of each ﬁeld in the FICMM. In contrast to BM, MM considers all occurrences of every word,
which means that longer ﬁelds, such as abstracts, contributed much more than shorter ﬁelds, such as titles, in the performance of FICMM. Over all, we believe that the signiﬁcant performance improvement of FICBM over BM on the entire text
(B-whole) and FICMM over MM on the entire text (M-whole) comes from the integration of discriminative ability of eachcomponent model.
5.2. The combination of different component models in FICM
As discussed in Section 2, to make good use of FICM, the component model should be accurate and diverse, which requires
us to assign the best model to each ﬁeld and to also maintain a diversity of component models. To examine this strategy, we
shall explore the effect of different combination of component models in FICM. Since Reuters1987 is too simple with only
two ﬁelds and four possible combinations in FICM, we focus on exploring different combinations of component models
on clustering datasets in the data collections Genomics2004 and Genomics2005, which consists of three ﬁelds and eight possible combinations. First, we identiﬁed which model is more suitable, BM or MM, for each independent ﬁeld. Second, we
checked if the performance of FICM could be improved by conﬁguring a better model to one ﬁeld with ﬁxed model settings
on the other two ﬁelds. Finally, we discussed the optimal conﬁguration of FICM compared with classical BM and MM.
5.2.1. Identifying the best model for each ﬁeld independently
To identify the best model for each ﬁeld, it is necessary to know both the important characteristics of each ﬁeld and thestrengths (and weaknesses) of each model. Alternatively, we may carry out some preliminary experiments on a small training dataset to elucidate an appropriate model for each ﬁeld. In Genomics2004 and Genomics2005, considering the distinct

Table 8The comparison of relative discriminative ability (RDA) of FICBM (FICMM) between superior datasets and neutral datasets in Genomics2004, Genomics2005
and Reuters1987.
Collection

Model

RDAsup (mean ± standard deviation)

RDAneu (mean ± standard deviation)

p-Value

Genomics2004
Genomics2005
Reuters1987

FICBM
FICBM
FICBM

.944 ± .04.982 ± .05
1.011 ± .11.914 ± .04.943 ± .04.928 ± .11

6.51e004
2.09e004
2.80e003

Genomics2004
Genomics2005
Reuters1987

FICMM
FICMM
FICMM

.971 ± .03.966 ± .03
1.042 ± .05.947 ± .04.957 ± .04
1.023 ± .05

4.20e002
0.430
0.081

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

11

features of the MeSH and title ﬁelds, we believe that the best model for MeSH and the title ﬁeld would be BM and MM, withthe following reasons. The MeSH terms are originally organized in a hierarchal structure for indicating the document theme.
Here, both BM and MM treat them ﬂatly without considering this hierarchical information. Some general terms, such as‘‘human”, ‘‘protein” and ‘‘genetics”, would appear in the MeSH ﬁeld very frequently, which actually brings little information
for clustering. The situation will become even worse for MM since it favors frequent terms for clustering. Conversely, by considering binary occurrences only, BM would be more robust against this situation. On the other hand, for the title ﬁeld, words
that appear usually are very informative and highly related for expressing the document topic. In this situation, MM, inwhich the probability for generating each distinct word sums to 1, adds some constraints to those relevant informative
words, and would be more appropriate than BM since the latter deals with the presence/absence of each word independently. To examine these hypotheses, we compared the performance of MM and BM on the title and MeSH ﬁelds using apaired t-test. As shown in Table 9, the assignment of these models is justiﬁed through the experimental results for both collections. For example, for Genomics2004, B-mesh outperformed M-mesh with S-Pair (+81/94, 5/6), and M-title outperformed
B-title with S-Pair (+89/92, 5/8).

Table 9The comparison of B-title with M-title, B-abstract with M-abstract and B-mesh with M-mesh in terms of S-Pair.
Data

M-title > B-title

B-abstract > M-abstract
K<8M-abstract > B-abstract
KP8

B-mesh > M-mesh

Genomics2004
Genomics2005(+89/92, 5/ 8)(+73/85, 6/15)(+35/43, 4/ 7)(+24/29, 12/21)(+35/37, 10/13)(+34/40, 7/10)(+81/94, 5/ 6)(+97/97, 1/3)

The comparsion of B−abstract and M−abstract on Genomics2004

The number of datasets with better performance10
B−abstract
M−abstract

9
8
7
6
5
4
3
2
1
0

3

4

5

6

7

8

9

10

11

12

K
The comparsion of B−abstract and M−abstract on Genomics2005

The number of datasets with better performance10
B−abstract
M−abstract

9
8
7
6
5
4
3
2
1
0

3

4

5

6

7

8

9

10

11

12

K

Fig. 1. Comparison of B-abstract with M-abstract for Genomics2004 and Genomics2005: the number of datasets with K topics where one model outperforms
another statistically signiﬁcant.

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
12S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

For the abstract ﬁeld, we speculate that BM would be more suitable on small datasets with few topics, while MM outperforms multivariate models deﬁnitely on large datasets, which comes from McCallum and Nigam’ﬁndings (1998). They compared MM with BM for naive Bayes text classiﬁcation, and found that BM performed very well on datasets with a limited
vocabulary, while MM usually performed better on datasets with a larger vocabulary size. This ﬁnding is also consistent with
our experiments which compared B-abstract with M-abstract. As illustrated in Fig. 1, B-abstract usually outperformed M-abstract on datasets with small number of topics, while M-abstract outperformed B-abstract on datasets with more topics. For
example, for both Genomics2004 and Genomics2005, M-abstract outperformed B-abstract statistically signiﬁcantly for all 10
datasets with 12 topics. The crossover of B-abstract and M-abstract with similar performance happens when the number of
topics in the dataset is around 8. And thus, by dividing the datasets into two groups: K P 8 or K < 8, we summarized thecomparison of B-abstract and M-abstract in Table 9.
Over all, in Genomics2004 and Genomics2005, the best model for MeSH and title ﬁelds would be BM and MM, respectively. And for the abstract ﬁeld, BM is more suitable than MM when the number of topics in the datasets is smallðK < 8Þ. Otherwise, MM is more suitable.
5.2.2. Replacing the model setting for only one ﬁeld in FICM
We compared the performance of different models with paired t-test by changing the model for only one component ﬁeld
and ﬁxing the models for the other two ﬁelds. The hypothesis is that assigning a better model to a ﬁeld in FICM will improvethe clustering performance.
As shown in Table 10, changing the component model from MM to BM for the MeSH ﬁeld improved the clustering performance signiﬁcantly for all combinations. For example, for Genomics2005, F-mmb outperformed F-mmm with S-Pair (+67/96, 0/4), and F-bbb outperformed F-bbm with S-Pair (+54/94, 2/6). Moreover, Table 11 shows consistently that replacing
MM with BM for the abstract would increase clustering performance when the number of topics is less than 8. The clustering
performance decreased when the number of topics is equal to or larger than 8. For example, for Genomics2004, F-bbb outperformed F-bmb with S-Pair (+34/43, 2/7) when K < 8 as shown in Table 11, while F-bmb outperformed F-bbb with S-Pair(+37/41, 6/9) when K P 8 in Table 12. These experimental results are highly consistent with our hypothesis that applying abetter component model will improve the clustering performance of FICM.
Compared with the abstract and MeSH, the effect from changing the model for the title is much weaker and more complicated as illustrated in Table 13. For example, for Genomics2005, F-mbm differed slightly from F-bbm with S-Pair (+5/53, 4/47), and F-mbb differed slightly from F-bbb with S-Pair (+1/52, 1/48). This indicates that model selection for the abstract or MeSH is more important than model selection for the title with respect to the performance of FICM. A main reason
may be that the title is much shorter compared with the other two ﬁelds, so that the performance of FICM is dominated by
the component model setting in abstract and MeSH. Another reason would be the relative high correlation between the title
and the abstract. One popular measure for correlation is the cosine of the angle between two corresponding vectors (cosine
similarity). For each dataset in the Genomics2004 and Genomics2005 collections, the cosine similarities between any two
ﬁelds for both MM and BM are computed, and averaged in Table 14. We displayed the highest correlated pair in every combination of data collection and model in boldface. We see that in all four cases, the title–abstract pair always obtained thelargest similarity, which was especially signiﬁcant in MM. For example, in Genomics2004, the average cosine similarity between the title and the abstract in MM was 0.46, while the average cosine similarity between the title and the MeSH was

Table 10
Changing model settings on the MeSH ﬁeld in FICM.
Data

F-bbb > F-bbm

F-mbb > F-mbm

F-bmb > F-bmm

F-mmb > F-mmm

Genomics2004
Genomics2005(+30/74, 0/26)(+54/94, 2/6)(+27/78, 2/22)(+51/90, 0/10)(+38/91, 0/9)(+46/97, 0/3)(+47/90, 0/10)(+67/96, 0/4)

Table 11
Changing model settings on the abstract ﬁeld in FICM when K < 8.
Data

F-bbb > F-bmb

F-bbm > F-bmm

F-mbm > F-mmm

F-mbb > F-mmb

Genomics2004
Genomics2005(+34/43, 2/7)(+25/33, 9/17)(+31/43, 3/7)(+18/30, 10/20)(+29/44, 2/6)(+21/30, 11/20)(+21/31, 9/19)(+27/42, 1/8)

Table 12
Changing model setting on the abstract ﬁeld in FICM when K P 8.
Data

F-bmb > F-bbb

F-bmm > F-bbm

F-mmm > F-mbm

F-mmb > F-mbb

Genomics2004
Genomics2005(+37/41, 6/9)(+40/44, 2/6)(+32/41, 6/9)(+37/44, 3/6)(+21/34, 9/16)(+27/34, 8/16)(+38/44, 4/6)(+31/39, 5/11)

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
13

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx
Table 13
Changing model setting on the title ﬁeld in FICM.
Data

F-mbm > F-bbm

F-mbb > F-bbb

F-bmm > F-mmm

F-bmb > F-mmb

Genomics2004
Genomics2005(+9/65, 4/35)(+5/53, 4/47)(+8/60, 1/40)(+1/52, 3/48)(+15/74, 0/26)(+35/82, 0/18)(+13/76, 3/24)(+13/68, 0/32)

Table 14
The average cosine similarity (mean ± standard deviation) between any two ﬁelds over all datasets in Genomics2004 and Genomics2005.
Data

Model

Title–abstract

Title–MeSH

Abstract–MeSH

Genomics2004
Genomics2004

BM
MM.30 ± .01
.46 ± .02

.26 ± .02
.23 ± .03

.25 ± .01
.25 ± .01

Genomics2005
Genomics2005

BMMM.31 ± .01
.46 ± .03

.27 ± .03
.23 ± .03

.24 ± .02
.24 ± .02

0.23, and the average cosine similarity between the abstract and the MeSH was 0.25. In this case, assigning different models
to the title and the abstract, respectively, will usually obtain good clustering results, which conﬁrms the importance of thediversity among component models. For example, as shown in Table 13, for Genomics2004, F-bmb outperformed F-mmb
with S-Pair (+13/76, 3/24), and F-bmm outperformed F-mmm with S-Pair (+15/74, 0/26), and for Genomics2005 F-bmb
outperformed F-mmb with S-Pair (+13/68, 0/32), and F-bmm outperformed F-mmm with S-Pair (+35/82, 0/18).
In short, assigning a better model to a ﬁeld in FICM usually improves the clustering performance, with exception on thetitle ﬁeld, which may due to its short length and relative high correlation with the abstract.
5.2.3. Discussion on the best model conﬁguration in FICM
As discussed in Section 2, FICM should work very well if each component model is accurate and diverse. According to thisprinciple, we attempted to determine the best conﬁgurations of FICM for clustering datasets in Genomics2004 and Genomics2005. The basic idea is to assign the best model to each ﬁeld with the constraint of keeping the diversity of each component model. Additionally, for Genomics2004 and Genomics2005, we also found that the effect of model selection on the title
ﬁeld was much weaker than the other two ﬁelds. Here, we ﬁrst compared different conﬁguration of FICM against the twoclassical models: BM (B-whole) and MM (M-whole). In this case, when compared with B-whole, the component setting on
the abstract ﬁeld in FICM is ﬁxed to BM, and when compared with M-whole, the component setting on the abstract ﬁeld
in FICM is ﬁxed to MM. In addition, by dividing the datasets into two groups, K P 8 and K < 8, we compared all eight possible conﬁgurations in FICM to ﬁnd the best combination.
In the ﬁrst case, since the model setting for abstract is already ﬁxed, we only need to set the models for the other two
ﬁelds: title and MeSH. As we discussed before, the best models for MeSH and title are BM and MM, respectively. Considering
the high correlation between title and abstract, we should assign different models to each ﬁeld. So, we assigned BM to MeSH,
and the complement model of abstract to title. Under these settings, we conducted a comparison experiment for BM by all
possible three combinations of B-whole, F-bbb and F-mbb. Table 15 shows S-Pairs obtained by these combinations. For MM, asimilar comparison experiment was done by using M-whole, F-mmm and F-bmb. Table 16 presented S-Pairs obtained by all
three combinations. We found that clustering performance could be improved, especially in the case of MM. For example, for
Genomics2004, F-mmm outperformed M-whole slightly with S-Pair (+15/63, 0/37), while F-bmb outperformed both Mwhole and F-mmm remarkably with S-Pairs (+78/98, 0/2) and (+58/93, 0/7), respectively. This result is totally true ofGenomics2005.
In addition, by dividing the datasets into two categories, K P 8 and K < 8, we determined the best conﬁguration of FICM.
In the former case, the best model would be F-bmb, while in the latter case, two models, F-mbb and F-bbb, with very close
performances, outperformed all other models signiﬁcantly. As shown in Table 17, when K P 8, F-bmb performed slightly better than F-mmb, which outperformed all other models signiﬁcantly for both Genomics2004 and Genomics2005. When K < 8,
F-mbb outperformed all other models signiﬁcantly for both Genomics2004 and Genomics2005, except F-bbb with a performance similar to that of F-mbb. The close performance of F-bbb and F-mbb reﬂects the weak effect of model setting in the title
ﬁeld. Over all, the experimental results further validate the effectiveness of our strategy of assigning the best component
model to each ﬁeld while keeping diversity in FICM to achieve good performance.

Table 15
The comparison of B-whole with F-bbb, B-whole with F-mbb and F-bbb with F-mbb in terms of S-Pair.
Data

F-bbb > B-whole

F-mbb > B-whole

F-mbb > F-bbb

Genomics2004
Genomics2005(+41/80, 0/20(+63/95, 1/5)(+49/86, 0/14)(+59/94, 0/6)(+8/60, 1/40)(+1/52, 3/48)

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
14

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

Table 16
The comparison of M-whole with F-mmm, M-whole with F-bmb and F-mmm with F-mbb in terms of S-Pair
Data

F-mmm > M-wholeF-bmb > M-wholeF-bmb > F-mmm

Genomics2004
Genomics2005(+15/63, 0/37)(+12/67, 3/33)(+78/98, 0/2)(+69/93, 0/7)(+58/93, 0/7)(+72/98, 0/2)

Table 17
The comparison between F-mbb (F-bbb) with others when K < 8, and the comparison between F-bmb (F-mmb) with others when K P 8 over all datasets in
Genomics2004 and Genomics2005.
Best modelCollection

F-mbb

Genomics2004

ðK < 8Þ

Genomics2005

F-bbb

Genomics2004

ðK < 8Þ

Genomics2005

F-bmb

Genomics2004

ðK P 8Þ

Genomics2005

F-mmb

Genomics2004

ðK P 8Þ

Genomics2005

F-bbm
 18

þ ;  09
 41
þ 25 ;  02
 48

21
0
þ ;9
 41

1
þ 28
48 ;  2
 41

þ ;  56
 44

þ 46 ;  22
 48

38
5
þ ;8
 42

2
þ 39
46 ;  7

F-bmm
 37

þ ;  20
 48

þ 29 ;  26
 44

35
0
þ ;2
 48

1
þ 33
41 ;  9 24

þ ;  10
 49
þ 30 ;  00
 50
þ 9 ;  70
 43

0
þ 18
42 ;  8

F-mbm
 18

þ ;  08
 42
þ 29 ;  03
 47

20
1
þ 46 ;  4
 32
þ 49 ;  11
 36
þ ;  47
 43

þ 46 ;  02
 48

32
6
þ ;9
 41

40
þ 47
;  03

F-mmm
 37

þ ;  40
 46

þ 34 ;  15
 45

36
1
þ ;2
 48

1
þ 34
43 ;  7
 43

þ ;  10 49
þ 48 ;  00
 50
þ 36 ;  30
 47

0
þ 44
50 ;  0

F-bbb
 1

1
þ ;  28
 22

1
2
þ 23
;  27–– 37
þ ;  69
 41

40
þ 44
;  26
 31

8
þ ;9
 41

6
þ 37
41 ;  9

F-bmb
 27

þ ;  18
 42
9
þ 21 ;  19
 31

34
2
þ ;7
 43

9
þ 25
33 ;  17––
 0
þ ; 8
 90 419 þ 15 ;  35

F-mbb
–
–
 1

1
þ ;  22
 28

2
1
þ 27
;  23 31

5
þ ;  11
 39

þ 38 ;  46
 44
6
þ 28 ;  13
 37

5
þ 34
41 ;  9

F-mmb
 27

þ ;  16
 44
9
þ 23 ;  16
 34

29
2
þ ;9
 41

8
þ 25
36 ;  14
 8

þ ;  09
 41

9
0
þ 35
;  15––5.3. The effect of ﬁeld weighting: assigning different weights to different ﬁelds in FICM
To further explore the capability of FICM, we can assign different weights to different ﬁelds. Here we focus on two best
conﬁgurations: F-bmb and F-mbb. We use a vector k ¼ ðkt ; ka ; km Þ to represent the weights assigned to ﬁelds, where kt ; ka and
km are the weights for the title, abstract and MeSH ﬁelds, respectively. For each ﬁeld, we examined four different k values: 3,5, 8 and 10. The experimental results of FICM on F-bmb and F-mbb with ﬁeld weighting, are shown in Tables 18 and 19,
respectively. For example, in Table 18, we ﬁrst present the performance of original F-bmb without ﬁeld weighting
ðk ¼ ð1; 1; 1ÞÞ, and then the performance of F-bmb with ﬁeld weighting and the corresponding S-Pair between the originalmodel and the one with ﬁeld weighting. We highlighted the conﬁguration in boldface that outperformed the original modelsigniﬁcantly. In spite of examining the effect of ﬁeld weighting on two different collections, Genomics2004 and Genomics2005, and two different conﬁgurations: F-bmband F-mbb, we were able to observe almost the same tendency from theexperiment results. That is, weighting the title ﬁeld moderately was able to improve the clustering performance signiﬁcantly. For example, on the Genomics2005 collection, with ðk ¼ ð3; 1; 1ÞÞ, the average NMI of F-bmb was improved from0.724 to 0.735 with S-Pair (+39/80, 1/20). The highest improvement was observed when kt was set to 3 or 5. Overweighing
the title ﬁeld seems impair the clustering performance. In fact, if we set k ¼ ð10; 1; 1Þ in the above example, the average NMI
slided from 0.735 ðk ¼ ð3; 1; 1ÞÞ to 0.727. On the other hand, adding more weights to the abstract and MeSH ﬁelds reduced
the clustering performance in almost all cases. For example, on the Genomics2005 collection, with ðk ¼ ð1; 3; 1ÞÞ, the average
NMI of F-bmb was reduced from 0.724 to 0.701 with S-Pair (+0/12, 45/88). Overall, the performance of FICM can be further
enhanced when we apply the Field Weighting extension to the suitable ﬁelds.

Table 18
The effect of changing weights on F-mbb in Genomics2004 and Genomics2005.
k

Genomics2004
NMI

Genomics2005
S-PairNMI

S-Pair(1, 1, 1).759 ± .06(3, 1, 1)(5, 1, 1)(8, 1, 1)(10, 1, 1).766 ± .06.765 ± .06.757 ± .06.750 ± .06(+22/73, 2/27)(+32/65, 7/35)(+27/49, 23/51)(+19/40, 32/60).735 ± .07.736 ± .06.732 ± .06.727 ± .06.724 ± .07(+39/80, 1/20)(+49/75, 5/25)(+49/69, 15/31)(+40/60, 20/40)(1, 3, 1)(1, 5, 1)(1, 8, 1)(1, 10, 1).737 ± .07.727 ± .07.719 ± .07.716 ± .07(+0/5, 61/95)(+0/2, 74/98)(+0/1, 85/99)(+0/2, 86/98).701 ± .07.692 ± .07.687 ± .07.682 ± .07(+0/12, 45/88)(+1/9, 66/91)(+0/3, 72/97)(+1/3, 79/97)(1, 1, 3)(1, 1, 5)(1, 1, 8)(1, 1, 10).751 ± .06.739 ± .06.728 ± .06.723 ± .06(+2/32, 24/68)(+2/15, 48/85)(+1/6, 66/94)(+1/5, 76/95).706 ± .06.689 ± .06.675 ± .06.667 ± .06(+2/13, 48/87)(+1/6, 74/94)(+0/4, 88/96)(+0/2, 92/98)

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
15

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx
Table 19
The effect of changing weights on F-bmb in Genomics2004 and Genomics2005.
k

Genomics2004
NMI

Genomics2005
S-PairNMI

S-Pair(1, 1, 1).756 ± .06.731 ± .06(3, 1, 1)(5, 1, 1)(8, 1, 1)(10, 1, 1).764 ± .06.763 ± .06.755 ± .06.751 ± .06(+21/76, 1/24)(+24/63, 4/37)(+18/48, 27/52)(+11/39, 33/61).738 ± .06.742 ± .06.740 ± .06.735 ± .06

ðþ21=74; 0=26Þ
(+32/80, 0/20)(+37/71, 7/29)(+26/58, 13/42)(1, 3, 1)(1, 5, 1)(1, 8, 1)(1, 10, 1).728 ± .06.716 ± .06.710 ± .06.708 ± .06(+0/1, 77/99)(+0/0, 92/100)(+0/0, 92/100)(+0/1, 91/99).706 ± .06.696 ± .06.691 ± .06.689 ± .06(+0/5, 75/95)(+0/2, 85/98)(+0/1, 87/99)(+0/1, 88/99)(1, 1, 3)(1, 1, 5)(1, 1, 8)(1, 1, 10).761 ± .06.755 ± .06.746 ± .06.739 ± .06(+11/61, 3/39)(+8/43, 24/57)(+5/31, 43/69)(+5/23, 52/77).724 ± .06.714 ± .06.700 ± .06.690 ± .06(+7/32, 28/68)(+5/20, 56/80)(+4/13, 72/87)(+3/9, 80/91)

6. Conclusions
We have presented a probabilistic model, FICM, for clustering multi-ﬁeld text documents. The advantage of FICM comesfrom the integration of the discriminative ability of each ﬁeld and the power of choosing the most suitable generative modelfor each ﬁeld. In order to achieve good performance, the component models of FICM should be accurate and diverse. We haveexperimentally shown that a direct extension of the classical BM and MMs by FICM, FICBM and FICMM, are able to achieve a
better performance, and by conﬁguring each ﬁeld with a suitable model, we obtain much better clustering results. The component model setting is practical when we have some prior knowledge on the ﬁelds, such as BM for MeSH ﬁeld in our work,
which can improve the performance signiﬁcantly. Alternately, we can carry out some preliminary experiments on the dataset to determine the suitable model for each ﬁeld. In addition, we introduced a ‘Field Weighting’ extension, which assigns
different weights to different ﬁelds in FICM, and found that it can further improve the clustering performance signiﬁcantly.
Over all, we emphasize that our idea of selecting the best model for each ﬁeld and of integrating them independently can beapplied to other documents with multiple ﬁelds, meaning that FICM is capable of improving the performance of clustering
documents in other applications.
In our experiments, the number of documents in each dataset ranges from 52 to 1960. We would like to examine the performance of FICM on some larger datasets in the future. Techniques for determining the number of topics in the dataset (Cheung, 2005) is likely to be incorporated into FICM. Finally, we hope that FICM will be applied to another clustering problem
where it also has multiple components that can be modeled separately and integrated.
Acknowledgements
The authors would like to thank anonymous reviewers for their helpful comments and advice. Shanfeng Zhu and Hiroshi
Mamitsuka have been supported in part by BIRD of Japan Science and Technology Agency (JST), and Shanghai Key Lab ofIntelligent Information Processing, Fudan University. In addition, this project has been partly supported by Startup Fundof Fudan University, the Shanghai Committee of Science and Technology, China (Grant Nos. 08DZ2271800 and
09DZ2272800) and The State Key Lab of Bio-Organic & Natural Products Chemistry, CAS.
References
Banerjee, A., Dhillon, I., Ghosh, J., & Sra, S. (2003). Generative model-based clustering of directional data. In The proceedings of the SIGKDD’03, Washington DC,
USA, August 24–27, 2003 (pp. 19–28).
Cheung, Y. (2005). On rival penalization controlled competitive learning for clustering with automatic cluster number selection. IEEE Transactions on
Knowledge and Data Engineering, 17(11), 1583–1588.
DeGroot, M. (1970). Optimal statistical decisions. McGraw-Hill.
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society:
Series B, 39(1), 1–38.
Dietterich, T. (2000). Ensemble methods in machine learning. In First international workshop on multiple classiﬁer system, MCS2000, Cagliari, Italy, June 21–23,
2000. Lecture notes in computer science (Vol. 1857, pp. 1–15).
Domingos, P., & Pazzani, M. (1996). Beyond independence: Conditions for the optimality of the simple bayesian classiﬁer. In Proceedings of the thirteenthinternational conference on machine learning (ICML’96), Bari, Italy, July 3–6, 1996 (pp. 105–112). Morgan Kaufmann.
Duda, R. O., Hart, P. E., & Stork, D. G. (2001). Pattern classiﬁcation (2nd ed.). New York: Wiley.
Ghosh, J. (2002). Multiclassiﬁer systems: Back to future. In Third international workshop on multiple classiﬁer system, MCS2002, Cagliari, Italy, June 24-26, 2002.
Lecture notes in computer science (Vol. 2364, pp. 1–15).
Ghosh, J. 2003. Scalable clustering methods for data mining. In Nong Ye (Ed.). Handbook of data mining (pp. 247–277). Mahwah, NJ: Lawrence Erlbaum
Associates.

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005

ARTICLE IN PRESS
16

S. Zhu et al. / Information Processing and Management xxx (2009) xxx–xxx

Hansen, L., & Salamon, P. (1990). Neural network ensembles. IEEE Transaction on Pattern Analysis and Machine Intelligence, 12, 993–1001.
Hersh, W. R., Bhupatiraju, R. T., Ross, L., Johnson, P., Cohen, A. M., & Kraemer, D.F. (2004). Trec 2004 genomics track overview. In E. M. Voorhees, & L. P.
Buckland (Eds.). The proceedings of the thirteenth text retrieval conference (TREC 2004), Gaithersburg, Maryland, November 16–19, 2004.
Hersh, W. R., Cohen, A., Bhupatiraju, R. T., Johnson, P., & Hearst, M. (2005). Trec 2005 genomics track overview. In E. M. Voorhees, & L. P. Buckland (Eds.). The
proceedings of the thirteenth text retrieval conference (TREC 2004), Gaithersburg, Maryland, November 15–18, 2005.
Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data clustering: A review. ACM Computing Surveys, 31(3), 264–323.
Jensen, L., Saric, J., & Bork, P. (2006). Literature mining for the biologist: from information retrieval to biological discovery. Nature Reviews Genetics, 7(2),119–129.
Lewis, D. (1998). Naive (bayes) at forty: The independence assumption in information retrieval. In Proceedings of the 10th European conference on machinelearning (ECML-98), Chemnitz, Germany, April 21–23, 1998. Lecture notes in computer science (Vol. 1398, pp. 4–15).
Li, B., Xu, S., & Zhang, J. (2007). Enhancing clustering blog documents by utilizing author/reader comments. In Proceedings of the 45th annual southeast
regional conference, 2007, Winston-Salem, North Carolina, USA, March 23–24, 2007 (pp. 94–99). ACM.
McCallum, A., & Nigam, K. (1998). A comparison of event models for naive Bayes text classiﬁcation. In AAAI workshop on learning for text categorization (pp.
41–48).
Meila, M., & Heckerman, D. (2001). An experimental comparison of model-based clustering methods. Machine Learning, 42(1/2), 9–29.
Nelson, S. J., Schopen, M., Savage, A. G., Schulman, J.-L., & Arluk, N. (2004). The mesh translation maintenance system: Structure, interface design, andimplementation. In M. E. A. Fieschi (Ed.), Proceedings of the 11th world congress on medical informatics; 2004 September 7–11; San Francisco, CA(pp. 67–69). Amsterdam: IOS Press.
Polikar, R. (2006). Ensemble based systems in decision making. IEEE Circuit and System Magazine, 21–45.
Porter, M. (1980). An algorithm for sufﬁx stripping. Program, 14(3), 130–137.
Ribeiro-Neto, R. B.-Y. B. (1999). Modern information retrieval. New York: Addison Wesley.
Rigouste, L., Cappé, O., & Yvon, F. (2007). Inference and evaluation of the multinomial mixture model for text clustering. Information Processing andManagement, 43(5), 1260–1280.
Wheeler, D. et al (2005). Database resources of the national center for biotechnology information. Nucleic Acids Research, D39–D45.
Yoo, I., & Hu, X. (2006). A comprehensive comparison study of document clustering for a biomedical digital library medline. In G. Marchionini et al. (Eds.).
ACM/IEEE joint conference on digital libraries, JCDL 2006 (pp. 220–229).
Zhong, S., & Ghosh, J. (2003). A uniﬁed framework for model-based clustering. Journal of Machine Learning Research, 4, 1001–1037.
Zhong, S., & Ghosh, J. (2005). Generative model-based document clustering: A comparative study. Knowledge and Information Systems, 8(3), 374–384.

Please cite this article in press as: Zhu, S., et al. Field independent probabilistic model for clustering multi-ﬁeld documents. Information
Processing and Management (2009), doi:10.1016/j.ipm.2009.03.005