DEIM Forum 2017 C3-1

Identifying Web Pages with Major Contents
based on Search Engine Suggests and Topic Modeling
Chen ZHAO† , Yi DING†, Tian NIE†† , Jiaqi LI† , Takehito UTSURO††† , Yasuhide KAWADA†† ,and Noriko KANDO††††† Graduate School of Systems and Information Engineering,
University of Tsukuba 1-1-1, Tennodai, Tsukuba, 305-8573, Japan†† Logworks Co., Ltd. Tokyo, 151-0051, JAPAN†††† National Institute of Informatics, Tokyo, 101-8430, JAPAN††† Graduate School of Systems and Information Engineering,
Department of Intelligent Interaction Technologies,
University of Tsukuba 1-1-1, Tennodai, Tsukuba, 305-8573, Japan
Abstract This paper addresses the problem of identifying irrelevant items from a small set of similar documents
using Web search engine suggests. Speciﬁcally, we collected volumes of Web pages through Web search engines andinspected the page contents using topic models. Among each cluster of pages sharing the same topic indicated by the
topic model, our technique discovers potential content organization in the current page cluster and identiﬁes pages
that are out of focus from that topic. The metrics in our approach mainly consist of search engine suggest frequency
and inter-document similarity measures. The intuition is that Web pages collected via the same search queries
are more likely to share similar contents. We verify this intuition by implementing a subtopic based document
selection framework and making quantitative evaluation against human made labeled data sets. Our evaluation
result reveals that suggest frequency analysis along with inter-document similarity measure is eﬀective at ﬁlteringoﬀ-topic documents in small data sets with satisfactory performance.
Key words search engine suggests, topic model, document ﬁltering, theme aggregation

1. Introduction
This paper mainly focuses on analysis techniques that distinguish important documents with major contents from irrelevant ones among a collection of documents all about aspeciﬁc topic. We ﬁrst collected search suggests provided
by Web search engines about some ﬁxed query focus, which
is a predeﬁned keyword on a general topic. Figure 1 shows
Figure 1 An Example of Search Engine Suggests Provided by the
Search Engine

an example on how a Web search engine provides various
search suggests around the query focus keyword ”job hunting” (”shyu-katsu” in the Japanese case). These suggests

Table 1 Numbers of Collected Suggests and Web Pages for Each
Query Focus

serve as indication of frequent user search logs and trending
topics over the time. We used all the collected search suggests as queries to extract Web pages from the search engine
meanwhile keeping track of which queries were used to acquire every page and saving them as page-wise information
for later stage analysis. For example, in the case we query
the search engine with the term ”job hunting hairstyles”(”shyu-katsu kamigata” for Japanese), we attach every Web
page collected from this query with the phrase ”job huntinghairstyles” as one of its search engine suggests. Next we ap-and ”anata no tsuyomi” (”your strengths”).

ply all the Web pages regarding a query focus which is ”job

2. 2 Collecting Web Pages

hunting” for this example to the LDA topic model. The LDA

We used Yahoo! Search BOSS API（注 1） to scrape web pages

topic model is a popular topic model among text processing

from the search engine. Using the web search engine suggests

community and it performs statistical sampling by Latent

we collected in the previous section combined with the query

Dirichlet Allocation [3] to model topic distributions to a col-focus keyword as queries (in the form of AND search), we al-lection of documents. Based on the topic distribution fromways collect the ﬁrst 20 pages returned per query. The set of

the LDA model, we assign every collect Web page a topic of

web pages queried by suggest s can be represented as D(s, N )maximum probability so that our Web page collection is re-where N is 20 as a constant standing for the top N pages.

arranged into a ﬁxed number of topic clusters, which will be

As previously mentioned we save the search engine suggests

explained in details in later sections. Our primary objective

for every Web page. Since diﬀerent search engine suggests

is to design on top of the LDA topic assignment a framework

could lead to the same Web page, one single Web page could

that automatically analyzes Web pages in each topic cluster,

have multiple suggests. So we maintain a suggest set Ë(d)so that pages with major subtopics will be identiﬁed versus

for each Web page d, so that Ë(d) contains all the suggests

pages with minor subtopics. In this paper a subtopic withinthat were used to search the page d. Therefore suggests of aan LDA topic cluster will always be referred to as a ”theme”

web page are saved as follows.

to avoid confusion. We deﬁne major themes as page contents shared by at least some number of pages in the same
topic cluster. The reason of such a deﬁnition stems from thesense that on-focus themes tend be more frequently coveredby Web pages while themes with rare occurrence are moreË(d) =¬¬

Òs ∈ S ¬d ∈ D(s, N )Ó

3. LDA Topic Model
3. 1 Topic Model

likely to be trivial contents. The purpose of this framework is

This paper employs LDA (Latent Dirichlet Allocation) [3]

to identify on-focus pages in a topic cluster and ﬁlter out lessto model topic distributions among documents. Given a pre-relevant trivial contents — pages with minor themes that are

set constant K representing the number of output topics,

not semantically close enough to all the other pages. This

The LDA topic model takes a collection of documents (inpaper presents all its analysis on two diﬀerent query focuses

our case Web pages for one query focus), treats every single

that cover various Japanese query results about job hunting

document as a sequence of words and estimates the word dis-and marriage issues. The number of total suggests collected

tribution p(w | zn ) (w ∈ V ) for every topic zn (n = 1, . . . , K)

and Web pages are listed in Table 1. The rest of the pa-

as well as a topic distribution p(zn | d) for every document

per is organized as follows. First it introduces some details

where V is the vocabulary set（注 2）. This paper adopts Gibb-

on search engine suggest assignment and LDA topic modelsLDA++（注 3） as the toolkit while the parameters are tuned

structure. Then our main approach is explained along withthrough a preliminary

the criteria how human made reference data is generated,evaluation by examining the number of topics as K = 40

followed by the evaluation results of the approach. Next is aand 50 for query focuses ”marriage” and ”job hunting”, re-brief glimpse of previous related research. Finally, it comes

spectively.

to conclusion.

3. 2 Assigning a Topic to a Web Page
Let D be the document set containing all collected Web

2. Collecting Search Engine Suggests and
Web Pages

pages and K be the number of topics. Given the topic model

2. 1 Collecting Search Engine Suggests

every d (d ∈ D). We then assign every document d the topic

For a given query focus keyword, we specify about 100rsearchtypes of Japanese hiragana characters to Google

is applied, we have a topic distribution p(zn | d) available forwith the highest probability among all its p(zn | d). The
following formula deﬁnes this process.

engine from which we then collect not exceeding 1000 suggests. These 100 types of Japanese hiragana characters in-D(zn ) =Ò¬¬d ∈ D ¬ zn =argmaxzu (u=1,...,K)P (zu |d)Ó

clude Japanese alphabet consisting of 50 characters, voiced
and semi-voiced variants of voiceless characters and Youon (a（注 1）：http://developer.yahoo.com/search/boss

variation of diphthong as language feature in Japanese). For（注 2）：In this paper, as the set V of vocabulary, we use the set of en-

example, once we type ”shu-katsu a” (”job hunting a”) into

try titles of the Japanese version of Wikipedia, where the version wethe search ﬁeld, a list of suggests are popped out all startingwith the reading character ”a” such as ”aisatsu” (”greeting”)

used in this evaluation was downloaded in March 2014 and has about
1,407,000 entries.
（注 3）：http://gibbslda.sourceforge.net/

Table 2 Numbers of Suggests satisfying Lower Bounds of Frequency
Suggest Frequency
Lower Bound
Syu-katsu(Job Hunting)Kekkon(Marriage)

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

921

846

692

592

495

425

379

322

264

224

187

132

88

53

30

12

3

946

938

884

798

706

604

533

453

376

305

242

182

132

88

50

27

8

The net eﬀect is that for every topic zn , there are a groupments containing each of the themes. If at least two otherof corresponding documents that are assigned to zn . Since

documents in the same cluster are found to contain simi-

we never assign multiple topics to the same document, there

lar information with d by common sense, d is labeled to be

is no overlap between topic clusters D(zn ) for n = 1, . . . , K.

in major contents. Thus, the actual reference data selects

4. Identifying Documents of Major Contents

documents that belong to major contents from the top r
documents as follows.
Dref (zn , r, df ) =Ò

4. 1 Overview
Our framework is designed in such a way that within each
topic cluster D(zn ) it selects documents of on-focus themes¬¬d ∈ Drank (zn , r) ¬ at least df documentsÓin Drank (zn , r) contain the same theme with dand leaves the rest as irrelevant theme documents. We namethe set of such selected documents as ”major contents” fortopic cluster D(zn ). We deﬁne a document d (d ∈ D) to be amajor content document given that in the current topic cluster D(zn ), there are at least df documents about the same
theme as d, including d itself. In other words, document d issaid to belong to major contents if we are able to ﬁnd df − 1other documents sharing one or more themes within D(zn ).
The intuition of this deﬁnition comes from the assumptionthat if a majority of documents covers some theme then thistheme is likely to be on-focus content otherwise it tends to betrivial. This section ﬁrst introduces how manually certiﬁed
reference data set is produced. Next it explains how suggest
frequency helps achieve the above semantics. It then discusses similarity measures used to help recheck and improve

4. 3 In-topic Suggest Frequency Based Document
Selection Method
The ﬁrst part of our approach relies on suggest frequency
count to determine whether a document belongs to major
contents with respect to its topic cluster. This method attempts to reproduce the manual work in section 4. 2 in an
automatic way by assuming that a document queried bysome suggest always covers contents relevant to that suggest,which is true only for non-spam Web pages. Furthermore,
search engine suggests with higher counts of occurrence aresupposed to be more reliable in terms of revealing document
themes because a suﬃcient number of documents are available to help verify this assumption.
As previously stated, every document is coupled with alist of Web search engine suggests as a record about which

the document selection results.

queries that page was retrieved with. Therefore within ev-

4. 2 Creating Reference Data
The reference data set is a sequence of manual labels in-ery topic cluster D(zn ) we count the occurrences of every

dependently created by purely manual work. This data set

suggest that belongs to documents in D(zn ). Table 2 lists

serves as a standard solution to the document selection problem for evaluation purpose. First, considering the practicalmanual workload, we restrain the reference data set within
the top r documents in each topic cluster ranked by topic
model probability p(zn | d).

¬¬d ∈ D(zn ) ¬ p(zn | d) ranks
at least r in D(zn ) by descending orderwith frequency above diﬀerent thresholds within topics. Formally this frequency is deﬁned by the following formula.

¬¬Ò

Ó¬¬

f (s, zn ) = ¬ d ∈ D(zn ) | s ∈ S(d) ¬Here S(d) is the set of all search engine suggests for doc-

Drank (zn , r) =Òoverall statistics on total number of search engine suggests

ument d. An alternative explanation of suggest frequency

ÓThen, complying with the deﬁnition of ”major contents”
speciﬁed in section 4. 1, human readers go through every
D(zn ) determining the themes contained in every document
based on real-life semantics and count the number of docu-

f (s, zn ) is the number of documents in D(zn ) containing
suggest s. Furthermore, we deﬁne the maximum suggest
frequency of d, fmax (d) as the most frequent suggest among
S(d).

¬¬

¬¬

fmax (d) = ¬ argmax f (s, zn )¬s∈S(d)

Figure 2 Query Focus Syu-katsu (Job Hunting) Macro Precisions and Recalls by varying
Suggest Frequencies

Figure 3 Query Focus Syu-katsu (Job Hunting) Micro Precisions and Recalls by varying
Suggest Frequencies

By the above deﬁnition, we are able to compute the max-

documents. In addition, we deﬁne the set of topics with non-imum suggest frequency of every document in separate clus-

empty major contents as Tf (flbd ), so that topics with empty

ters D(zn ). Our expectation is that document suggests S(d)major contents will be treated as pure garbage clusters.

would well indicate document themes. For example, if D(zn )contains ”hairstyle” and ”email reply”, we infer that document d covers themes on both hairstyle tips and email communication issues in its context. Whereas the maximum

¨ ¬

©

| ∅
Tf (flbd ) = zn ¬Df (zn , flbd , r) =4. 4 Incorporating In-topic Document Similarity
Measuressuggest fmax (d) reﬂects the maximum number of possible

Although the assumption in the previous section that sug-

documents in D(zn ) sharing identical suggest(s) with d, wegests serve as hints about themes in a document is true forapproximate the semantics on major contents by selecting

most Web pages in our dataset, there exist a lot of spam

documents whose fmax (d) satisﬁes some lower bound flbd asand erroneous pages where this assumption does not standdeﬁned below.

as real world Web text is mostly messy. Therefore we needÒDf (zn , flbd ) = d ∈ D(zn ) | fmax (d) >= flbdÓ

to further examine documents by distance measures to verify
whether a document is truly sharing common characteristics

Then, we restrict the evaluation data within the top r docu-

with the others to be eligible for major contents. This paper

ments in each topic cluster ranked by topic model probability

utilizes weighted Jaccard distance [6] for computing correla-p(zn | d) as the set Df (zn , flbd , r) given below:

tion between documents. The distance between document d

Df (zn , flbd , r) = Df (zn , flbd ) ∩ Drank (zn , r)
For every topic cluster D(zn ), we select Df (zn , flbd , r) to be
major content documents and all the rest to be trivial ones
that are ﬁltered from major contents as relatively irrelevant

and d is deﬁned as follows.

È Ëw∈d d min(dw , dw )Jaccard(d, d ) = 1 − È

Ë


w∈d



d

max(dw , dw )where dw and dk are count of word w in document d and

d . Then we take the similarity between d and d is computed

cluster D(zn ), the model output D(zn , flbd , df , slbd , r) is evaluated against the labeled reference Dref (zn , r, df )（注 4） so that

as 1 − Jaccard(d, d ) as

Ë d min(dw , dw )Ëw∈d d max(dw , dw )we compute the precision and recall for each zn based on

Based on the above measures, we select d from D(zn )as a major content document if d satisﬁes the condition
that there are at least df − 1 other documents in D(zn )above some similarity threshold slbd . Furthermore, all interdocument similarities are computed within the top r documents Drank (zn , r) to simulate the reference data set. We
deﬁne all such documents similar to d as the nearest neighbor
set N expressed as

ual topic cluster, we integrate them to produce the ﬁnal

w∈d

Sim(d, d ) =N (d, zn , slbd ) =whether a document is correctly classiﬁed as part of major contents. With evaluation outcomes from each individ-¬¨ d ∈ Drank (zn , r) ¬ d =| d, Sim(d, d ) >= slbdÓThe major contents D(d, zn , df , slbd ) is then selected by Ds

cus. The ﬁnal macro/micro precision and recall for a queryfocus are deﬁned as follows.

M acro Recall(flbd ) = D (z , r, d )  D(z , f , d , s , r)z (n = 1, . . . , K)D D (z (z, r,,dr,)d ) =| ∅
ref

n

n

f

ref

zn ∈T (flbd )nbased on the nearest neighbor set size.

d ∈ D

macro/micro precisions and recalls for the entire query fo-

 |N (d, z , s , r)| > d − 1
=

n

lbd

f

lbd

f

n

ref

f

Ds (zn , slbd , df , r) =rank (zn , r)nlbd

M acro P recision(flbd ) =f D (z , r, d )  D(z , f , d  , s , r)
D(z , f , d , s , r)
T (f )
ref

The above similarity measure scheme can be stated in plain
words that only if there exist a least number of neighbor

n

n

lbd

f

lbd

f

n

zn ∈T (flbd )

lbd

f

lbd

lbd

documents close enough to the current one, can the current
document be recognized as part of major contents.
Now that we have deﬁned how similarity measures certify major contents as Ds (zn , slbd , df ), we now combine thistechnique with the frequency count method in the previous
section to generate the ﬁnal output. We apply our model

M icro Recall(flbd ) = D (z , r, d )  D(z , f , d , s , r) D (z , r, d )
ref

n

n

f

lbd

f

lbd

zn ∈T (flbd )

ref

n

f

zn (n=1,...,K)

upon each topic D(zn ). The model selects documents satisfying both suggest frequency and similarity criteria as theset D(zn , flbd , df , slbd ), which is the intersection of the documents by both methods. For every LDA topic cluster zn ,D(zn , flbd , df , slbd , r) = Ds (zn , slbd , df , r)

 D (z , f , r)
f

n

lbd

M icro P recision(flbd ) = D (z , r, d )  D(z , f , d , s , r) D(z , f , d , s , r)
ref

n

f

n

lbd

f

lbd

f

lbd

zn ∈T (flbd )n

lbdzn ∈T (flbd )is the ﬁnal output from the model and will get evaluatedagainst the reference data set Dref (zn , r, df ).

5. 2 Evaluation Report

Similar to Tf (flbd ) in the previous section, the set of topics

The ﬁnal document selection model accepts 5 parame-with non-empty major contents in the ﬁnal model is deﬁned

ters to generate major contents D(zn , flbd , df , slbd , r) as theas T (flbd )model output. r is set to be 30. As noted before, df is

 



| ∅
T (flbd ) = zn D(zn , flbd , df , slbd , r) =constantly 3. slbd is pre-modulated to 0.10 for query focus”job hunting” and 0.15 for ”marriage”. With df and slbd
ﬁxed, the model is applied for multiple runs with diﬀerent
suggest frequency thresholds for easier observation on how

5. Evaluation
5. 1 ProceduresThis paper experiments the proposed model on Web page
collections of two query focuses as listed in Table 1. The
topic model has a preset output topic number K = 40 for”job hunting” and K = 50 for ”marriage”. For every topic

the model performance varies with diﬀerent flbd . Figure 2
to Figure 5 showcase precisions and recalls corresponding to
flbd ranging from 2 to 18 for both query focuses. In addition to ﬁnal model output D(zn , flbd , df , slbd , r), Figure 2 to
Figure 5 also include evaluation on suggest frequency based（注 4）：In this paper, df is constant and df = 3 for all cases.

Figure 4 Query Focus Kekkon (Marriage) Macro Precisions and Recalls by varying Suggest Frequencies

Figure 5 Query Focus Kekkon (Marriage) Micro Precisions and Recalls by varying Suggest Frequenciesselector Df (zn , flbd , r) without rectiﬁcation by any similar- M icro Recallf (flbd ) =ity measures, in order to conﬁrm how much improvement
similarity measures contribute to. The precision and recall



Dref (zn , r, df )zn ∈Tf (flbd )measures of Df (zn , flbd , r) are deﬁned in the same approachas ﬁnal model output. Df (zn , flbd , r) evaluations are pre-

Dref (zn , r, df )zn (n=1,...,K)

Df (zn , flbd , r)sented in dashed lines ﬁgures and deﬁned as Macro/Micro
P recisionf /Recallf below.



M icro P recisionf (flbd ) =M acro Recallf (flbd ) =zn ∈Tf (flbd








Dref (zn , r, df ) Df (zn , flbd , r)Dref (zn , r, df ))zn (n = 1, . . . , K)

Dref (zn , r, df ) =| ∅Dref (zn , r, df )zn ∈Tf (flbd )

Df (zn , flbd , r)zn ∈Tf (flbd )D(zn , flbd , r)

5. 3 Examples of Major Content Selection
This section describes a concrete use case example on how   

M acro P recisionf (flbd ) =zn ∈Tf (flbd

Dref (zn , r, df ) Df (zn , flbd , r)
Df (zn , flbd , r))Tf (flbd )the document selection scheme practically works. Details are
depicted in Figure 6. This example contains a topic cluster
D(zn ) for query focus ”marriage”. This topic contains documents mainly focusing on common issues to consider whenchoosing marriage partners such as occupations, incomes andpersonality. The right column lists the model output. Sincedocuments with ID 1, 2, 3, 4 share the same theme about

themes and how it is achieved by manual labor. Next itinﬂuence of occupations on marriage they are selected as ma-

designed an automatic framework to simulate the manual

jor contents. Similar case goes for ID 5, 6, 7 which all coverwork. The major approach relies on the search engine sug-

contents on qualiﬁcations for marriage, especially for men.

gests provided by the search engine as a critical characteris-

Remaining documents such as 8, 9, 10 are not major con-tic to infer semantic themes contained in each document. To

tents either because that they share no common theme withhelp enhance such expectation from search engine suggests

at least 2 others in the cluster or that their suggest frequen-

it shows that inter-document similarity measures can verify

cies are below the preset threshold flbd = 3. For this topic

the suggest frequency schemes and improve model output toalone, the recall is calculated as the proportion of correctly

certain extents depending on scenarios of diﬀerent query fo-

selected documents in the reference data (the reference set

cuses. Future work involves re-evaluating topic clusters after

is not shown). The precision in this case is the proportion of

the proposed model is applied using topic coherence metrics

documents in the reference set among those selected by thementioned in the related works.

proposed method.

References

6. Related Work
The major content selection scheme presented in this paperﬁnds its insight mostly from existing topic modeling metrics.
Blei et al. [3] proposes perplexity as a topic quality metric.
de Wall and Barnard [7] discusses the problem of vocabulary
dependence in perplexity and raise the concept ”topic stability” as an evaluation technique. Similarly there are otherworks [9], [15] studying correctness of topic modeling. This
paper diﬀers from those works by focusing on in-topic contents instead of overall probabilistic behavior of the topic
model.
As for in-topic content evaluation, there also exist quite afew quality evaluation schemes that measure how meaningful
topics are including Chang et al. [5] using manual evaluation
on held-out keywords and other work [1], [4], [10]?[12], [14]that rely on external knowledge resources such as Wikipedia,
WordNet（注 5）, search engine information and news corpora.
Lau et al. [8] and Röder et al. [13] do some systematic comparison among various topic evaluation techniques.

Al-

Sumait et al. [2] recognizes junk topics through unsupervised
analysis so that topics will be ranked by semantic legitimacy.
Compared to all the aforementioned approaches, techniques
proposed in this paper not only performs topic evaluation
but also directly ﬁlters junk contents thus eﬀectively reﬁning
a topic by discovering beneﬁcial contents with respect to a
topic. The expected outcome is more coherent topic modeling consisting of less garbage and topic aggregation withhigher quality.

7. Conclusion
The objective of this paper is to extract closely correlated
documents and ﬁlter less relevant items from a set of alreadysimilar documents clustered by the LDA topic model. It ﬁrst
discussed the intuition of recognizing document relevant on（注 5）：https://wordnet.princeton.edu/[1] N. Aletras and M. Stevenson. Evaluating topic coherence using distributional semantics. In Proc. of 10th IWCS, pages
13–22, 2013.
[2] L. AlSumait, D. Bardara, J. Gentle, and C. Domeniconi.
Topic signiﬁcance ranking of LDA generative models. In
Proc. ECML/PKDD, pages 67–82, 2009.
[3] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet
allocation. Journal of Machine Learning Research, 3:993–
1022, 2003.
[4] H. Chan and L. Akoglu. External evaluation of topic models: A graph mining approach. In Proc. 13th ICDM, pages
973–978, 2013.
[5] J. Chang, J. Boyd-Graber, S. Gerrish, C. Wang, and D. Blei.
Reading tea leaves: How humans interpret topic models. In
Proc. NIPS, pages 288–296, 2009.
[6] F. Chierichetti, R. Kumar, S. Pandey, and S. Vassilvitskii.
Finding the Jaccard median. In Proc. 21st ACM-SIAM
Symposium on Discrete Algorithms, pages 293–311, 2010.
[7] A. de Waal and E. Barnard. Evaluating topic models withstability. In Proc. PRASA, pages 79–84, 2008.
[8] J. H. Lau, D. Newman, and T. Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence
and topic model quality. Proc. 14th EACL, pages 530–539,
2014.
[9] D. Mimno and D. Blei. Bayesian checking for topic models.
In Proc. EMNLP, pages 227–237, 2011.
[10] C. C. Musat, J. Velcin, S. Trausan-Matu, and M.-A. Rizoiu.
Improving topic evaluation using conceptual knowledge. In
Proc. 27th IJCAI, pages 1866–1871, 2011.
[11] D. Newman, S. Karimi, and L. Cavedon. External evaluation of topic models. In Proc. in Australasian Document
Computing Symposium, pages 11–18, 2009.
[12] D. Newman, J. H. Lau, K. Grieser, and T. Baldwin. Automatic evaluation of topic coherence. In Proc. HLT-NAACL,
pages 100–108, 2010.
[13] M. Röder, A. Both, and A. Hinneburg. Exploring the spaceof topic coherence measures. In Proc. 8th WSDM, pages
399–408, 2015.
[14] K. Stevens, P. Kegelmeyer, D. Andrzejewski, and D. Buttler. Exploring topic coherence over many models and many
topics. In Proc. EMNLP/CoNLL, pages 952–961, 2012.
[15] H. M. Wallach, I. Murray, R. Salakhutdinov, and D. Mimno.
Evaluation methods for topic models. In Proc. 26th ICML,
pages 1105–1112, 2009.

      

   

         
 6* 7  +#*( *   5 %"  

)$% 5 )$% 5    5  ' 5    *  )$%  '  )$%    
       

6 7


#
$
%



 

&



'*  
     





   

  
   



#"



    
 



 



  
    $%!
  $&!

  

    (!

              




+

       

)  (         !

    
            !

  
    $#!
    !

   $%!
  
$'!

    
       (!
   






   $%!
  
$&!



 

       

    (!

        

   $#!

            
   !

       





%"







 

    %!
       




      *  )$%  '  )$%   ,$)" -3 '2. / %1 ,2. / + "401 -3 +%1  ,"401 -3 +%1 

,

!
!

,#"

   +&' *   ' ,#)" -3 '2. / %1 ,2. / + "401 -3 +%1   *" -3 '2. / %1 ,2. / + 

Figure 6 A Detailed Example of In-topic Major Contents Selection

!

*

!

*#)&