DEIM Forum 2016 C3-6

Extracting Representative Phrases from Wikipedia Article
Sections
Shan Liu†

Mizuho Iwaihara‡

Graduate School of Information, Production and Systems, Waseda University
2-7 Hibikono, Wakamatu-ku, Kitakyushu-shi, Fukuoka-ken, 808-0135 Japan
E-mail: †vickey-liu@toki.waseda.jp, ‡iwaihara@waseda.jp
Abstract Nowadays, Wikipedia has become one of the most important tools for searching information.
Since its long articles are taking time to read, as well as section titles are sometimes too short to capturecomprehensive summarization, we aim at extracting informative phrases that readers can refer to.
Existing work on topic labelling works effectively and performs well on document categorization, but
inadequate for granularity of detailed contents. Besides, existing keyphrase construction methods just
perform well on very short texts. So we try to extract phrases which represent the target section content
well among other sections within the same Wikipedia article.

We also incorporate related external

articles to increase candidate phrases. Then we apply FP-growth to obtain frequently co-occurring word
sets. After that, we apply improved features which characterize desired properties from different aspects.
Then, we apply gradient descent on our ranking function to obtain reasonable weighting on the features.
For evaluation, we combine Normalized Google Distance (NGD) and nDCG to measure semantic
relatedness between generated phrases and hidden original section titles.
Keywords Wikipedia,Co-occurring Word Sets,Gradient Descent

1 Introduction

focus away from AI, and toward methods and models borrowed

As a free, open encyclopedia, Wikipedia provides abundantfrom statistics and probability theory, and there are overlaps

information sources. People are allowed to edit items on Wikipedia,

between data mining.”

where thousands of new items added every day, and thousands ofmodifications per hour, resulting in many different authors in
different backgrounds. In this case, there are different emphases
due to different authors even in one section. Most current
approaches to topic construction yield ranked lists of unigrams torepresent topics. However, it has long been known that unigrams
account for only a small fraction of human-assigned index terms[7]. However, for a person who is unfamiliar with the topic may noteasily capture the content quickly. On the other hand, there are only
one section title and several subtitles where titles present as phrases,

Figure 1. Section “History and relationships to other fields” inwhich are too general to present the content of sections thoroughly.

article “machine learning”

Namely, one or two phrases cannot cover the main content of

But we cannot capture the main content by the section title

sections. For instance, the following section titled “History and“History and relationships to other fields” since its over-general.

relationships to other fields” in article “Machine Learning” and theBut if there are such phrases, like “artificial intelligence”,

main content is: “Machine learning grew out of the quest for“probability theory” and “data mining” etc. we may mainly grasp

artificial intelligence, started to flourish in the 1990s by shifting

the section content quickly.

DEIM Forum 2016 C3-6

So there is a need to extract n-gram keyphrases to present moreused in query expansion techniques.

information which allow people to take a glimpse of these phrases

Unlike most of these methods, which focus on topic labelling by

and capture the general content quickly. A simple way is to extract

unigrams or keyphrase extraction on long documents [2, 6] or very

chunks in sections by statistical probability and latent topic model

short text like microblogs [10], queries [8], we concentrate on[3]. However, Wikipedia articles are open to people from all over

Wikipedia article sections, where some of them are long texts while

the world with various writing styles. For instance, it is common

others may be short texts.

that a keyphrase is slightly separated by other words and words in

Many researchers also apply word sequence segmentation [5] andthe phrase are in different sequences. But chunks are consecutive

chunks on keyphrase extraction [3], which are based on statistics orandpart-of-speech. In this approach, however, keyphrases separated by

fixedsequences,where

non-consecutive

phrasesareforbidden.To address that, we make improvements over our

several other words cannot be captured.

previous work [11]. We use a traditional method [4] of frequentco-occurring word extraction to obtain a set of words as keyphrase

3 Framework

candidates. Then we apply improved feature models for generating

Since section titles in Wikipedia articles are sometimes too short torepresentative phrases in one section that can represent the content

capture comprehensive summarization, we aim at extracting

well and comprehensively. Firstly, we incorporate related articles in

informative phrases that readers can refer to. So we are working on

Wikipedia that are similar to the target section, so that more quality

representative keyphrase extraction on article sections. When given

but hidden content can be extracted. We apply FP-growth [4] toan article with several sections, we want to return several

obtain frequent patterns which are sets of words co-occurring

representative phrases for every section that these phrases canfrequently.

reflect the main content of these sections.

Meanwhile, we improve the four features that used to measure

The phrases we extracted shouldn‟t be limited in target section,phrase qualities, which are coverage, phraseness, uniqueness andsince phrases in the section may not be enough or hard to judge

potentialness. Uniqueness and potentialness are proposed by Han etpopularity. So we need external articles to evaluate popularities onal. (2015) [11]. Coverage and phraseness are proposed bysimilar phrases and provide latent topics. Due to the different

Danilevsky et al. (2014) [7]. Finally, we apply gradient descent toemphases and writing styles of different authors, we apply

find optimum weighting between the four features. Based on the

FP-growth [4] to extract co-occurring patterns as order-free word

weight scores we rank candidate phrases.

sets. Since these word sets are order-free, we apply hits in searchorganized as follows.

The rest of this paper is
Section 3

engine to find are a reasonable order that form a sequence phrase

sketches out our basic framework, including our improved method

Section 2 covers related work.

for every word set. Then we define four quality measurements toand algorithms. Section 4 displays our experiment result, in Section

measure these phrases in four different aspects. Finally, we apply

5, we introduce an evaluation method, and in Section 6, we address

gradient descent to find the optimal weights for these foura conclusion and future work.

measurements and obtain the overall rank on phrases.

2 Related work
Existing approaches to unsupervised phrase extraction are
generally graph-based or unigram-based, which extract ranked
representative unigrams first, then combine them into a phrase. Liuet al.[13] transform traditional graph-based ranking method as
follows:

single random walk into multiple random walks specificto various topics by building a Topic PageRank (TPR) on word

Figure 2. Frameworkgraphs to measure word importance with respect to different topics.

3.1 Preprocessing

Graph-based topic labelling [2] by making use of graph-based data

For a target article, we first download all the Wikipedia external

structure in DBpedia performs pretty well on topic labelling.

related articles which contain the words of the target article title.

Symonds et al. [8] combine an unigram relevance model andWe preprocess these articles as well as the target article by filtering

linguistic term dependencies to model word associations which arestop words. For a target section in the target article, we computeDEIM Forum 2016 C3-6

cosine similarities based on TF-IDF for obtaining top-k related

w should be contained; constant is given by experience, here we set

articles for every section in the target article.

to 13; 𝑐𝑢𝑟𝑅𝑎𝑛𝑘 refers to the current rank of word w according to

Because related articles just play a role of support, phrases in thethe frequency; #𝑟𝑎𝑛𝑘𝑠 refers to the number of ranks in the target

target section should be more significant than those in related

section. In this way, we iteratively add records that are related to

articles. Before applying FP-growth [4] to related articles, wethe target section.

concentrate more on the target section. We give more weight on

Then apply FP-growth [4] on the target section, related articles,

co-occurring words in the target section. In order to obtain hidden

and adding records together for once, and we obtain the frequent

topics, we apply LDA (Latent Dirichlet Allocation) [1] onto thepatterns of order-free word sets.

target section and related articles, to estimate word-topic

3.3 Candidate keyphrases

distributions and topic-document distributions, which are used toWhen we search information on a search engine, such as Google,

obtain the likelihood of words occurring in the target section andBing, Yahoo etc, the engine will return results with their hit counts

article.

to us.

3.2 Frequent patterns
Frequent patterns are those appear frequently in a data set. The
frequent patterns do not reflect word orders but co-occurrence. For
example, a record contains a set of items, such as milk and bread,which appear frequently together in a transaction data set, is afrequent item set. The customer baskets (records) are as same as

Figure 3. Hits result

articles, where the basket items are as same as words. So if word

The frequent co-occurring word sets are order-free in FP-growth.

sets co-occur in many articles, it is likely to be a good candidate

But what we need are meaningful phrases with sequence as

keyphrase.

baskets:

candidate keyphrases, then we enumerate possible permutations{„milk‟,„bread‟,„sugar‟,„eggs‟},

and applied to search engine (we chose “Bing” search engine in our{„milk‟,„bread‟,‟butter‟}, {„sugar‟,„eggs‟}, if we set the frequency

research) as a query. We choose the highest hits as a representative

count equals two, we obtain the most frequent patterns

ordering of this word set.

{„milk‟,‟bread‟} and {„sugar‟,‟eggs‟}. The frequency count is

3.4 Quality measurements

named as support.

To evaluate quality of these candidate keyphrases, we introduce theForexample,{„milk‟,„bread‟,‟cereal‟},

there

are

four

Since phrases usually occur once or twice in one section, it is hardfollowing four measurements from different aspects: coverage,

to conclude significance of a phrase simply by its frequency in thephraseness, uniqueness, potentialness. Coverage is from [7],target section, or in the article. So we need to concentrate more on

phraseness is originally from [7] but we extended for our case.

the words occurred in the target section. Then we try to add more

Uniquness and potentialness are proposed in our previous work[11]records that contain frequent words occurring in the target

but further improved in this paper. In the following definitions, thesectionas an input to FP-growth[4]. For instance, the word

section corpus C consists of the target section, augmented with“white”, ”house” and “Wellesley” are frequent in the target section

thetop-krelated articles.

(i.e. with frequencies of 10, 13 and 8 respectively), so we may addmore records (i.e. 2, 3 and 1 records respectively) that contain theseA representative keyphrase should cover many articles. For

words. So the adding records are: {“white”, “house”, “Wellesley”},

instance, in the target section titled “early life and education” of{“white”, “house”} and {“house”}.

arcicle Hirrary Clinton, the phrase “political president” covers fourTo achieve that, first, we calculate the frequency of each word

articles, while phrase ”student year” covers eight articles, so theoccurringin the target section and give a rank for each word basedlatter is better than the former. The scoring function of coverageis

on the frequency. Then we apply the following function for ranking

as follows:

words:
𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡#records(w) =∗ (#𝑟𝑎𝑛𝑘𝑠 − 𝑐𝑢𝑟𝑅𝑎𝑛𝑘) (1)#𝑟𝑎𝑛𝑘𝑠
Where #records(w) means the number of adding records that word

Coverage[7]𝑆𝑐𝑜𝑣 (𝑝) =

𝑓(𝑝)
|𝐷|

(2)

Where 𝑓(𝑝)refers to the frequency of phrase p occurring in corpus
C; |D| refers to the number of articles in section corpus C.

DEIM Forum 2016 C3-6



Phraseness𝑆𝑝𝑜𝑡 (𝑝) =The words in a representative keyphrase are likely to co-occur.

∑𝑤∈𝑝 𝑆𝑝𝑜𝑡 (𝑤|𝑠)
|𝑝|

(7)

Since candidate keyphrases are generated from FP-growth, which

Here, |p| refers to the number of words in phrase p.

find frequent word sets, there is a high chance that words in

3.5 Ranking function

keyphrases are just combinations of frequent words, even they are

By each of the above four features, we can rank candidate phrases.

far apart in texts, like a frequent word at the beginning of the target

To obtain the combined ranking of phrases, we define a ranking

section while the other frequent word at the tail. So we restrict

function for candidate keyphrases.

words of a keyphrase to co-occur in one sentence. Besides, we need𝑆(𝑝) = 𝜃0 𝑆𝑝𝑕𝑟 (𝑝) + 𝜃1 𝑆𝑐𝑜𝑣 (𝑝) + 𝜃2 𝑆𝑢𝑛𝑖 (𝑝) + 𝜃3 𝑆𝑝𝑜𝑡 (𝑝)(8)to considerthe correlations between the target section and related

Here, 𝑆(𝑝) refers to the score of phrase p in the target section. Let

articles.

𝜽 = ,𝜃0 , 𝜃1 , 𝜃2 , 𝜃3 - denotes the weight𝑛𝑆𝑝𝑕𝑟 (𝑝) = ∑𝑖=0𝑛−𝑖𝑓𝑖 (𝑝)∗∏
𝑛𝑤∈𝑝 𝑓𝑖 (𝑤)vector on the fourfeatures.

(3)

3.6 Gradient descent

Here, i denotes similarity rank by TF-IDF cosine similarities to the

We define a ranking function which combines the four features by atargetrelatedlinear function.We apply gradient descent to obtain the optimalarticles, 𝑓𝑖 (𝑝)denotes the frequency of phrase p where p occurs inparameter settings for the ranking function. Firstly, we formalize

an identical sentence of article i, and 𝑓𝑖 (𝑤) denotes the frequency

the optimizing function as follows:

section,ndenotesthenumberof

top-kℎ(𝜽) = 𝜃0 𝑥0 + 𝜃1 𝑥1 + 𝜃2 𝑥2 + 𝜃3 𝑥3

of word w in article i.
A representative phrase should be more frequent in the target

and 𝑥𝑖 is the variable on the i-th feature.

section rather than other sections in the target article, to extract

The cost function is defined as follows:1

𝐽(𝜃) = ∑𝑛𝑖=0(ℎ𝜃 (𝑥𝑖 ) − 1)2

phrases that represent the target section well among other sections.
|𝑆|𝑓𝑠 (𝑤)
𝑆𝑢𝑛𝑖 (𝑤) = log (∗)
𝑓(𝑠) ∑𝑠′∈𝑆,𝑠′≠𝑠 𝑓𝑠′ (𝑤) + 1(4)

2

For j=1 to m{
𝜃𝑖 = 𝜃𝑖 − 𝛼to the number of sections containing word w, 𝑓𝑠 (𝑤) is thefrequency of word w in section s. Then the score of phrase p is theaverage score of words in phrase p.
𝑆𝑢𝑛𝑖 (𝑝) =∑𝑤∈𝑝 𝑆𝑢𝑛𝑖 (𝑤)
|𝑝|

(10)

We apply batch gradient descent as follows:

Here |S| is the number of sections in the target article, 𝑓(𝑠) refers

𝜕𝐽(𝜃)
𝜕𝜃𝑖

(11)

} (for all i)
Where m is the number of training examples (also corresponding to
the number of candidate keyphrases), 𝛼 is the learning rate. After(5)

obtaining the optimum weight vector, we rank candidate phrases by
the ranking function 𝑆(𝑝).

Here, |p| refers to the number of words in phrase p.
(9)where ℎ(𝜽) refers to the score of phrase p in the target section,

Uniqueness [11]Potentialness[11]Potentialness is intended to measure how phrases are related to

4. Experimentlatent topics of the target section. We assume that phrases that are

4.1 Dataset

highly related to latent topics of the section are more suitable for

We download 17 featured English Wikipedia articles (which are

section titles. We evaluate the likelihood of words from word-topic

selected as excellent articles in Wikipedia). Since we need to

distribution and topic-document distribution by Latent Dirichletcompare our results with original section titles that are hidden, we

Allocation(LDA) [1].

choose 51 sections which are rich in section titles and subsection
titles. External related articles are also downloaded from Wikipedia,𝑘

𝑆𝑝𝑜𝑡 (𝑤|𝑠) = ∑ 𝑝(𝑤|𝑡𝑗 ) ∗ 𝑝(𝑡𝑗 |𝑠)

(6)

𝑗=0

in which the words of the article title of the target article occur
consecutively.

Here, k refers to the number of topics, 𝑝(𝑤|𝑡𝑗 ) is the word-topic

4.2 Parameters

distribution computed by gibbsLDA, and 𝑝(𝑡𝑗 |𝑠) is the

The section corpus C of each target section consists of the target

topic-document distribution.

section and top-k related articles based on the combination of
TF-IDF and cosine similarity. Here, we set k = 15. For gradientDEIM Forum 2016 C3-6

descent, we set the learning rate α=0.0005.

We divide our dataset

10

1.057

0.175

0.248

0.355

into two parts, where Corpus 1 has seven articles, while Corpus 2

15

1.097

0.155

0.241

0.369

has ten articles to see if the parameters of gradient descent are

20

1.148

0.126

0.224

0.367

Average

1.070

0.163

0.240

0.361

different in different corpus.
Articles in Corpus 1: {"Hillary Clinton", "Attachment theory",

To see more clearly, We put the average result together for"History of Minnesota", "Domitian", "Political integration of India",

comparison.

"Philosophy of Mind", "Nikita Khrushchev"}.

Table 4. Comparison on different corpus

To verify if the parameters are related to the scale of the training

Corpus

𝜽𝟎

𝜽𝟏

𝜽𝟐

𝜽𝟑

dataset, we test on different scales of candidate keyphrases. We do

1

0.965

0.163

0.239

0.423

experiments on four different scales: top-k (here, we let k =

2

1.167

0.165

0.237

0.310

3

1.077

0.162

0.254

0.351

5, 10, 15, 20 respectively) candidate keyphrases which are ranked

by the ranking function in Section 3.5.

Above all, the average results of parameters in different corpus

Table 1. Parameters determined from Corpus 1
𝛉and different top-k phrases are close. So we apply parameters from

𝜃0

𝜃1

𝜃2

𝜃3

5

0.826

0.196

0.279

0.439

4.3 Result

10

0.961

0.176

0.246

0.412

After we apply 𝛉 to the ranking function in equation(9), we can

15

1.014

0.155

0.226

0.420

obtain the ranked list of keyphrases for each section. In fact, the

20

1.058

0.125

0.206

0.423

result includes 51 sections, we show three sections to inspect their

Average

0.965

0.163

0.239

0.423

qualities.

Top-k

the larger corpus 𝛉 = ,1.077, 0.162, 0.254, 0.351- to the ranking
function for every phrase.

We can see that the obtained parameters under different top-k are

Table 5. Result of section “Early life and education” in article“Hillary Clinton”

performed quite stable.
Articles in Corpus 2: {"Greek mythology", "Barack Obama",

Rank

Phrase"Bryan Gunn", "John Sherman", "Wood Badge", "General1

hillary clinton

relativity", "Society of the Song dynasty", "Richard Nixon",

2

college political"Uruguayan War", "William the Conqueror"}3

clinton political

We do the same on these 10 articles.

4

law school

5college school

Table 2. Parameters determined from Corpus 2
𝛉

𝜽𝟎

𝜽𝟏

𝜽𝟐

𝜽𝟑

5

1.055

0.213

0.263

0.293

10

1.160

0.174

0.235

0.310

15

1.258

0.122

0.215

0.310

20

1.197

0.152

0.235

0.316

Average

1.167

0.165

0.237

0.310

Table 6. Result of section “First lady of the United States” inTop-karticle “Hillary Clinton”

We can see that parameters are stable in different top-k and almost
the same to the corresponding parameters. Then we combine thetwo corpora, to find if the parameters are still stable.

1white house2

first house3

first white house4

first white

5

white house office

From the results of the two sections above, we can see thethe role of supplement, which means our phrase extraction model

Table 3. Parameters determined from Corpus 3
𝜃0

𝜃1

𝜃2

𝜃3

1.006

0.192

0.267

0.313

Top-k
5

Phraseextracted phrases are quite similar to target section titles and play

Articles in Corpus 3: all 17 articles

𝛉

Rank

performs pretty well. Keyphrases, such as “law school”, “whitehouse” etc. are quite complete and representative phrases, whichmeans phraseness play a quite significant role. The keyphrases we
extracted can distinguish sections even in the same article (Hillary

DEIM Forum 2016 C3-6

Clinton), which means uniqueness worked.

methods for measureing semantic distance, such as using
ontological knowledge such as WordNet.

But here we adopt

5 Evaluation

semantic distance measured by co-occuring frequencies in web

In this section, we discuss an objective evaluation based on

documents, a well-known measure of this direction is thesemantic closeness to the hidden section titles, where closeness is

NGD(Normalized Google Distance) [9]. Then we apply thebased on co-occurences over web documents.

distance results on nDCG(Normalized Discounted Cumulative

We compare

various combinations of the four features, where the mehodGain) [12] to measure the quality of ranking.

combining all the features by the parameter 𝛉 is called Combined

measure captures semantic closeness to the original hidden titles,

Measure .

but this does not evaluate comprehensivenss and informativeness of

5.1 Variations for comparison

phrases, which original titles sometimes lack, and we reserve these

We compare the following combinations of the features: Combined

evaluations for future work.

Measure-cov refers to the Combined Measure that ignores the effect

5.3 Relevance scoreof coverage, Combined Measure-phr refers to the Combined

In our work, we need to calculate the semantic distance between

Measure that ignores the effect of phraseness, Combined

the keyphrases and original titles, so we adapt NGD(Normalized

Measure-uni refers to the Combined Measure that ignores the effect

Google Distance) [9] which is a semantic similarity measureof uniqueness, and Combined Measure-pot refers to the Combined

derived from the number of hits returned by the search engine for

Measure that ignores the effect of potentialness.

given phrases or words. Phrases with the same or similar meanings

These variations represent the possible settings for parameters

in a natural language sense tend to be "close" in units of𝜽 = ,𝜃0 , 𝜃1 , 𝜃2 , 𝜃3 - which we described in Section 3.5.

Normalized Google Distance, while phrases with dissimilar

Table 7. Parameters settings for different variations

Not that thismeanings tend to be farther apart. The NGD score between two
phrases 𝑝1 and 𝑝2 is

Method,𝜽𝟎 , 𝜽𝟏 , 𝜽𝟐 , 𝜽𝟑 -Combined Measure,1.077, 0.162, 0.254, 0.351-

Combined Measure-cov,1.077, 0.0, 0.254, 0.351-

Combined Measure-phr,0.0, 0.162, 0.254, 0.351-

where N denotes the total number of web pages indexed by a given

Combined Measure-uni,1.077, 0.162, 0.0, 0.351-

search engine; 𝑓(𝑝1 ) denotes the number of pages containing

Combined Measure-pot,1.077, 0.162, 0.254, 0.0-

phrase 𝑝1 ; 𝑓(𝑝1 , 𝑝2 ) denotes the number of pages containing both

NGD(𝑝1 , 𝑝2 ) =max*log 𝑓(𝑝1 ),log 𝑓(𝑝2 )+−log 𝑓(𝑝1 ,𝑝2 )(12)log 𝑁−min*log 𝑓(𝑝1 ),log 𝑓(𝑝2 )+Besides, we also arbitrarily combine two of the four features asphrase 𝑝1 and 𝑝2 . NGD(𝑝1 , 𝑝2 ) = 0 means phrase 𝑝1 is the same

comparison.

as phrase 𝑝2

Table 8. Parameters settings for combination of two features

As NGD is a normalized distance score between phrases, we

Method,𝜽𝟎 , 𝜽𝟏 , 𝜽𝟐 , 𝜽𝟑 -define the relevance score between two phrases as follows:

CovPhr

,1.077, 0.162, 0.0, 0.0-rel(𝑝1 , 𝑝2 ) = 1 − NGD(𝑝1 , 𝑝2 )CovPot,0.0, 0.162, 0.0, 0.351]

After that, we can obtain the relevance score between keyphrases

CovUni,0.0, 0.162, 0.254, 0.0-

and its target section/subsection titles. Then we combine the

PhrUni,1.077, 0.0, 0.254, 0.0-

PhrPot,1.077, 0.0, 0.0, 0.351-

UniPot,0.0, 0.0, 0.254, 0.351-

5.2 Quality measurement
To evaluate how representative of these keyphrases we extracted,
we need a golden standard. But it is unrealistic for human to give
keyphrases for every section as a golden standard, since different
people may give different answers based on their own
understanding. So we adopt the idea that the original section andsubsection titiles are ideal, and phrases that are semantically close
to these titles should be high quality. There are a number of(13)

relevance score between keyphrase and its section title (rel(p, s))and the relevance score of keyphrase and its subsection titles( rel(p, sub𝑖 ) ) to calculate the relevance score between thekeyphrase and its target section.
𝑟𝑒𝑙(𝑝) = 𝛼 ∗ 𝑟𝑒𝑙(𝑝, 𝑠) + 𝛽 ∗∑𝑛𝑖=1 𝑟𝑒𝑙(𝑝, 𝑠𝑢𝑏𝑖 )𝑛(14)Here, n refers to the number of subsection titles; rel(p) refers to therelevance score between phrase p and the target section; rel(p,s)
refers to the relevance score between phrase p and section
title; 𝑟𝑒𝑙(𝑝, 𝑠𝑢𝑏𝑖 ) refers to the relevance score between phrase p
and ith subsection titles, 𝛼 and 𝛽 are weighting parameters. SinceDEIM Forum 2016 C3-6

we pay more attention to section titles than subsection titles, we set

UniPot0.6801𝛼 and 𝛽 to 0.7 and 0.3, respectively.

Coverage

0.8257Now every keyphrase has a relevance score to the target section,

Phraseness0.7870we can obtain a ranking of these keyphrases according to theUniqueness0.5660relevance scores.

Potentialness

0.7300

5.4 Normalized Discounted Cumulative Gain

Table 10. Section“marriage and family, law career and first

DCG (Discounted Cumulative Gain)[12] measures the gain of alady of arkansas”

phrase based on its rank position in the result list. The gain is

MethodnDCG

accumulated from the top of the result list to the bottom with the

Combined Measure

0.9271

gain of each result discounted at lower ranks. Namely, if a highCombined Measure-phr

0.8929Combined Measure-cov

0.6846

Combined Measure-pot

0.9005

Combined Measure-uni

0.9201

CovPhr

0.8920

CovUni0.8556

CovPot0.8575

PhrUni0.9390

PhrPot0.9203

UniPot0.8818Coverage

0.8737

Phraseness0.9307

Uniqueness0.8880Potentialness

0.8472

relevant phrase ranked low or an irrelevant phrase ranked high inthe result list will be discounted.
The ranking list we obtained according to the relevance scores is
the ideal ranking list, which used to obtain the ideal (standard)
DCG value.

𝐼𝐷𝐶𝐺𝑛 = ∑𝑛𝑖=12𝑟𝑒𝑙(𝑝) −1
𝑙𝑜𝑔2 (𝑖+1)(15)Here, i refers to the ranking position of phrase p according to therelevance score, and n is the number of phrases in the target section.
We can also obtain the DCG value by the above equation (15) but
based on the positions of phrases in our methods (Combined
Measurement and its variations).
Then we apply the value of DCG and IDCG in every section to
obtain the normalized result.
𝑛𝐷𝐶𝐺 =𝐷𝐶𝐺
𝐼𝐷𝐶𝐺

Then we take the average nDCG values of each method as the(16)

nDCG ranges in 0.0 and 1.0. In a perfect ranking algorithm, theevaluation score for comparison, as shown in Table 11.
Table 11. Average nDCG of all 51 sections of 17 articles

DCG will be the same as the IDCG, producing an nDCG of 1.0.

MethodAverage nDCG

5.5 Evaluation results

Combined Measure

0.8393After we apply above evaluation method to the 51 sections, we can

Combined Measure-phr

0.8345

obtain the nDCG value of each section. Here, we show the result of

Combined Measure-cov

0.8364

two sections in the article “Hillary Clinton”.

Combined Measure-pot

0.8229Combined Measure-uni

0.8381

Table 9. Section “first lady of the united states”
MethodnDCG

CovPhr

0.8276

Combined Measure

0.7495

CovUni0.8003

Combined Measure-phr

0.7451

CovPot0.8463

Combined Measure-cov

0.7656

PhrUni

0.8247

Combined Measure-pot

0.8079PhrPot0.8360Combined Measure-uni

0.7463

UniPot0.8302

CovPhr

0.7682

Coverage

0.8234

CovUni0.6743Phraseness0.8237

CovPot0.7150Uniqueness0.8210

PhrUni0.7909Potentialness

0.8380

PhrPot0.7613

From the tables above, we can see the Combined MeasureDEIM Forum 2016 C3-6

performs very well among these methods. Although there are othermethods maybe slightly better than our combined measure. Since

CIKM‟09, pages 1725-1728, 2009.
[7] M.Danilevsky, C. Wang, N. Desai, X. Ren, J.Guo, and J.

section titles have different characteristics, some of themHan,“AutomaticofTopicalcontaining popular phrases, others containing words from theKeyphrases on Collections of Short Documents”,

Proc.ofsection body, and some of them containing words unique to2014 SIAM Int. Conf. on Data Mining (SDM‟14),2014.

Construction

andRanking

sections. So the methods which are better than our method varies in[8] M. Symonds, P.Bruza, G.Zuccon, L.Sitbon, I. Turner, “Is thedifferent sections and overall results, while our method performs

Unigram Relevance Model Term Independent? Classifying

quite well and stable.

Term Dependencies in Query Expansion”, Proc. of theSeventeenth Australasian Document Computing Symposium

6 Conclusions(ADCS ‟12), pages 123-127, 2012.

We proposed a generative model which can automatically extract[9] R.L. Cilibrasi, P.M.B. Vitanyi, “The Google Similarity

representative phrases for sections in articles. The performance also

Distance”, IEEE Trans. Knowledge and Data Engineering,shows the ability to represent the section content and complete thepages 370-383, 2007.

insufficiency of section titles. This model can also be extended on[10] W. Zhao, J. Jiang, J. He, Y. Song, P. Achananuparp, E. Lim, X.

normal texts without section title, by setting the weight on

Li, “Topical Keyphrase Extraction from Twitter”, Proc. of theuniqueness to 0, and it‟s flexible on the length of text, while many

49th Annual Meeting of the Association for Computational

existing works need the text length limitation, either long document

Linguistics: Human Language Technologies (HLT '11), pages

or short texts, like microblogs.

379-388, 2011.

For future work, there are still several problems need to be[11] X. Han, R. Wang, M. Iwaihara, “Automatic Construction andimproved, including relatively bad performance of coverag. We

Ranking of Keyphrases on Wikipedia Article Sections”, DEIM

may give a reasonable weighting to the target section and different

Forum 2015.

related articles. We can see from the results that several ranked[12] Y. Wang, L. Wang, Y. Li, D. He, W. Chen, T.-Y. Liu, “Aphrases share quite similar meanings. So we need to find a way to

Theoretical Analysis of NDCG Ranking Measures”, Proc. of

reduce redundant phrases.

the 26th Annual Conference on Learning Theory,2013.
[13] Z. Liu, W. Huang, Y. Zheng, and M. Sun,“Automatic

7 References

keyphrase extraction via topic decomposition”, Proc. of the[1] D. Blei, A. Ng, and M. Jordan, “Latent Dirichlet Allocation”,

2010 Conf. on Empirical Methods in Natural Language

Journal of Machine Learning Research, 3, 2003.
[2] I.Hulpu, C.Hayes, D.Greene, “Unsupervised Graph-based Topic
Labelling using DBpedia”, Proc. of ACM WSDM‟13, pages
465-474, 2013.
[3] J. Lau, D. Newman, S.Karimi and T. Baldwin,“Best Topic Word
Selection for Topic Labelling”, Proc. of the 23rd Int. Conf. on
Computational Linguistics (COLING'10), pages 605-613, 2010.
[4] J. Han, J. Pei, and Y. Yin, “Mining Frequent Patterns without
Candidate Generation”, Proc. of the 2010 Conference on
Empirical Methods in Natural Language Processing (EMNLP„10), pages 366-376, 2010.
[5] J. Liu, J. Shang, C. Wang, X. Ren, and J. Han, “Mining Quality
Phrases from Massive Text Corpora”, Proc. of the 2015 ACM
SIGMOD Int. Conf. on Management of Data (SIGMOD'15),
pages1729-1744, 2015.
[6] K. Hofmann, M.Tsagkias, E.Meij, M.Rijke, “The Impact ofDocument Structure on Keyphrase Extraction”, Proc. of ACM

Processing (EMNLP‟10),pages 366-376, 2010.