DEIM Forum 2017 F6-3

Finding titles representing segments of Wikipedia
Articles from keyphrases
Hao HuShan Liu

Mizuho Iwaihara

Graduate school of Information Production and System, Waseda University
2-7 Hibikino, Wakamatu-ku, Kitakyushu-shi,Fukuoka-ken, 808-0135 Japan
E-mail:waseda-tezuka@fuji.waseda.jp,

vickey-liu@toki.waseda.jp, iwaihara@waseda.jp

Abstract: Wikipedia is a free online encyclopedia that aims to allow anyone to edit any article or create them.
However, articles tend to become long and complex, so giving appropriate titles or key phrases to untitled segments isnecessary for reader assistance. In this paper, we show methods to select titles for representing article segments. Key
phrase extraction has been studied for years, but we concentrate on selecting a title phrase for a given target segmentfrom candidate phrases, which needs to reflect local and global contexts. In this paper, we evaluate five features we
proposed before, and one new feature which is based on word embedding. These features are combined to produce aranked list of candidate titles. We construct over a candidate title set consisting of titles of articles, sections andsubsections, and anchor texts of inner links (inter-article links) where the hidden title of the target segment is the groundtruth. We compare performance of various feature combinations by precision@K, reciprocal rank and average precision.

Keyword: Wikipedia, Finding Representative Title, Candidate Ranking
1. Introduction
Wikipedia is a free, web-based, global volunteer
collaborative repository of various articles on
different aspects. This large and general reference
work consists of more than 40 million arti cles inmore than 250 different languages. Due to the large
number of visitors and everyone could edit article s
in his own opinion, articles tend to become longand complex for readers to capture the content witha clearer view. So, a title whi ch could represent thewhole content is so important when author intends
to remind readers what he or she is going toemphasize. In Wikipedia, a large number ofarticles are usuall y constructed of several sections
with subsections inside. These titles vary in forms
from sentences to simpl y single words to refer. In
addition, a subst antial number of articles areunfinished and changed over time. Like an article
for biographies of people, several events in which
a person is involved usuall y do not end up when
the article was created. The author would fill his or
her work with an ending, while the title could notcover the current content or sometimes leave it as a
blank. So, it is necessary for us to find an
automatic method to generate a ne w title for anarticle and its sections, and subsections.
In conventional methods, most approaches liketo extract ke yphrases from the target article itself
to get close to the objective view of humans. Ke yphrase extraction is an intensively studied area in
the field of text mining. Extraction methods areusually based on statistical combined with NLP(Natural Language Processing) field. For instance,
the basic TF-IDF (term frequency and inverse
document frequency) combined with chucking to
generate n-gram keyphrases. Another example is
the generative topic modeling, which assumes thateach document is a mixture of a small number of
topics and that each word’s cre ation is attributable
to a combination of the document’s topics. Thesewords could later build a word set for ke y phrase
generation. In our previous work[7], weconcentrated more on the phrase generation part tomine possible keyphrases. Following, we used
FP-growth[11] for frequent co -occurring word
extraction. Keyphrases are built from a set ofwords, which has an advantage of flexible phrase
combinations no matter where a possible word
occurs in the se gment. In addition, we incorporateneighboring articles for ke yphrase extraction that

means similar or linked articles in Wikipedia.
Applying
phrasegeneration methods
like
FP-growth[11] could mine possible word from theneighbor we build.
However, different from our previous work ,finding ke yphrases most important or fitting as atitle of the target se gment is still requiring
improvements. Ke yphrase extraction is so
subjective and varies with each individual. In thispaper, we inte grate five features proposed in [7]with a new fe ature based on topi c vectors utilizing
word embedding to estimate phrase qualit y from acandidate set whi ch is fit as a title of the target
segment. We use a linear function to combine thesefeatures and train the m by gradient descent toobtain appropriate parameters for an article corpus.
These features are combined to produce a ranked
list of candidate title s set with di fferent
combinations. Top-ranked phrases are generated as
titles. In evaluation part, we construct over a
candidate title set consisting of titles from articles,
their sections, subsections, and anchor texts of
inner links (inter-article links) where the hidden
title of the target segment is the ground truth .
The remainder of this paper is organized as:
Section 2 is mainl y about related work, Section 3
is the review of our previous work, Section 4 is thecandidate object study in our work, Section 5 isabout experimental studies, Section 6 is theconclusion and future work.

2. Related WorkFor existing approached to supervised andunsupervised phrase extraction job, the methods
can be cate gorized as follows.
1. An unsupervised method for ke yphrase
extraction is utilizing TF-IDF [8] to rank
candidate ke yphrases (here chucks ascandidates) and select top-ranked ones.
2. TextRank [9] is one of graph-based ranking
methods for extraction intended to build aword graph in which edges represent semantic
relatedness between two co-occurring words.
3. KEA [12] (Ke yphrase Extraction Al gorithm)
KEA algorithm is a supervised extraction
algorithm for long documents. A Naïve-Bayesclassifier on term frequency and term position
features is trained o produce phrase ranking

lists.
4. LSA(Latent Semantic Analysis) [13] &LDA(Latent Dirichlet Allocation) [2]:The
former is a topic modeling technique , learning
word and document representations byapplying singular value decomposition to awords-by-documents co-occurrence matrix.
LDA is a generative topic model assumingwords in each document were generated by a
mixture of topics, where a topic is represented
as a multinomial probability distribution overwords.
5. FP growth(frequent pattern mining) [11] Thismethod intends to extract frequently occurring
word sets to obtain an order-free word set. In
our previous work, we adopted this method,as our target texts are varying in length,
meaning that certain se gments are just a fe w
sentences over dozens of sentences. In addition,
FP-growth can be applied to articles which are
related to the target article, to discover morecandidate phrases.
The above methods show different approaches tophrase generation. However, TF-IDF prefers highfrequency term, KEA method is a supervised
method and a large number of efficient training
segments are necessary. LD A could be used to
discover topically-related, but not appearing in thetarget segment can be discovered, but its training
corpus should be sufficiently large. Related
Wikipedia articles have to be carefully selected for
topic extraction.

3. Phrase Generation
In this section, we discuss finding keyphrases
for the target segment as a candidate title set. We
could consider the usage of existing methods
described in Section-2. In our previous work[7],we apply frequent pattern mining on the target
segment. Below we briefly describe the method.

3.1 Related articles
Here we all related articles as articles either
linked from the target article, or having significant
overlaps with the title words of the target article.
We can sample candidate phrases from the these
related articles. Such related articles can provide
candidate phrases which may not occur in the text

of the target segment. We appl y standard
preprocessing, such as stop words, to the corpus of
the related articles.

𝑆𝑢𝑛𝑖 (𝑤) = log (

∗

𝑓𝑠(𝑤)

) (2)

∑𝑠′ ∈𝑆,𝑠′ ≠𝑠 𝑓𝑠′ (𝑤)+1

The uniqueness of phrase p is defined as theaverage of the uniqueness scores of the words in p:

3.2 Frequent Patterns
Frequent patterns on words are order-free setsof words which frequently co-occur frequently in
documents[11]. FP-growth can extract frequent
word sets be from the corpus, then they can beutilized as candidate phrases. This approach allows
us to find phrases having different word orders andcontaining unrelated words in the middle. But forfinding the most popular word orders on phrases,
we can utilize query results on search engine s, andn-gram corpus such as Google -N gram.

|𝑆|

𝑓(𝑠)

𝑆𝑢𝑛𝑖 (𝑝) =∑𝑤∈𝑝 𝑆𝑢𝑛𝑖(𝑤)

(3)

|𝑝|

Potentialness [7]:The feature potentialness is to evaluate
relatedness of a phrase to a se gment by cosine
similarities on latent topic vectors. We construct
topic vectors on the corpus C by Gibbs LDA[2],with a given topic number k. Potentialness cancapture relatedness of phrase words even when thewords are not appearing in the segment.
𝑘𝑆𝑝𝑜𝑡 (𝑤|𝑠) = ∑ 𝑝(𝑤|𝑡𝑗) ∗ 𝑝(𝑡𝑗 |𝑠)

4. Phrase Quality Estimation
To compare the fitness of candidate phrases as
the title of the target segment text, we utilize
existing features and introduce a new feature . The
fiveexistingfeaturesare
Coverage[10],Phraseness[7], Uniqueness[7], Potentialness[7],Sim-PF[7]. We then introduce the new feature
Embedding-Vector, which evaluate semantic
relatedness between the word vectors, generated byparagraph2vec[4] of the target and related articles.
In the following, the corpus is denoted by C.

4.1 Quality Phrase Features
Coverage [10] :A representative keyphrase should cover man y
articles. Coverage gives a hi gh score to phrases
that occur frequentl y in the corpus C of the related
articles:𝑆𝑐𝑜𝑣 (𝑝) =

𝑓(𝑝)
|𝐷|

(4)

𝑗=1

(1)

Where 𝑓 (𝑝) refers to the frequency of phrase p
occurring in corpus C and |𝐷 | refers to the numberof articles in C.
Uniqueness [7] :A representative phrase should be more frequentin the target se gment rather than the othersegments in the same arti cle.
Uniqueness captures such locality of the phrase
within the target article. In (2), here 𝑓 (𝑠) is thenumber of sections in the target article. |𝑆| is thenumber of sections in the target article. 𝑓𝑠 (𝑤) is
the frequency of word w in se gment s.

In (4), k is the latent topic number, 𝑝(𝑤|𝑡𝑗 ) is theword-topic distribution computed by Gibbs LDA.
The potentialness for phrase p is defined as theaverage of the potentialness values of the words inp:
𝑆𝑝𝑜𝑡 (𝑝) =∑𝑤∈𝑝 𝑆𝑝𝑜𝑡 (𝑤|𝑠)
|𝑝|

(5)

Phraseness [7] :A word set is considered as a good phras e if thewords in the set often co-occur in an identicalsentence. The phraseness defined in (6) evaluates
how often the words of a phrase p appear in one
sentence:𝑛𝑆𝑝ℎ𝑟 (𝑝) = ∑𝑓𝑖 (𝑝)

i=1 𝛱𝑤∈𝑝 𝑓𝑖 (𝑤)(6)Here, i denotes the i-th article in corpus C, ndenotes the number of the articles in C, 𝑓𝑖 (𝑝)denotes the times phrase p occurs in an identicalsentence of article i, and 𝑓𝑖 (𝑤) denotes the times
word w occurs in article i.
Sim-PF (Similarity-weighted Phrase Frequency)[7] :This feature gives a high score to a phrase if p isfrequently occurring in se gments having highsimilarities with the target segment. We rank thearticles in C, from i=1 to i=n, by the TF-IDF cosine
values with the target segment. Then we computethe product of the ranking and log-frequency of p
in article i.
𝑛

𝑆𝑎 (𝑝) = ∑

𝑛−𝑖

𝑖=1 𝑛

∗ log(𝑓𝑖 (𝑝) )(7)

Utilizing similarit y ranking instead of raw
similarity scores gives us an ideal wei ghting on
articles.

Embedding-Vector:
Inspired by [4 ], we transform the text of asegment s into a fixed-length feature vector. The
previous
fivefeaturesmainl y focus
on
distributions of words in the target se gment andrelated articles. On the other hand, paragraphvectors allow us to measure semantic relatedness
between the target segment and candidate phrases
in a coherent manner, where vectors can be trained
over the whole English Wikipedia articles. In thisfeature, we evaluate by cosine similarities between
the vectors of the phr ase and se gments, wheresegments are weighted by similarity ranking to thetarget segment:
𝑆𝑣𝑒𝑐 (𝑝) = ∑𝑛𝑖=1

𝑛−𝑖𝑛2∗ 𝐶os(𝑉𝑝 , 𝑉𝑠𝑖 )(8)Here, 𝑉𝑝 is the vector for phrase p, 𝑉𝑠𝑖 is thesegment vector for the i-th segment in the top-n
similar articles, where the first segment is thetarget se gment itself, and the second and following
segments are the remaining segments in the corpus
C, ranked by the TF-IDF scores with the target
segment. Finally, the score is normalized t o 1 bydividing by n. In this paper, we choose n to be 10.

4.2 Linear function for ranking
The six features are combined with a linear
function to calculate score s for candidate phrases.
The candidate phrase having the hi ghest score will
be selected as the title of the target segment. . The
score function is defined as follows:
𝑆(𝑝) = 𝜃0 𝑆𝑐𝑜𝑣 (𝑝) + 𝜃1 𝑆𝑝ℎ𝑟 (𝑝) + 𝜃2 𝑆𝑢𝑛𝑖 (𝑝)+𝜃3 𝑆𝑝𝑜𝑡 (𝑝) + 𝜃4 𝑆𝑠𝑖𝑚 (𝑝) + 𝜃5 𝑆𝑣𝑒𝑐 (𝑝)
(9)
Here,
𝜽 = [𝜃0 , 𝜃1 , 𝜃2, 𝜃3 , 𝜃4 , 𝜃5 ]denotestheweight ve ctor on the six measurement features.
To find an optimum weight vector 𝜽, we appl ygradient descent [7].

4.3 Classifiers of features
Besides linear function, we can also consider
utilizing various binary classifiers for combing
the features. Since our goal is to find a title forthe target se gment, we try to rank all candidates
according to the fitness of each classifier.

Four classifiers here are applied in this paper:
Logistic Re gression, Naïve Bayes, Support Vector
Regression and Random Forest.
In Logistic Regression and Naïve Bayes, werank candidates by possibility score only when
the candidate is a quality one ( candidate label is
1).
Support Vector Regression is a variation of
SVM (Support Vector Machine) for re gression.
We choose RBF kernel for our task and rank
candidates according to re gression score s.
Except Random Forest, the other classifiers aretrained by 63 different fe ature combinations to
obtain the best result. Random Forest generates anumber of decision trees, where each tree isformed by the random feature combination.
Random Forest is also trained here for the
regression task to rank candidates.

5. Experiment
In this part, we perform experiments to test theperformances of various feature combinations by
precision @K, re ciprocal rank and AP (Average
Precision).

5.1 Dataset:
We extend the dataset in our previous work[7],choosing typical English Wikipedia articles which
could represent different aspects like biography ofpeople, historic event, Science, Politic.
Data set corpus is filled with a large number ofsegments and these segments have subsection titles
and their upper titles, inner link titles in thesegment. In addition, for each article in this corpus,
we would download its related article s to form theneighbor. Usually, the related article ’s size isthousands to support. Corpus C represents for theneighbor Top-K most similar article to target
segment. In this experiment, we set the K value 10.
The following table shows our dataset details :CorpusQuantity
Se gments
266
Candidate titles
7,179
Table.1
Dataset Corpus
In this experiment, we construct a Corpus set
consisting of 3 parts.
1. The segment title and its upper title if
possible2. The inner link title appears in the segment
which is prett y related to the target segment
itself.
3. Other se gment titles in the same article,
they are like bad phrases.
Average number
Quantity
Se gment title and 2
upper title
Other titles in the 9same article
Inner link titles
15
Candidate Titles
26
Table.2 Candidate Title Set and Average Numberof candidates.

5.2 Golden Standard
These three parts showed in Table.2 construct a
candidate title set for a target segment. We treat
part 1 titles(segment titles and their upper titles) as
golden standard and test if our work could findthese original titles at Top-K in the ranking list.

5.3 Evaluation
Since our work is evaluated by the position ofcorrect answers in the ranking list, here we
introduce 3 conventional methods to estimate
performance.
Precision@ K: Precision at k documents (P @K) is
still a useful metric (e.g. P@5 or "Precision at 5"
corresponds to the number of relevant results on
the first search results page), but fails to take into
account the positions of the relevant documents
among the top k. Another shortcoming is that on aquery with less relevant results than k, even a
perfect system will have a score less than 1. It iseasier to score manuall y since onl y the top k
results need to be examined to determine if they
are relevant or not.
In this experiment, we are concerned about
Precision@ K result and set K value at 1, 3, 5.
Reciprocal Rank: Considering the position of thefirst relevant document position, Reciprocal rank
is the inverse rank number
1
𝑅𝑎𝑛𝑘 𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑛𝑢𝑚𝑏𝑒𝑟

Reciprocal Rank =(13)

AP(Average Precision): Average precision for aset of queries is the mean of the average precision
scores for each query.
𝑄𝐴𝑃 =𝐴𝑣𝑒 ∑𝑞=0 𝐴𝑣𝑒𝑃(𝑞)𝑄(14)Here in formulation 13, q query in our experimentmeans every se gment, 𝑃(𝑞 ) denotes for Precision@K. Q is the number of total se gments.
Like in table.3, we have a ranking list for se gment“Early life” in the article “Hillary Clinton”. This
table would be shown as the following example:
Precision@1, 3, 5. Average Precision, Reciprocal
Rank. The original title is listed at pos ition second
and we could get the data as table.4 shows.
Candidate title
Score
Westchester County
0.1107
2006 re-election campaign
0.1040
United States Senate
0.0793
Jeanine Pirro
0.0704Jonathan Tasini
0.0670John Spencer
0.0450District Attorne y
0.0442
Table.3 Candidate Ranking of Segment: “2006re-election campaign” in “Hillary Clinton” article
by five features.
Earl
Precisi Precisio Precisio Aver Invers
y life on
n
n
age
e
@1
@3
@5
Preci Rank
sion
Cov
1/1
1/3
1/5
0.51
1/1
Cov
1/1
1/3
1/5
0.51
1/1
Phr
Cov
0/1
1/3
1/5
0.17
1/2
PhrP
ot
Cov
1/1
2/3
2/5
0.68
1/1
PhrS
im
Cov
0/1
1/3
1/5
0.17
1/2
PhrU
ni
Covp 1/1
1/3
1/5
0.51
1/1
ot
SixF 0/1
2/3
2/5
0.35
1/2
eatur
es
Table.4 Evaluation table on segment “2006re-election campaign” in article “Hillary Clinton”.

5.4 Training partFor dataset which consists of 266 se gments, we
split them into two parts.
The First part, we have 196 se gments fortraining. They are selected from the article "Greek
mythology", "Barack Obama", "Bryan Gunn","John Sherman", "Wood Badge ", "Generalrelativity", "Society of the Song dynasty","Richard Nixon", etc.
These se gments are used for Gradient Descenttraining to obtain the fixed parameters for eachfeature.
In addition, we leave 70 segments to test theperformance of our work. The y are from the article"Hillary Clinton", "Attachment Theory", "Historyof Minnesota", "Domitian", "Political inte grationof India", etc.
At last, We have the six features in our
experiment which me ans 63 different combinations.
Table 3 would show the final fixed parameters for
8 examples.
Cov
Phr
Sim
Uni
Vec
Pot
Six

-0.04

NoCov

-0.01

0.11

0.06

0.05

0.05

-0.01

0.08

0.06

0.04

0.04

0.1

0.06

0.05

0.05

0.06

0.06

0.06

0.06

0.05

NoPhr

-0.04

NoUni

0.01

0.03

NoSim

-0.04

0.00

0.11

NoVec

-0.03

-0.01

0.11

0.07

NoPot

-0.02

-0.02

0.12

0.07

0.06
0.08

Table.5 Seven examples of different feature
combination parameters after training from linear
model.

5.5 Result
In this part, we would show the final work
performance on 63 di fferent feature combinations.
The table 4 shows the Top-8 hi ghest AP score in
Precision @1 ,3 ,5 and Reciprocal R ank.
Method P@1
P@3 P@5AP
InverseRank
CovPhr 0.289
0.202 0.156 0.2164 0.4512
Vec
CovVec 0.2753 0.183 0.294 0.2119 0.4470
CovVecPot
Cov

0.2753

0.183

0.289

0.21090.44750.2318

0.188

0.270

0.19420.4046

CovPot0.2318

0.188

0.270

0.19420.4046

VecPot0.2463

0.173

0.251

0.19030.4188

Table.6 Top-6 hi ghest Average Precision fromlinear function.
Method
P@1
P@3 P@ AP
Inverse5
Rank

Logistic R 0.275 0.227 0.28 0.2244 0.4689
CovVecPot
Linear
0.289 0.202 0.26 0.2164 0.4512
CovPhr
Logistic R 0.289 0.193 0.26 0.2131 0.4470
CovPhrVecNaïve B
0.246 0.227 0.27 0.2128 0.4502
CovVecPot
Linear0.275 0.183 0.29 0.2119 0.4470
CovVecSVR
0.231 0.207 0.32 0.2112 0.4463
VecPotRandom0.159 0.154 0.24 0.1539 0.3648
ForestTable.7 Top-7 highest Average Precision acrossdifferent classifiers
From table 6, we see that our work could findthe correct title from the title candidate set inwhich the hi ghest AP score reaches 67.34%
precision. In addition, Embedding-vector, coverageand Phraseness features support the function to
obtain the best result.
Reciprocal rank here is the mean average scoreacross different segments. The larger the score is,
The higher position golden standard appears in the
ranking list. That is, we could find the correct
answer quickly. In our experiment Method
Embedding-Vector, Potentialness and Coveragefeature combination in Logistic Re gression shows
the best result.
Besides the linear model Logistic Regression,
we also study different classifier performances.
Performed as Table 7, linear function with
Coverage,
Phrasenessfeaturecombination
performs well. However, other classifiers likeNaïve Bayes also has good result as its inverse
rank score is close to the linear function highest
score ranking at 2nd. While other classifier ’s
performance, such as random forest is not very
ideal.
Since the limit of pages, we could not show all
the 63 combination results.
6. Conclusion and Future Work
Our work aims to find an appropriate title forthe target se gment is working according to thefinal result. When adding the appropriate
candidate phrase like FP-growth, we could findquality phrases to form a ne w title automatically.
Although, In table.6, feature Embedding-vectormay show better performance. We still have topoint out that the Embedding-Vector method is
not stable. It is necessary to retrain the article
corpus until a balanced state.
Referenced Work
[1] R. Baeza-Yates, B. Ribeiro-Neto, et al.
Modern information retrieval .
[2] D.M.Blei, A.Y.N g, and M.I Jordan. Latent
Dirichlet Allocation. Journal of Machine
Learning Research, 3:993-1022, 2003.
[3] S.
Liu,M
Iwaihara.
“Extracting
Representative Phrases from Wikipedia Article ”
2016 DEIM Forum.
[4] Le, Mikolov “Distributed Representations ofSentences and Documents”. The 31th
International
ConferenceonMachineLearning,[5] I.Hulpu, C.Hayes,D.Greene, “Unsupervised
Graph-based Topic labelling using DBpedia”,
Pro. of ACM WSDM’13, pages 465 -474, 2013.
[6] Z. Liu, W. Huang, Y. Zheng, and M.
Sun,”Automatic ke y phrase extraction via
topic decomposition”, Proc. Of the 26 t hAnnual Conference on Learning Theory, 2013.
[7] S.
Liu,
M.
Iwaihara,“
Extracting
Representative Phrases from Wikipedia Article
Sections”, ICIS Conference, Special Sessions,
June 2016.
[8] Salton and C. Buckle y “Term-wei ghting
approaches in automatic text retrieval. ”
Information pro cessing and management,
pages 513-523 1988
[9] Mihalce a and P. Tarau. “Textrank: Bringing
order into texts” In Proceedings of ACL, pages
825-833[10]
M. Danilevsky, C. Wang etc, ”Automatic
Constuction
andRankingofTopical
Ke yphrasesonCollections
OF
Short
Documents”, Proc.of 201 4SIAM Int. Conf. on
Data Mining (SDM’14),2014[11]
J.Han, J.Shang etc,“Mining Frequent
Patterns without Candidate Generation”
Conference on Empirical Methods in Natural
Language
Processing(EMNLP),
pages
366-376 2010[12]
Ian H.Witten, Gordon W.Paynter etc“KEA:
Practical
Automatic
Ke yphrase
Extraction” ACM DL 1999 pages 254 -255
[13]
S.C. Deerwester, S.T. Dumais etc,
“Indexing by latent semanti c anal ysis” Journal
f the America Societ y of Information Sciencepages:391-407,1990