DEIM Forum 2017 G5-1

Estimating Reference Scopes of Wikipedia Article Inner-links
Renzhi Wang†

Mizuho Iwaihara‡

Graduate School of Information, Production and Systems, Waseda University
2-7 Hibikono, Wakamatu-ku, Kitakyushu-shi, Fukuoka-ken, 808-0135 Japan
E-mail:†ouninnyuki.ips@asagi.waseda.jp, ‡ iwaihara@waseda.jp
Abstract Wikipedia is the largest online encyclopedia, and utilized as machine-knowledgeable and semantic resources.
Links within Wikipedia indicate that two articles or parts of them related about their topics. Existing link detection methods
focus on article titles because most of links in Wikipedia point to article titles. But there are a number of links in Wikipedia
pointing to corresponding segments, because the whole article is too general and it is hard for readers to obtain the intention of
the link. We propose a method to automatically predict whether the link target is a specific segment and provide which segment
is most relevant. We propose a combination method of Latent Dirichlet Allocation (LDA) and Maximum Likelihood Estimation (MLE) to represent every segment as a vector, and then we obtain similarity of each segment pair. Finally we utilize variance, standard deviation and other statistical features to predict the results. We also try Word2Vector model to embed all thesegments into a semantic space and calculate cosine similarities between segment pairs, then we utilize Random Forest to train
a classifier to predict link scopes. Through evaluations on Wikipedia articles, our method achieved reasonable results.
Keyword Wikipedia, link suggestion, LDA, word2vector, PMI

1. Introduction
Wikipedia articles are edited by various volunteers fromall over the world, with different thoughts and styles. One
of Wikipedia’s characters is featured articles. Wikipedia
includes many high quality articles that reach the standardof featured article criteria. These articles are usually e dited by experienced authors and checked by Wikipedia’s
administrators. Featured articles are supposed to bewell-written, comprehensive, well-researched, neutral andstable. Wikipedia is structured via a number of links b etween different articles, which imply that the two linked
articles are closely related. Majority of links within Wi k-

Fig. 1. Wikipedia links

ipedia are pointing to article titles, and only small fra c-

Current link detection methods are focused on article

tions point to segment titles. However, when readers

titles [1,2,7,11,12]. They often first generate a candidate

browse topic via links, sometimes they are only interested

set for an article by utilizing existing connections of art i-in certain segments while the link itself is pointing tocles, then rank all articles in the candidate set, and selectarticle titles, thus the readers could get lost in such longthe most similar article as the final result. But in our case,articles. To avoid such situations, administrators and ed i-

it is hard to generate a candidate set by using existing

tors often modify link target text from an article title to aconnections, due to shortage of links that point to segment

specific segment title. Figure 1 shows an example that antitles. Besides, the length of segment texts are usually

editor modified the link target from the article title to theshort, so it is necessary to improve vector representation

specific segment. In article “Super Mario 64” there is aof segment texts, so that we can obtain better similarity

link, first pointed to article GameCube, and then the editor

comparisons between segment pairs which helps in ca n-corrected the link to the segment “Controll er”.

didate set generating process.
In this paper, we discuss the following link suggestion
problem: Given a link source, which is a position in a
Wikipedia article, we find the most related link target,which is either a whole article, or a segment in an article.

pose a method which can rank pages using co -citation andOur previous research utilized LDA based features to train

page title information. They use LTRank to identify sim i-a classifier to predict link target[ 16]. In this paper ourlar pages and select top similar articles as the prediction

approach employs Latent Dirichlet Allocation (L DA) [3,4]results. This method needs the organizational stru ctures ofmethod for topic detection, where segments are repr e-

the articles; LTRank is not suitable for detecting links at

sented as vectors of word probabilities. To improve acc u-

the segment level, because there are not enough se g-

racies of conventional LDA-based methods, we havement-level links.

combined LDA with Maximum Likelihood Estimation

Junte proposed a method utilizing TF -IDF and the vec-

(MLE) in a nonlinear way, which enables us to compute

tor space model to detect document -to-document links andsemantic similarities on the segment level [16]. In thisanchor-to-BEP links [19]. Best Entry Point (BEP) is sim-

paper,

word

ilar to our task. The best entry point here is a specificco-occurrence, we utilize features based on Normalized

article belonging to a general article. The difference is

Pointwise Mutual Information (NPMI)[5], to measurethat Junte’s work is still focusing on the whole article,

likelihoods of words co-occurring in different segments.

while our task is to detect the best segment in an article.

We also evaluate similarities

between different segments,

Junte’s research regards the source article as the query andutilizing word embedding. Word2vec[13,14] is an open

selects top-K similar articles as the result. Then deep-firstsource project released by Google which achieve s state ofiteration is repeated to find the final result. This method

the art performances in various natural language tasks. A

uses TF-IDF and the vector-space model to compute sim-

number of researches proved that word2vec performs very

ilarities. But TF-IDF is heavily affected by corpus con-well in calculating text similarity, especially on texts thatstruction.

have

considering

specificabout

effectivenesspatterns

of

such

as

David et al. [12] proposed a machine learning-based

wordvec(“king”)-wordvec(queen) ≈link detector to detect links between Wikipedia articles. In

wordvec(“man”)-wordvec(woman).

their method, they did not simply evaluate textual simi-

It is hard to predict whether a link should point to alarity between two articles, but for each article pair, they

segment, by only using similarities between segment pairs.

evaluate five features: link probability, relatedness, di s-

We compute similarities between one target segment andambiguation confidence, generality, location and spread.

all the segments in another article, to obtain similarity

Then they train a classifier, for predicting whether there

distributions in one article. We define statistical features

should be a link between an article pair. Its result was

based on these similarity distributions. Then we train amuch better than other similarity-based methods.

classifier to determine whether the link should point to a

3. Proposed method

specific segment rather than the whole article. When we

3.1 Target corpus

confirm that the link should point to a segment, we com-

As the world’s largest encyclopedia, Wikipedia is org a-pare the similarities between segment pairs to find thenized as a large, complex network, where articles aremost related segment. To solve the imbalanced data prob-

connected by interlinks. Given a target, we assume thatlem, we utilize the logistic regression as a filter before

interlinks from a target article to another article assists

final prediction.

complementing the content of the target article, by inco r-

In order to improve the result, we propose a stacking

porating the content of another article. In order to utilize

model to replace a single classifier. Our evaluation results

this link structure for incorporating the contents of theshow that our method is effective.

linked articles, we construct a suitable corpus for the

The rest of this paper is organized as follows. Section 2

target article. Given a target article A, we regard the unionshows related work. Section 3 introduces our assumptionof all the articles that A links to and A itself as the corpus.

and proposed method. In Section 4 we describe our dataset

Certain links may point to a specific s egment, but wein detail, explain our experimental process and we present

include its whole article in the corpus. Wikipedia featured

evaluation results in various situations. In Section 5, we

articles are less erroneous and stable, so they are suitableaddress a conclusion and future work.

for our experiments. Our dataset consists of randomly

2. Related work

sampled featured articles. Since our objective is to suggest

Automatically discovering missing links in Wikipedia

a segment-level link, we decompose the articles in thehas been discussed in the literature. Sisay et al. [ 1] pro-

corpus into segments based on their logical structures,such as paragraphs.

The following steps are operated on

segments.

ous part and latent part:𝑝(𝑤|𝐷) =3.2 Representing segments as vectors
We argue that linked articles bring additional infor-

1
1
1[
𝑃 (𝑤|𝐷) + 𝛽𝑁𝑑 𝑃𝑀𝐿 (𝑤|𝐶𝑜𝑙𝑙)]𝑒 𝛼𝑁𝑑 +1 𝑒 −𝛽𝑁𝑑 +1 𝑀𝐿
𝑒
+1

Here,

(2)

is the number of terms appearing in the se g-mation to the central article and affect the topics of thement. The first part of the formula is the probability of thecentral article. The LDA model [3,4] is a popular model

word appearing in the document by term frequencies. The

that can extract topics from the corpus. Figure 2 shows theweight proportion of the obvious part and latent part will

structure of LDA model, where documents are regarded asaffect the word probability. This document -level proba-

topic distribution and topic is regarded as a word distr i-bility is combined with the collection -level probability bybution.

the smoothing parameter . We adjust the value oftotion𝜃, and a topic z is represented over words by word

optimize the obvious part. Smoothing parameteris todistribution𝜙.

adjust the ratio of the obvious part and latent part. By

Adocumentdissampled

from

a

topic

these two formulae, we can obtain the probabilities of all
the words in the corpus. People utilize perplexity to evaluate the language model. The perplexity is smaller when
the fitness between the model and data is better. In our
experiment, we need to determine the best parameters. We
put the experiment data into the formula and select theparameters based on which the perplexity is minimal .
We represent each segment as a vecto r, whose element
is the probability of the segment generating the word. We
could use all the words in the corpus as elements of thevector. But to reduce the dimensions of the vectors, weonly select words which appear in more than three se g-

F i g . 2 . LDA generation process

ments.

So we can obtain the probability of a term in a doc u-

Another popular method of present document as a vector is the word2vec model. Its input is large text corpus

ment by the following formula:̂̂
𝑃𝐿𝐷𝐴 (𝑤|𝑑, 𝜃̂, 𝜙̂) = ∑𝐾
𝑧=1 𝑃(𝑤|𝑧, 𝜙 ) 𝑃(𝑧|𝜃 , 𝑑)(1)

and output is the word vector for each unique word.

Here, 𝜃̂ and 𝜙̂ are the posterior estimates of 𝜃 and 𝜙,

Word2vec model embeds all the words into a low dimen-

respectively. LDA does not perform very well on long tailwords, so there are a number of variants over the basic
LDA model. One of the variants is to combine LDA andMaximum

Likelihood

Estimation.

Theauthorsof[8,9,15,17,18] utilize linear combination of word -levelprobability from LDA, document -level probability andcollection-level probability to smooth the results. The
document-level probability and collection-level probability are obvious parts which can be observed by simple
term occurrences. The LDA model can extract the word
probability from latent topics which can be regarded aslatent part. However, the existing methods adopted a li near combination of obvious part and latent part. But in theassumption of LDA, the word probability of the document
is based on the corpus while the document -level probability is based on the current document, so linear combin ations may not be the best choices, because it is ad hoc tocurrent corpus. To optimize the combination method, we
propose the following nonlinear combination of the obv i-sion space, which all the word is one point at this space
and the distance between two words can present its similarity. Document2vec model can present the document tovector, but for new document it needs retrain the model .
Our test data is different from training data, so it does not
fit for our algorithm. We select the word2vec model as ourvector arithmetic.
Pointwise Mutual Information (PMI) is a measure if
association used in information theory and statistics. In
NLP task, PMI has been often used for finding colloc ations and association between words. It measures how
likely two words are to co -occur. Our object is to computethe similarity between two segments but not words.
Mihalcea propose a method to compute the similarity
between two sentences based on the normalized PMI of allword pairs in these two sentences. [10] The scoring function is as follow:
𝑠𝑖𝑚(𝑆1 , 𝑆2 ) =∑𝑤∈*𝑆2 +(𝑚𝑎𝑥𝑆𝑖𝑚(𝑤,𝑆1 )∗𝑖 𝑓(𝑤))1 ∑𝑤∈*𝑆1 +(𝑚𝑎𝑥𝑆𝑖𝑚(𝑤,𝑆2 )∗𝑖 𝑓(𝑤))(+)∑𝑤∈*𝑆1 + 𝑖 𝑓(𝑤)∑𝑤∈*𝑆2 + 𝑖 𝑓(𝑤)2(3)Where maxSim(w,S2 (1) ) is the maximum lexical similarity between the word w in segment S 1(2) and all the words

distributions. We introduce the following features fromthree aspects.
Range of similarity distribution
Similarities between pairs of segments are measured byin segment S 2(1) calculated by normalized PMI. idf(w) issegment vectors based on either TIFIDF or LDA.

the inverse document frequency of the word w calculated

article consists of multiple segments, so we obtain a ve c-

from the corpus. This similarity score has a value between

tor of segment similarities for one article. We note that

0 and 1, with a score of 1 indicating identical segments,

similarity

and a score of 0 indicating no semantic overlap between

Therefore, we characterize these simi larity vectors by

the two segments. By this method, we do not present

descriptive statistics of similarity distributions of thesegments as vectors but directly calcula te segment pairfollowing:

similarities.

and maximum, minimum and mean of the similarity values

3.3 Obtaining Similarity Distributionsof the segments in the article.

After representing each segment as a vector, we use c o-valuesaresodiversebetweenOne

articles.

The number of the segments in the article,

Dispersion of similarity distribution

sine similarity to measure the similarity between two

Based on our assumption, if a link points to a segment

segments. We need to evaluate the similarity distribution

there will be at least one segment in the link target article

between segment SA in articles A and each segment inwhich is highly similar to the link source. In an ideal

article B. So for SA in article A, and the link from SA thatsituation, if the link points to an article title, not a specificpoint to article B, we compute the similarity between SA

segment, then the similarities of all the segments in theand each segment in B.

article toward the link source segment are close between

3.4 Feature Extraction

each other, so the dispersion of these similarity values is

A Wikipedia interlink points to an article title or asmall.

Thus dispersion of similarities within one article

specifies, a link

is an important clue for determining whether the linkshould point to a segment title when the link source andshould be on the article level or segment level. Thus wethe link target segment are remarkably similar and d e-

introduce the following features:

scribe more details. Simply comparing the similarity b e-

deviation, and coefficient of variations.

tween segment SA in articles A and segment SB in article

are derived from statistical properties of the distribution

B, and the similarity between SA and article B is not aof segment similarities in one article.

good idea, because the segment SB of the link target is a

Outliers

segment title. As Wikipedia’s guideline

1variance, standard
These features

part of article B, hence they are related, and it will greatly

According to our assumption, if a link points to a se g-affect the results. To solve this problem, we adopt thement title, then it is more likely that there exists an outlier

following assumption.

segment, having an outstandingly larger similarity than

Assumption: If segment SA has a link to segment SB,other segments.

We adopt the conventional concept ofthen SB should be the most related segment with SA in

outliers such that, if the difference between one segment

article B, and the other segments in B are just slightly

and mean is more than the double of the standard deviation,

related with SA. In other words, if we rank the similarities

then we believe the segment is an outlier. If the difference

between SA and each segment in article B, then the simi-

is more than triple of the standard deviation, the segment

larity between SA and SB should be a prominent outlier. If

is a large outlier. In this paper, we use the number of large

segment SA is linked to article B, then all the segments inoutliers in one article as a feature.

B should be slightly related with SA, but there is no ob-

3.5 Filtering

vious outlier.

In Wikipedia the number of link point to segment andBased on this assumption, we construct our feature set

point to article title is imbalance. Our previous method

to incorporate statistical features on segment similarity

used under-sampling method to balance the training data.
In this paper, we first train a logistic regression model to1

https://en.wikipedia.org/wiki/Help:Link#Section_li
nking_.28anchors.29

predict the probability of the link should be point to se gment. Then we filter the test data by filtering the sample
with low probability, and using RF model to predict thelast samples.

3.6 Prediction

as our reference data. The ratio between the segment links

We train a classifier based on the above features to predict

and article links is nearly 1:100.

whether a target link points to article title or a specificobserved in most of articles, because the average number

segment. The features we introduced are statisticalof links in one article is 41, so most of articles just havemeasures on similarity distributions, which do not havearticle-level links. We divide the dataset into small subsets,

obvious linear relationships. So we utilize the nonlinear

to control the ratio between positive and negative samples,classifier model random forest [ 6] as our classifier.

where positives are segment-level links.

In most machine learning tasks, ensemble model ismuch better than unique model. We propose a stacking
model to replace unique random forest, only using random
forest as the top classifier in the stack top.

This ratio is rarely

We use therandom forest classifier for our prediction.

4.2 Experiment preparement
In the step of presenting segment, in word2vec model,
we use the whole wikipedia as the training data to train themodel and we set the dimension as 50. And we also try themodel result trained by others, Because one segment usually contains several words, so we use the sum of all word
vectors of words in segments to present the segment ve ctor.
From the dataset, we observe that the ratio between
positive and negative samples is imbalanced.

We need acareful setup for training, because the classifier tries toadjust the parameters to put all samples into the correct

F i g . 3 . Stacking model

classes. If negative samples are overwhelming majority,
As Fig3 shows, we have 3 features set, we stack these 3

the positive samples could be regard ed as invalid values

feature set and regard the lower layer result as the higher

by the classifier. There are two approaches to tackle thislayer ’s new feature.

problem. The first one is randomly sampling negatives toWhen the target link is predicted as pointing to a se g-balance positives and negatives, and then train using thisment, we need to determine which segment should besampled set. But its disadvantages are obvious. This sa m-

pointing to. In this step, instead of simply selecting thepling process will lose a large number of effective data.

most similar segment as the result, we also require that the

The lost negative samples will cause learning errors, d e-

selected segment must be an outlier in the similarity di s-grading precision.
Another solution is resampling positives until positives

tribution.

and negatives are balanced.

4. Experiment Evaluation
4.1 Dataset
Featured articles are considered to be high -quality articles in Wikipedia, and they are well organized and mai ntained. We adopt links from featured articles as our golden
standard, assuming that they are appropriately given, sothat these links point to segments when there are specifically relevant segments in the target articles.

But it can cause over fitting

easily, and it does not help the classifier to learn positive

We ra n-domly selected 1000 featured articles as our dataset.
Table 1. Dataset infomation

samples, because resampling does not increase new pos itive samples, instead just balancing the dataset by repea ting addition of small positive samples.
In this experiment, we use sampling negatives to balance the dataset. We randomly sample negatives several
times, and train the classifier.
We do experiments on unique random forest as theclassifier and stacking model as the classifier. We set thedefault parameters 50 trees in the forest and the deep is 5 .
For the filter we utilize logistic regression which penalty

DatasetCount
Articles1000
Segments
7478
Links pointing to segments
1689
Links pointing to article titles
153918There are totally 1689links pointing to segments and
153918links pointing to article titles. We use these links

is L2 to avoid over fitting. We set filter threshold as 0.5.
On the prediction results, we calculate average prec ision, recall, and F1-score.
4.3 Feature importance
Before we test in the dataset, first we have to measure
whether our features are effective to distinguish the pos i-tive class. We use correlation analysis methods to test

preceding work, we choose our baseline based on a simple

relationship between each feature and the reference data.

idea that if words in a link source occur in an article title,

Mann-Whitney is a nonparametric test of the null h ypoth-then the link should point to the article title. Otherwise, if

esis that two samples come from the same population

words in a link source occur in a segment, then the link

against an alternative hypothesis, such that a particularshould point to the segment. If words occur in multiple

population tends to have larger values than the other. The

segments, then the most frequent segment is considered as

test results are shown in Table 2.

the target of the link.

From this test we can find max similarity, mean simi-

4.5 Classification result

larity, variance, standard deviation, small similarity ou tlier are related to the reference classification. But in our

Our first step is to determine whether a link should

assumption, the large similarity outlier should be related.

point to an article title or segment. To test our method in

For further analysis, we performed Two -Sample Kolmo-

different ratios of positives and negatives, we controlle d

gorov-Smirnov test. This is a nonparametric test for

the ratio from 1:1 to 1:100. In each dataset, we use halfequality of continuous, one-dimensional probability dis-data to train and the other half data to test and all thetributions that can be used to compare a sample with asamples are random select from th e dataset. Results arereference probability distribution (one -sample K–S test),

shown in Tables 4 to 7.

or to compare two samples (two -sample K–S test).

The
Table 4. Pos:Neg=1:1

results are shown in Table 3.
Table 2. Mann-Whitney test on features
FeatureElement countMax similarity
Min similarity
Variance
Mean similarity
Coefficient of variation
Standard deviation
Small similarity outlier
Large similarity outlier

Significance
0.454
0.018
0.423
0.023
0.002
0.611
0.023
0.010
0.079

decision
retainrejectretainrejectrejectretainrejectrejectrejectTable 3. Kolmoogorov-Smirnov test on features
FeatureSignificance
Decision
Element count
0.227
retainMax similarity
0.000
rejectMin similarity
0.636
retainVariance
0.000
rejectMean similarity
0.001
rejectCoefficient of variation
0.714
retainStandard deviation
0.000
rejectSmall similarity outlier
0.821
retainLarge similarity outlier
0.612
retainWe can see in Table 3 that the small similarity outlier is
not very effective, but from these two tests, we can see
variance and standard deviation are strongly effective.
So the test results support our assumption. These i mportance results indicate that our features are effective andthe classifier is expected to distinguish segment links via
these features.
4.4 Baseline
Previous researches are all focusing on linkability toarticle titles, not targeted to segments. Since there is no

Pos:Neg=1:1
LDA feature
W2V feature
PMI feature
LDA + W2V feature
LDA + PMI features
W2V + PMI features
LDA+W2V+PMI features
Random result
Baseline

Precision
61.0%
65.0%
62.0%
53.8%
53.5%
63.2%
72.2%
50%
61%

Recall
82.4%
76.5%
76.3%
82.3%
88.3%
70.5%
76.5%
50%
57%

F1
70.2%
71.0%
68.4%
65.1%
66.6%
66.7%
74.3
50%
59%

T a b l e 5 . Pos:Neg=1:10
Pos:Neg=1:10
LDA features
W2V features
PMI features
LDA + W2V features
LDA + PMI features
W2V + PMI features
LDA+W2V+PMI features
Random result
Baseline

Precision
13.0%
14.3%
12.0%
14.4%
12.1%
11.7%
13.2
10%
12.8%

Recall
70.5%
76.5%
74.1%
76.5%
76.5%
82.4%
88.2%
50%
46.2%

F1
22.0%
24.1%
19.9%
24.2%
21.0%
20.4%
23.0%
16.6%
20.0%

T a b l e 6 . Pos:Neg=1:50
Pos:Neg=1:50
LDA features
W2V features
PMI features
LDA + W2V features
LDA + PMI features
W2V + PMI features
LDA+W2V+PMI features
Random result
Baseline

Precision
2.3%
2.6%
2.2%
2.5%
2.3%
2.3%
2.6%
2%
2.1%

Recall
48.9%
77.8%
66.6%
75.6%
67.6%
73.3%
57.8%
50%
45%

F1
4.5%
5.1%
4.3%
4.9%
4.4%
4.4%
5.0%
3.8%
4.0%

Table 7. Pos:Neg=1:100

W2V (Google news) features are basically same, word2vec

Pos:Neg=1:100
Precision
RecallF1
LDA features
1.2%
53.2%
2.3%
W2V features
1.2%
56.1%
2.4%
PMI features
1.0%
61.0%
1.9%
LDA + W2V features
0.9%
53.9%
1.8%
LDA +PMI features
0.9%
56.0%
1.7%
W2V + PMI features
0.9%
48.0%
1.7%
LDA+W2V+PMI features 0.9%
65.2%
1.8%
Random result
1%
50%
1.9%
Baseline
1.1%
41%
2.1%
The second step is to determine which segment is mostmodel get higher precision but lower recall . And the modeltrained by Wikipedia performs better than trained by
Google news, it maybe because our experiment data is also
from Wikipedia. But in selecting the most related segmentprocess, our method performs best.
We also compare the result of filteri ng and without filtering. The Table 11 shows that filtering performs well in
our task. It increases precision a lot, but decreases recall
less, and finally increasing the F1 score.

relevant to the link source. In our method, for a link in the

Table 12 shows when we select different filter threshold,source segment the most similar segment is sele cted,

the result will change a lot. If the filter is strict, false

based on cosine similarities between two segments. After

positive samples increase a lot and recall decrease. If thethat, we calculate whether the most similar segment is anfilter is general, the false negative samples will increase,outlier in the entire similarity distribution. If the outlier is

it will decrease the precision value. In our experiment, we

satisfied, we select the segment as our prediction result.

set the threshold as 0.5, it ge ts best result.

The result is shown in Table 8.
Table 10. Pos:Neg=1:10
Table 8. Predicting the most related segment
Average segment count=8
LDA features
W2V features
PMI features
Random result

Accuracy
21.3%
32.0%
40.0%
12.5%

4.6 Discussion
From Table 4 to Table 8, the results show that our
method is most accurate to predict links pointing tosegments. Our method performs better than the baseline.

Pos:Neg=1:10
LDA features
W2v(Google news)
W2v(Wikipedia)
Baseline

Recall
70.5%
68.2%
76.5%
46.2%

F1
22.0%
22.1%
24.1%
20.0%

T a b l e 1 1 . Pos:Neg=1:10
Pos:Neg=1:10
With filtering process
Without filtering process

Correlation analysis shows that our statistics features onsimilarity distributions are effeciti ve, to capture patterns

Precision
13.0%
13.2%
14.3%
12.8%

Precision
16.5%
14.3%

Recall
64.4%
76.5%

F1
27%
24.1%

T a b l e 1 2 . Pos:Neg=1:10

the influence of imbalance data is still strong. We can find

Pos:Neg=1:10
Precision
Recall
F1
Without filtering process
14.3%
76.5%
24.1%
Filter threshold <0.6
25.0%
11.7%
17%
Filter threshold <0.5
22.8
47.1%
31%
Filter threshold <0.4
15.5%
52.9%
24%
Filter threshold <0.3
18.0
52.9
26%
Besides set a filter, we also stack the features into athat when the ratio of postives and negatives is more thanstacking model. Usually the top layer should be the most

10:1, precision goes down significantly. We still need toaccuracy features so we set W2V features in top layer.

improve to deal with imbalanced datasets, since most of

Table 13 shows the result of stacking model. Compare thethe real world data is imbalanced.

single features the stacking model actually improve thewhen segment-level links occur.
Our method works well when the positive and negative
samples are balanced to 1:1. However, even though wealready use the sampling method to balance the dataset,

As our dataset is imbalanced, we can observe that theresultsampling process is very i mportant in our method. Table 9shows that if we ignore the sampling process, the F1 score
decreases a lot.
T a b l e 9 . Pos:Neg=1:10
Pos:Neg=1:10
Precision Recall
F1
W2V features
under sampling
14.3%
76.5%
24.1%
Without under sampling
10.5%
46%
17.1%
As Table 10 shows, the results of LDA features andT a b l e 1 3 . Pos:Neg=1:10
Pos:Neg=1:10
Only W2V features
Top=W2V features
Mid=LDA features
Bottom=PMI features
Top=W2V features
Mid=PMI features
Bottom=LDA features

Precision
14.3%
14.8%

Recall
76.5%
82.4%

F1
24.1%
25.2%

14.3%

82.3%

24.3%

language models. In: Proc. 27th International ACM

5. Conclusion and future work
In this paper, we proposed an text similarity basedalgorithm to determine whether links in Wikipedia articles

SIGIRConf. Research and Development Information
Retrieval, pp. 186–193(2004)

point to a related segment, or article title. We believe that

10. Mihalcea, R., Corley, C., & Strapparava, C. Co r-

our research is the first work of link detection on segment

pus-based and knowledge-based measures of textlevel. Our approach is combining the LDA model withsemantic similarity. In AAAI (2006, July). (Vol. 6, pp.

MLE with a nonlinear combination. But it can still be

775-780).

improved, we currently use document length to smooth the

11. Milne, D., Ian, H.W.: An Effective, Low-Cost Meas-parameterure of Semantic Relatedness Obtained from Wikipe-

optimizations can be expected. We also compare LDA

dia Links. In: Proc. AAAI Workshop on Wikipedia

based method with word2vec model and PMI basedand Artificial Intelligence: an Evolving Synergy,

method. We introduced statistical features on segment

Chicago, pp. 25–30 (2008)

obviouspart.

Further

improvements

oversimilarity distributions, to train a classifier for predicting

12. Milne, D., Ian, H.W.: Learning to link Proceeding.

whether a link points to segments or article titles. Our

In: CIKM '08 Proceedings of the 17th ACM confe r-

method performs well when the dataset is balanced. In

ence on Information and knowledge managemen t, pp.

future work, we plan to design a strong feature to improve509-518(2008)

accuracies when the dataset is much imbalanced. We also

13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregtry to utilize category hierarchies when selecting the mostCorrado, Jeffrey Dean: Distributed Representations

related segment.

of Words and Phrases and their Compositionality,
NIPS '13, Pages 3111–3119 (2013)

6. Reference1.

Adafre, S. F., & de Rijke, M. Discovering missing

14. Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey

links in Wikipedia. In Proceedings of the 3rd inter-

Dean: Efficient Estimation of Word Representations

national

in Vector Space, ICLR '13 Proceedings of Workshop

workshoponLink

discovery pp.

90-97(2005).
2.

Besnik, F., Katja, M., Avishek, A.: Automated News
Suggestions for Populating Wikipedia Entity Pages.
In: Proc. CIKM '15 Proceedings of the 24th ACM

for Ad-hoc Retrieval. In: Proc.29 t h ACM SIGIR Conf.,

International on Conference on Information and

pp. 178–185 (2006)5.

cific Segments as Link Targets in Wikipedia. In Di g-

aspect hidden markov model. In: Proceedings ofital Libraries: Knowledge, Information, and Data in

7.

Blei, D.M., Ng, A.Y., Jordan, M.J.: Latent Dirichlet
allocation. Journal of Machine Learning Research 3,

topics in wikipedia article segments. International

993–1022 (2003)

Conference on Asian Digital Libraries. Springer I n-

Bouma, G. (2009). Normalized (pointwise) mutual

ternational Publishing, pp.252-259(2014).
for language models applied to ad hoc information

Breiman L. Random forests[J]. Machine learning,

retrieval. In: Proc. 24th ACM SIGIR 2001, pp. 334 –45(1): 5-32 (2001)

34 (2001)

Knoth, P., Novotny, J., & Zdrahal, Z. (2010, August).

19. Zhang, J., & Kamps, J.: A content -based link detec-

Automatic generation of inter-passage links based ontion

semantic similarity. In Proceedings of the 23rd I n-

In International Workshop of the Initiative for the

ternational Conference on Computational Linguistics

Evaluation of XML Retrieval pp. 395-400(2009)

Lavrenko, V., Croft, W.B.: Relevance-based language
models. In: SIGIR 2001,pp. 120–127 (2001)

9.

18. Zhai, C., Lafferty, J.: A study of smoothing methods

GSCL, 31-40.

(pp. 590-598).
8.

an Open Access Society (pp. 394 -405).
17. Wang R, Wu J, Iwaihara M. Finding co -occurring

information in collocation extraction. Proceedings of
6.

16. Wang, R., & Iwaihara, M. (2016). Suggesting Sp e-Blei, D.M., Moreno, P.J.: Topic segmentation with an
SIGIR (2001)

4.

tions (2013)
15. Xing, W., Croft, W.B.: LDA-Based Document Models

Knowledge Management pp 323 -332(2015)3.

at International Conference on Learning Represent a-

Liu, X., Croft, W.B.: Cluster-based retrieval using

approach

using

the

vectorspace

model.