Automatic Term Recognition based on the Statistical Differences ofRelative Frequencies in Different Corpora
Junko Kubo, Keita Tsuji, Shigeo Sugimoto
Graduate School of Library, Information and Media Studies, University of Tsukuba
1-2, Kasuga, Tsukuba, Ibaraki, 305-8550, Japan
E-mail: kuboj35@slis.tsukuba.ac.jp, keita@slis.tsukuba.ac.jp, sugimoto@slis.tsukuba.ac.jp

Abstract
In this paper, we propose a method for automatic term recognition (ATR) which uses the statistical differences of relative frequencies
of terms in target domain corpus and elsewhere. Generally, the target terms appear more frequently in target domain corpus than inother domain corpora. Utilizing such characteristics will lead to the improvement of extraction performance. Most of the ATR methods
proposed so far only use the target domain corpus and do not take such characteristics into account. For the extraction experiment, weused the abstracts of a women's studies journal as a target domain corpus and those of academic journals of 39 domains as other domain
corpora. The women's studies terms which were used for extraction evaluation were manually identified terms in the abstracts. The
extraction performance was analyzed and we found that our method outperformed earlier methods. The previous methods were basedon C-value, FLR and methods which were also used with other domain corpora.

1.

Introduction

measures which use only the target domain corpus. By
comparing ATR measures, we showed our ATR measureoutperformed the existing measures.

Automatic Term Recognition (ATR) is useful for updating
terminology, dictionaries and various information
retrieval. While many methods have been proposed in thepast, most of them utilize only the target domain corpus(i.e., the corpus which contains the target terms). The
features of the terms are: “(a) they appear frequently in
documents of a target field, (b) they are not common
words in the target fields and (c) they appear lessfrequently in the corpora of other fields” (Uchimoto et al.,2000). Based on this idea, we propose an ATR method
which utilizes the statistical difference of relative
frequencies of the academic terms in the target corpus andin other domain corpora. We compared our ATR resultswith others and showed our ATR measure outperformed
the others.

2.

Data

Below, we explain the corpus and the target terms used forour experiment.

2.1 Corpus
We used the following two corpora, i.e., target domain
corpus from which we extract terms and other domain
corpora from which we obtain words which are not theterms used in the target domain:(1) Target corpus: We used the abstract texts of the
Women’s Studies International Forum 1 as the target
domain corpus (henceforth “WSIF corpus”). The reason
we chose women’s studies as the target domain is that thefirst author is familiar with it and can identify the target
terms (the terms which should be extracted) for the corpus.
The number of abstracts was 1,212. They were publishedbetween 1982 and 2007. The number of tokens wasapproximately 180,000.

The idea of using target domain corpus and other domain
corpora for ATR is not new (Drouin, 2003; Harter, 1975;
Nagao et al., 1976; Anselmo et al., 2001; Uchimoto et al.,2000). The differences between the preceding methods
and ours are: (1) the measures proposed in some of thepreceding studies are purely heuristic and do not have astatistical basis. Our method uses “the difference between
two proportions” in the statistics which follows thenormal distribution. In that sense, the statistical meaningof the value of our measure that represents the termhood iseasy to understand, (2) we use academic texts not thegeneral texts (e.g., newspapers) as other domain corpora.
Some earlier methods used newspaper texts as otherdomain corpora. In that case, words that frequently appear
in the academic texts but are not domain specific terms(e.g., “research” and “evaluation”) are likely to be
extracted. Our method does not have such drawbacks, (3)we compared our ATR measures which use target domain
corpus and other domain corpora with the existin g(2) Other corpora: We used the abstracts of the 39
journals 2 (e.g. “Applied Economics”, “International
Journal of Public Administration”, “Urban Studies”, etc.
For details, see the Appendix A) for the other domain
corpora (henceforth “OTHER corpora”). They were
published between 1981 and 2007. We randomly
extracted 1,212 abstracts from each journal and used them1

http://www.sciencedirect.com/science/journal/02775395
http://www.taylorandfrancisgroup.com/

2670

for the experiment. The reason we used the same numberof abstracts as WSIF is that the relative frequencies ofwords slightly depends on the text size and we should
eliminate such size-dependencies. The number of tokens
was approximately seven million in total.

propose the measure MDPα.
MDPα is defined as:
MDP(T) (if T does not occur more thanMDPα(T)=
α times in each of the OTHER corpora)
where α is 5,010, 25,(otherwise)
50 or 100.

These texts were POS tagged using Brill tagger and word
sequences whose POS patterns were as follows:
We will show an example to illustrate the calculation of
MDP. Suppose that we extract “feminist” as T from atarget corpus (women’s studies) to get W0 and the otherdomain corpora to get Wi (1≦i≦3).

{(Adjective | Noun)*Noun +} or{(Adjective)*(Noun)+(Preposition)*(Noun)+}.
There were 20,266 candidate terms in the WSIF corpus
and 630,541 candidate terms in the OTHER corpora.

The items listed below are the number of candidate terms(e.g. “W0 = 20000” and “W1 = 21000”) and the frequency
of “feminist” in the corpora (e.g. “f0 (feminist) = 100” and“f1 (feminist) = 15”).

2.2 Target Terms
We used the following “target terms” manually identifiedin WSIF corpus. The number of terms was 2,104.

3.

・W0 = 20000,
・W1 = 21000,
・W2 = 20000,
・W3 = 19000,

Methods

In this section, we first explain our measure MDP to
extract terms from the corpora. Next, we explain theexisting measures UC, C-value and FLR. Candidate terms
are extracted as terms in descending order of their scoresand the extraction performance is compared.

3.1

i = 1:

100
15

20000 21000
 8.20115  1
1 
 115 


1 


 41000  41000  20000 21000 

i = 2:

100
30

20000 20000
 6.15130  1
1 
 130 
1






 40000  40000  20000 20000 

i = 3:

1000
20000 19000
 9.76100  1
1 
 100 


1 


 39000  39000  20000 19000 

MDP and MDPα

We propose the following MDP (Minimum of the
Difference between Population Proportions) to use forATR:

f0 (feminist) = 100
f1 (feminist) = 15
f2 (feminist) = 30
f3 (feminist) = 0

MDP(T) min D
1i Nf 0 (T ) f i (T )

W0
Wi

Di 

 1
1 
 
 W0 Wi  i T 1   i T The result of the calculation of MDP is 6.15 which is theminimum value. Suppose that we extract “article” as Tfrom the corpora in the same way as in the above example.
The number of candidate terms (i.e., W0 and Wi) is thesame as above.

where f0(T) represents the frequency of candidate T in
the corpora of the target domain and fi (T) represents
those in the other domains (1≦i≦39). W0 and Wi
represent the number of candidate terms in the corporaof the target domain and other i-th domains,respectively. πi (T) is defined as follows: i (T ) ・f0 (article) = 20
・f1 (article) = 30
・f2 (article) = 75
・f3 (article) = 15

f 0 (T )  f i (T )W0  Wii = 1:

2030

20000 21000
 1.2450  11 
 50 


1 


 41000  41000  20000 21000 

i = 2:

2075

20000 20000
 5.6595  11  95 


1 


 40000  40000  20000 20000 

Di follows a normal distribution.
MDP focuses only on the difference of frequencies in thetarget corpus and in other domain corpora. However, if
the words frequently occur in the OTHER corpora, they
are not likely to be terms no matter how large thedifference was. MDP does not consider the words’
frequency in the OTHER corpora. Based on this idea, we671

i = 3:candidate T appeared in the corpus. t(T) is thefrequency of occurrence of T in longer (alreadyextracted as the above word formation) candidate
terms, and c(T) is the number of those candidate
terms.

20
15

20000 19000
 0.6935  11 
 35 


1 


 39000  39000  20000 19000 

The result of the calculation of MDP is -5.65 which is theminimum value. We can see from these examples that the
MDP value of the term “feminist” is higher than “article”.
Thus, it seems that “feminist” has a high probability of
being a women’s studies terms.

3.2

We will show an example to illustrate the calculation of
C-value. Suppose that we extract “feminist” and “feminist
theory” as T from the target corpus. The items listed
below are the candidate terms and these frequencies.
・socialist feminist theory (7)・contemporary feminist theory (6)・feminist theory book (2)・feminist theory (30)・feminist movement (20)・feminist organization (6)・feminist group (4)・socialist feminist (10)・radical feminist (15)・feminist (100)UC, C-value and FLR

We used the following C-value and FLR as therepresentative measures which use only the target domain
corpus. In addition, we chose the measure by Uchimoto etal. (2000) (henceforth “UC”) as representative one whichuses not only the target domain corpus but also the otherdomain corpora.
UC (Uchimoto et al., 2000) is defined as:UC(T )  TF 

TF  1 DF  FFi 

3

C-value (feminist)  log 2 1 200 

where TF is the number of occurrences of T in the
corpus of the target domain, and DF is the number of
documents in the corpus of the target domain whichcontains T, and FFi is the number of fields 3 whichcontain T.

100 
0
9 

C-value (feminist theory)  log 2 2 45 
15 
  403

As known from the formula and the example, asingle-word term’s C-value comes to be 0. We can see
from the example that the C-value of the term “feminist
theory” is higher than “feminist”. Therefore, it seems that“feminist theory” has a high probability of being awomen’s studies term.

We will show an example to illustrate the calculation of
UC. We used the T and these frequencies are the same in
the MDP example. We set DF of the target domain for UC
example.
・DF (feminist) = 40
・DF (article) = 10

FLR (Nakagawa et al. 2003) is defined as:
3

UC (feminist)  100 

100  1 
    9.26
40  3 

T

FLR(T )  f ' (T )( ( FL (t i )  1)( FR (t i )  1))

1
2T

i 1

3

UC (article)  20  20   1   0.63

where f ’(T) is the number of times the term candidate
T appeared in the corpus as an independent phrase orcompound (in other words, it is not included in alonger phrase or compound). ti is the i-th constituent
word of T, FL(ti) is the total number of adjoining
nouns on the left side of t, and FR(ti ) is the total
number of adjoining nouns on the right side of t.

10  4 

We can see from the example that the UC value of theterm “feminist” is higher than “article”. Therefore, it
seems that “feminist” has a higher probability of being awomen’s studies term.
C-value (Frantzi et al. 2000) is defined as:

We will show an example to illustrate the calculation of
FLR. We used the candidate terms and these frequencies
are the same in the C-value example.

t (T ) C - value(T )  log 2 T  f (T ) c(T ) FLR (feminist)  100  (38  1)  (75  1) 2  5444.26
1

where |T| is the number of constituent words of theterm candidate T, f(T) is the number of times the term

FLR (feminist theory)  30  (38  1)  (75  1)  (45  1)  (2  1)4
1

= 758.68

3

The NACSIS database was partitioned into 59 field corpora
according to the names of their academic societies.

672

We can see from the example that the FLR of the term
“feminist” is higher than “feminist theory”. Therefore, it
seems that “feminist” has a higher probability of being awomen’s studies term.

4.

Figure 1 shows that the precision of extraction based on
MDP is much higher than that based on C-value and FLR.
As we previously mentioned, MDP uses other domain
corpora while C-value and FLR use target corpus only.
From this result, we can say that using other domain
corpora is effective for ATR. Figure 2 shows that the
precision of extraction based on MDP is higher than thatbased on UC especially at the low recall. From theseresults, we showed MDP outperformed the earlier threemeasures.

Results and Discussions

We calculated the precision and recall of extracting terms
based on the four ATR measures and MDPα.
The precision is defined as (the number of extracted terms
which were listed in target terms)/(the number ofextracted words). The recall is defined as (the number ofextracted terms which were listed in target terms)/(thenumber of terms which were listed in target terms andexisted in the corpus).

4.1

Now, let us look at extraction errors. Table 2 (in Appendix
B) shows the top twenty candidate terms extracted basedon the MDP, UC, C-value and FLR. Two types of errors
were observed in the terms which were extracted based on
MDP and other measures. The first type of error is thewrongly extracted words which are common in theacademic field but are not the domain specific terms. This
type of error was frequently observed among the words
which were extracted based on C-value (“case study”, etc)
and FLR (“paper”, etc). On the other hand, these kinds oferrors were rare among the words which were extracted
based on MDP and UC. The reason is that MDP and UC
assign high scores only to candidate terms which rarely
occur in the OTHER corpora.

MDP and other existing ATR measures

The precision and recall of extracting terms based on thethree ATR measures (MDP, C-value and FLR) are shownin Figure 1. In Figure 1, we compare the existingmeasures which use only the target domain corpus with
MDP which uses not only the target domain corpus but
also the other domain corpora. In Figure 2, the precisionand recall of extracting terms are based on the two ATR
measures (MDP and UC) which use both the target
domain corpus and the other domain corpora.

The second type of error is wrongly extracted common
words such as “work” by C-value and “self” by MDP. The
main reason is that MDP focuses only on the difference of
frequencies in the target corpus and in other domain
corpora and it does not take into consideration how often
the words occur in the OTHER corpora, i.e., MDP does
not consider the words frequency in the OTHER corpora.
Based on this idea, we propose another measure, MDPα
which will be described in the next section.

4.2

MDPα

We calculated the precision and recall of extracting terms
based on the MDPα. The results are shown in Figure 3.
Figure 3 shows that the precision of extraction based on
MDP5 is the highest especially at the low recall. Table 1
shows the top 10 candidate terms extracted based on the
MDP and MDP5. Table 3 (in Appendix B) shows the top
twenty candidate terms extracted based on the MDPα

Figure 1: Precision and Recall for MDP, C-value and FLR

We can see in Table 1 that MDP5 failed to extract the coreterms in the women's studies such as “woman” and“gender”, while MDP could do that. However, note that
we do not have to extract these core terms whenterminological dictionaries for that domain already exist
and these core terms are included in them. Therefore, weshould choose between MDP and MDP5 depending on the
purpose, and whether we would like to extract both thecore terms and newly-coined terms or just the latter.

Figure 2: Precision and Recall for MDP and UC

673

References
Anselmo, P., Felisa, V. and Julio, G. (2001). Corpus-based
Terminology Extraction Applied to Information Access,In Proceedings of the Corpus Linguistics 2001,
Lancaster, pp.458--465.
Drouin, P. (2003). Term Extraction using Non-technical
Corpora as a Point of Leverage. Terminology, 9(1),pp.99--115.
Frantzi, K., Ananiadou, S. and Mima H. (2000).
Automatic Recognition of Multi-word Terms: theC-value/NC-value Method, International Journal on
Digital Libraries, 3(2), pp.115--130.
Harter, S. P. (1975). A Probabilistic Approach to
Automatic Keyword Indexing – Part I. On theDistribution of Specialty Words in a Technical
Literature, Journal of the American Society for
Information Science, 26(4), pp.197--206.
Harter, S. P. (1975). A Probabilistic Approach to
Automatic Keyword Indexing – Part II. An Algorithm
for Probabilistic Indexing, Journal of the American
Society for Information Science, 26(5), pp.280--289.
Nagao, M., Mizutani, M. and Ikeda, H. (1976). An
Automatic Method of the Extraction of Important
Words from Japanese Scientific Documents, Journal of
Information Processing Society of Japan, 17(2),pp.110--117.
Nakagawa, H., Mori, T. and Yumoto, H. (2003). Term
Extraction Based on Occurrence and Concatenation
Frequency. Journal of Natural Language Processing,10(1), pp.27--45.
Uchimoto, K., Sekine, S., Murata, M., Ozaku, H. andIsahara, H. (2000) Term Recognition Using Corpora
from Different Fields, Terminology, 6(2), pp.233--256.

Figure 3: Precision and Recall for MDPα

rank

MDP

MDP5

1

woman

feminist2

feminist

feminism

3

genderfeminist theory4

feminismstruggle5

self

motherhood

6

movement

femininity

7

articlerape

8feminist theory

patriarchy

9waymasculinity

10

lesbiandomestic violence

Table1: Top ten candidate terms extracted based on the
MDP and MDP5

5.

Conclusion

In this paper, we showed that our ATR measure whichuses MDP or MDP5 outperformed the traditional
measures based on UC, C-value and FLR. Unlike C-valueand FLR, MDP (or MDP5) needs other domain corpora in
addition to the target domain corpus. The cost ofobtaining such corpora is the drawback of MDP. However,
the academic texts are now prevailing and it is getting
easier to obtain corpora of various domains. We think that
this problem is now being ameliorated.
Further work is needed to clarify the amount of the target
corpus and the other domain corpora that is sufficient for
extracting terms.

674

Appendix A
The list of journals used as “OTHER corpora” is shown
below.


























Aerosol Science and Technology
Applied Economics
Australian Journal of Earth Sciences
Avian Pathology
British Poultry Science
Clinical Toxicology
Current Eye Research
Early Child Development and Care
Educational Gerontology
Electric Power Components and Systems
Energy Sources, Part A: Recovery, Utilization, andEnvironmental Effects
Ergonomics
Ferroelectrics
International Journal of Computer Mathematics
International Journal of Control
International Journal of Electronics
International Journal of Mathematical Education in
Science and Technology
International Journal of Neuroscience
International Journal of Polymeric Materials

















International Journal of Public Administration
International Journal of Radiation Biology
International Journal of Remote Sensing
Journal of Applied Statistics
JournalofClinicalandExperimental
Neuropsychology
Journal of Macromolecular Science, Part A
Journal of Modern Optics
Journal of Natural History
Journal of Statistical Computation and Simulation
Medical Mycology
Medical Teacher
Molecular Physics
Pathology
Petroleum Science and Technology
Phase Transitions
Radiation Effects and Defects in Solids
Systems Biology in Reproductive Medicine
The Journal of Adhesion
Urban Studies
Xenobiotica

Appendix B
rank MDP
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

UC

C-valueFLR

woman
feminist
feminist theory
woman
feministhijab
violence against woman
feminist
gendermotherhood
United States
study
feminism
masculinity
higher education
genderself
rape
feminist research
articlemovement
patriarchy
young woman
research
articleMari
human rightwork
feminist theory
BFM
black woman
paper
way
feminist perspective
group of woman
state
lesbianfeminist researchdomestic violence
experiencestruggle
peace movement
feminist perspective
group of woman
young woman
violence against woman
South Africa
movement
motherhood
feminist objectivity
world warrole of woman
violence
meaning of home
American woman
world
identity
cosmetic surgery
case study
class
femininity
migrant woman
number of woman
familyrape
study student
same time
experience of woman
body
symbolic violencerole of woman
young woman
patriarchy
gender justicedivision of labour
American woman
masculinity
virginity
Muslim woman
status of woman
Table 2: Top twenty candidate terms extracted based on the MDP, UC, C-value and FLR

675

rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

MDP5

MDP10

MDP25

MDP50

MDP100feminist

feminist

feminist

feminist

woman

feminismfeminismfeminism

genderfeminist

feminist theory

feminist theory

self

feminism

genderstruggle

lesbianfeminist theory

self

feminism

motherhood

struggle

lesbianfeminist theory

self

femininity

young woman

struggle

lesbianmovement

rape

motherhood

young woman

strugglefeminist theory

patriarchy

violencemotherhood

young woman

waymasculinity

femininity

violencemotherhood

lesbiandomestic violence

rape

identity

violence

strugglefeminist perspective

patriarchy

femininity

identity

young woman

Irish woman

masculinity

rape

femininity

motherhood

American woman

domestic violence

patriarchy

rape

violence

academyfeminist perspective

masculinity

body

identity

oppression

Irish woman

domestic violence

patriarchy

femininity

feminist research

American woman

feminist perspective

masculinity

rape

post

academyIrish woman

domestic violence

body

diary

oppression

American woman

feminist perspective

patriarchy

black woman

feminist research

academyIrish woman

masculinity

nationalism
post
oppression
American woman
domestic violence
Table 3: Top twenty candidate terms extracted based on the MDP5, MDP10, MDP25, MDP50 and MDP100

676