Classification of Research Papers Focusing on Elemental Technologies
and Their Effects
Satoshi Fukuda, Hidetsugu Nanba, Toshiyuki Takezawa

Akiko Aizawa

Graduate School of Information Sciences,
Hiroshima City University 3-4-1 Ozuka-higashi, Asaminami-ku,
Hiroshima 731-3194, Japan{fukuda, nanba, takezawa}@ls.info.hiroshima-cu.ac.jp

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyodaku,
Tokyo 101-8430 JAPANaizawa@nii.ac.jp

Abstract
We propose a method for the automatic classification of research papers in the CiNii article database in terms of the KAKEN
classification index. This index was originally devised to classify reports for the KAKEN research fund in Japan. It is organized as athree-level hierarchy: Area, Discipline, and Research Field. Traditionally, research papers have been classified using machine-learningalgorithms, using the content words in each research paper as features. In addition to these content words, we focus on elemental
technologies and their effects, as discussed in each research paper. Examining the use of elemental technology terms used in eachresearch paper and their effects is important for characterizing the research field to which a given research paper belongs. To
investigate the effectiveness of our method, we conducted an experiment using KAKEN data. From the results, we obtained average
recall scores of 0.6220, 0.7205, and 0.8530 for the Research Field, Discipline, and Area levels, respectively.

Keywords: text classification, information extraction, research paper

1. Introduction
The volume of scientific information has increased
exponentially in recent times, because of the increase in
the number of active researchers, which has made itdifficult for researchers to find relevant research papers.
For example, if a researcher were to retrieve papers using
the query "mobile phone", many papers from a variety of
research fields, such as Database Engineering,
Educational Technology, and Social Science, would beshown in the retrieved results, even though the researcher
would only need papers from one particular field. This
motivated us to investigate the automatic classification of
search results from academic repositories.
Several methods for text classification have been
proposed. A typical approach is to extract content words
from each text and then to use them as features formachine-learning algorithms. In addition to these content
words, we focus on phrases that play particular semanticroles, the elemental (underlying) technologies used ineach research paper, and their effects. Elemental
technologies and their effects are considered useful forcharacterizing research fields. For example, "Support
Vector Machines" (SVMs) and "Hidden Markov Models"(HMMs) are often used as elemental technologies inIntelligent Information fields, such as Natural Language
Processing, Speech Recognition, or Image Processing,whereas these technologies are seldom used in
Humanities or in Agriculture. Expressions of effects arealso useful for the classification of research papers. For
example, expressions such as "improvement of precision"
are often used in evaluations in the Intelligent
Information field, while "improvement of educational
effects" and "improved motivation" are often used in the
Educational Technology field. Therefore, we extractelemental technology terms and expressions of theireffects from the research papers and use them asadditional features for machine-learning-based textclassification.
The remainder of this paper is organized as follows.

Section 2 describes related work. Section 3 explains our
method for the classification of research papers. Section 4reports on the experiment, and discusses the results. We
present some conclusions in Section 5.

2. Related Work
There has been much relevant research in the field ofcross-genre information access. An example is thesubtask of research paper classification at the Patent
Mining Task conducted in NTCIR-7 (Nanba et al., 2008)
and NTCIR-8 (Nanba et al., 2010) Workshops. In thissubtask, research papers were classified using theInternational Patent Classification (IPC) system, which isa hierarchical patent-classification system usedworldwide.
Most participant groups employed the k-Nearest
Neighbor (k-NN) method, which is a comparatively easysolution when dealing with a large classification index,
because the classification is based only on extracting
similar examples, with no training process being required.
Furthermore, the k-NN method is itself a ranking, whichenables it to be applied directly to the IPC code ranking.
In our system, the number of research fields is muchsmaller than for NTCIR-7. Therefore, we could use two
methods for the basic framework; namely, the SVM andk-NN methods.
For the Patent Mining Task, Xiao et al. (2008) used thek-NN framework, in which various similarity-calculation
and ranking methods were examined. In our work, we
also examined several ranking methods. In our pilot study,
we employed the Listweak method as our ranking method,which we will describe in detail in Section 3.2.
The participant groups for NTCIR-8's subtask wereasked to extract expressions of elemental technologies
and their effects from research papers and patents. In
their work, Fukuda et al. (2012) proposed a system thatapplied a domain-adaptation method, using both research
papers and patents, and confirmed its effectiveness. We
utilize their system for extracting elemental technologies

and their effects from research papers, and use for theclassification of research papers, as described in detail in
Section 3.1.

3. Automatic Classification of Research
Papers Focusing on Elemental Technologies
and Their Effects
In this section, we describe a method for the automaticclassification of research papers in the CiNii article
database in terms of the KAKEN classification index,
focusing on elemental technologies and their effects. In
Section 3.1, we explain how to extract and use elemental
technologies and their effects. In Section 3.2, we present
an overview of our system.

3.1. Automatic Creation of Lists of Elemental
Technologies and Their Effects for each Field
3.1.1. Extraction of Elemental Technologies and Their
Effects from Title and Abstract
Elemental technologies and their effects are considered
useful for characterizing each research field, as discussed
in Section 1. Therefore, we extract elemental
technologies and their effects from research papers using
the information extraction method of Fukuda et al. (2012),
which is based on machine learning. They formulated
information extraction as a sequence-labeling problem,
after which they analyzed and solved it using an SVM.
The tag set was defined as follows.
 TECHNOLOGY includes algorithms, materials,
tools, and data used in each study or invention.
 EFFECT includes pairs of ATTRIBUTE and
VALUE tags.
 ATTRIBUTE and VALUE include effects of atechnology that can be expressed by a paircomprising an attribute and a value.
A tagged example is given in Fig. 1.
Through <TECHNOLOGY>closed-loop feedback control</TECHNOLOGY>, the system could <EFFECT><VALUE>
minimize</VALUE>the<ATTRIBUTE>power
loss</ATTRIBUTE></EFFECT>.

Fig. 1: A tagged example (translation from Japanese)
Fukuda et al. conducted an experiment using the dataset
from the NTCIR-8 Patent Mining Task. From theirexperimental results, which included partial-match results,
they obtained recall and precision scores of 0.2756 and0.5393, respectively, for the analysis of research papers.
3.1.2. Extraction of Key Phrases and Creation of
Their Lists
We now explain the procedure for extracting elemental
technologies and their effects from research papers. First,
we extract items in three categories (elemental
technology, attribute, and value) from the <TITLE>,<ABSTRACT>, and <KEYWORDS> sections of672,397 Japanese KAKEN data entries, using thetechnical trend analysis system created by Fukuda et al.
that enables comprehensive collection. Using thisinformation, we create an "Elemental Technology list",
an "Attribute list", and a "Value list".
In addition to the above categories, we extract items in
two categories (author and publication) from the<AUTHORS> and <PUBLICATIONS> sections of the283,686 KAKEN data entries annotated with a research
field code relevant to our experiments, using regularexpressions to create an "Author list" and a "Publication
list". Here, we consider that authors and academic
conferences tend to specialize in particular research fields.
Using as a key phrase the name of an author involved invarious research fields or an academic conference
involving researchers specializing in different disciplines
may not lead to the correct assignment of the research
field. We therefore examine the number of research fields
associated with each author and academic conference inthe 283,686 KAKEN data entries annotated with aresearch field code relevant to our experiments. The
Japanese author names associated with fewer than threeresearch fields are placed in "Author list 1", with theremaining Japanese author names being placed in"Author list 2". The names of academic conferences
associated with fewer than 10 research fields are placedin "Publication list 1", with the remainder being placed in"Publication list 2". Examples of key phrases in the sevenlists and their numbers are shown in Table 1. The weight
given to each list was determined via a pilot study.
We also consider that if there is no entry for the<ABSTRACT> section of a research paper, it might not
be possible to annotate a research field code because of alack of information. We therefore use a co-authorshipnetwork (Backstrom et al., 2006; Zhang et al., 2008;
Sendhilkumar et al., 2012). In this paper, we create
Japanese co-authorships from the CiNii article database
and KAKEN data, respectively. Currently, we use the coauthorships that appear at least five times in the
5,924,669 CiNii data entries and at least once in the283,686 KAKEN data entries annotated with a research
field code relevant to our experiments. This rule wasdetermined via a pilot study. This enables us to obtain
3,268,625 co-authorship pairs from the CiNii article
database and 1,094,510 pairs from the KAKEN data.
Key phrase
list
Author list 1
Author list 2
Publication
list 1
Publication
list 2Elemental
Technology
list
Attribute list
Value list

Examples of key
phrases
Omatsu Machiko,Hirai Seiji
Sarai Akinori,
Suzuki Satoru
Japan BiogeographySociety
Information Processing Society of Japan
PCR method,
Monoclonal,
Local government
Precision, Procedure,
Precipitation
Efficiency, Decrease,
Clear

Number

Weight144,10850

15,567

1

88,598

40

2,838

4

424,482

14

589,116

6

68,5203

Table 1. Examples of key phrases belonging to each list

3.2. System Configuration
The goal of our study is to classify research papers in theCiNii article database into the KAKEN classificationindex. Our system comprises two modules; namely, anIndexing Module and a Document Classification Module(see Fig. 2). We now describe both of these modules.

We use the ranking method proposed by Xiao et al.
(2008). First, our system extracts the top k documents*+ with the highest similarities (k nearest
neighbors) and calculates a score Score(c) for the
research field of the extracted documents. Here, Score(c)can be regarded as a measure of the likelihood that the
input document has label c. Next, these research fields are
sorted in terms of scores. Finally, our system assigns thehighest similarity of the research field to the input
document and outputs it. In our system, the following
ranking method (the Listweak method) is used, as chosen
via a pilot study.
Fig. 2: Schematic diagram of our system
3.2.1. Indexing Module
We create a query file for a research paper in the CiNii
article database using the key phrase lists described in
Section 3.1. Here, we introduce a method for setting theweights in each section. Changing the weight given to
words appearing in the various sections of a document
has been confirmed as effective by multiple experiments(Larkey, 1998; Fall et al., 2003).
First, we extract the noun words (plus any prefix) fromthe <ABSTRACT> section of the research paper entry.
At this time, one-letter words and pure numerical words
are deleted. Next, if the word is contained in the"Elemental Technology list", the "Attribute list", or the"Value list", the weight corresponding to that list is
assigned. However, if the word is not contained in any of
the three lists, a weight of 1 is assigned. We also extract
noun words from the <TITLE> section. If the word iscontained in the "Elemental Technology list", a weight of
17 is assigned. Otherwise, a weight of 1 is assigned.
Finally, we extract the Japanese author names from the<AUTHORS> section and the society names or the
publication names from the <PUBLICATIONS> section.
We then extract any author names that relate to the coauthorships in the KAKEN and CiNii article databases. If
an extracted author name is contained in either "Author
list 1" or "Author list 2", the weight corresponding to thatlist is assigned. If an extracted society (publication) nameis contained in either "Publication list 1" or the"Publication list 2", the weight corresponding to that list
is assigned. However, if an extracted author name orsociety (publication) name is not contained in any of thelists, we do not use it.
We create an index file from the KAKEN data entries
using the above method. However, we extract the nounwords from the <TITLE>, <ABSTRACT>, and<KEYWORDS> in the first step and do not use the coauthorship information.
3.2.2. Document Classification Module
For our basic framework, we use two classification
methods, namely k-NN and SVM.
k-NN method
 Similarity measure
In the design of our k-NN classifier, we use the SMART
measure (Salton, 1971) to calculate the similarity
between the query file for research papers in the CiNii
article database and the index file for the KAKEN data.
Ranking method

( )

∑

(

)

(

)

( )

where is a parameter in the range (0, 1). The term
can be regarded as a penalty that punishes documents oflower rank. In our system, is set to 0.95 by default.
SVM method
We use the SVM method as another approach in the
document classification module. We choose a linear
classifier for the various kernel functions, because ourmethod should achieve high speed automaticclassification to be useful as a retrieval system. We now
describe the method that annotates research fields for a
research paper using SVM.
First, we create the classification categories for the
research fields from the index file. Next, we apply eachclassifier to the query file. If a classifier outputs a positive
value, the research field that it represents is assigned tothe query file (research paper). Here, our task should beto annotate one research field for each given research
paper. However, in the above method, if all classifiers
output negative values for a query file, a candidate
research field does not exist. A simple solution to thisproblem is to rank the classifier results. We use thedistance from the hyperplane to rank the query files, andwe assign the research field that represents the highest
ranking.

4. Experiments
4.1. Experimental Methods
4.1.1. Datasets
KAKEN data
Research papers in the CiNii article database are notannotated with any research field codes. Therefore, we
made use of research project data (hereinafter referred toas "KAKEN data") from the database of Grants-in-Aid
for Scientific Research (KAKEN). In the KAKEN data,
each research project is annotated with a research fieldcode. Each project report contains a publication list, andsome research papers in the list have been linked
manually to the CiNii article database. We consideredthat such reports were also annotated with the research
field codes for these papers, and we therefore used themas training and test data in our experiment.
The KAKEN classification index comprises threehierarchical levels; namely Area, Discipline, andResearch Field, and each research project in the KAKEN
data is annotated with a Research-Field-level code.
Examples from the KAKEN classification index areshown in Table 2. The fields used have been modified
over the course of several years. We used the research
fields in Area, Discipline and Research Field that were in
use in 2011, for which the Area (first level) contains 10
fields, the Discipline (second level) contains 69 fields,
and the Research Field (third level) contains 297 fields.

Baseline methods
 BASE_k-NN: Does not use thetechnology and effect features in k-NN.
 BASE_SVM: Does not use thetechnology and effect features in SVM.

elemental
elemental

4.2. Experimental Results and Discussion
Area(first level)
Interdisciplinary Fields

Humanities

Discipline(second level)
Informatics
Science Educationand Educational
Technology
Philosophy
History

Research Field(third level)
Intelligent Informatics,
Software
Science Education,
Educational
Technology
Religious Studies,
History of Thought
Japanese History,
Asian History

Table 2. Examples from the KAKEN classification index
The KAKEN data contain 672,397 entries written in
Japanese and published during the period 1965-2011.
Each data item comprises several sections. We use
28,400 KAKEN data items for a training dataset that
contains seven sections (<ID>, <TITLE>, <AUTHORS>,<ABSTRACT>, <KEYWORDS>, <PUBLICATIONS>,
and <FIELD>). Currently, we have created 200 data
entries per research field, thereby avoiding bias at thethird level caused by differing amounts of data. We have
10 research fields at the first level, 44 research fields atthe second level, and 142 research fields at the third level.
Note that the number of research fields at the first andsecond levels has created bias.
CiNii article database
From the CiNii article database, we use 1,000 data itemswith an <ABSTRACT> section (the Abst dataset) and
1,000 data items without such a section (the Title dataset).
These data are classified manually at the third level in the
KAKEN classification index for use as test data.
Currently, we have created 20 data items for eachresearch field, giving a total of 100 research fields. Thereis therefore no bias at the third level. For the <FIELD>
section, we use the research fields covered by the training
data.
4.1.2. Evaluations
We used recall and Mean Reciprocal Rank (MRR) as
evaluation measures. We covered the research field list
for the top-3 output using our system. We also evaluatedthe performance for the research field that was annotatedin the training and test data at the first, second, and thirdlevels.
4.1.3. Comparison Methods
We conducted tests using our two methods and twobaseline methods.
Our methods
 k-NN: Determined by the score calculated bysumming the similarities, which penalizes
documents with lower rank.
 SVM: Uses a method that annotates a query filewith the research field code representing the highest
classifier result.

The experimental results are shown in Tables 3-5. For thek-NN method, we show the best performance using athreshold k (=1~50) for the Title dataset and the Abst
dataset. In these tables, the k-NN method performed bestfor each hierarchical level of both datasets. Therefore, thek-NN method is more useful than the SVM method forour task. Moreover, the k-NN method obtained higher
recall and MRR values than the baseline methods. This
result indicates that including elemental technologies andtheir effects can be useful.
We investigated the overall effectiveness of extracting
elemental technologies and their effects and using themas features for each research field, such as Engineering or
Social Science. Here, we investigated the research fields
at the first level, which deals with the most generalclassification of research fields. The top-1 recall scoresfor the k-NN and BASE_k-NN methods are shown in
Table 6. This table also shows the number of correct
answers and research papers. From Table 6, there were
improvements not only for the recall scores in theEngineering and Chemistry fields but also for the Social
Science and Humanities fields. These results show thatelemental technologies and their effects are useful for avariety of fields. We also found that the recall score forInterdisciplinary Fields, which refers to complex research
fields across two or more research areas with muchinterdisciplinary/cross-sectional research, was improved.
Interdisciplinary Fields tends to focus on the areas ofengineering and biological systems. For example, theInformation Science and Biomedical Engineering fields
belong to Interdisciplinary Fields. From Table 6, ourmethod would appear to be effective for the Engineering
field. As a result, we consider that our method will also
be effective for Interdisciplinary Fields that deal with
two or more research areas.

5. Conclusion
In this paper, we report on the construction of a method
for the automatic annotation of research papers in theCiNii article database using KAKEN research field codes.
We have focused on the terms used for elemental
technologies and their effects in academic resources,
which are used as keywords to improve the classification
performance. To investigate the effectiveness of our
method, we conducted an experiment using KAKEN data.
From the results, we obtained average recall scores of
0.6220, 0.7205, and 0.8530 at the third, second, and firstlevels, respectively, when using the k-NN version of our
method.

References
Backstrom, L., Huttenlocher, D., Kleinberg, J. and Lan, X.
(2006). Group Formation in Large Social Networks:
Membership, Growth, and Evolution. Proceedings ofthe 12th ACM SIG KDD International Conference on
Knowledge Discovery and Data Mining, pp. 44-54.

OurmethodBaseline

Our
methodBaseline

Our
methodBaseline

k-NN
SVM
BASE_k-NN
BASE_SVM

k-NN
SVM
BASE_k-NN
BASE_SVM

k-NN
SVM
BASE_k-NN
BASE_SVM

BASE_k-NN

k-NN
BASE_k-NN

RECALL
@1
@2
@3
Title
Abst
Title
Abst
Title
Abst
0.6830 0.7580 0.8240
0.8870 0.8700
0.93000.5650 0.6830 0.6740
0.7940 0.7040
0.83700.6790 0.7130 0.8190
0.8460 0.8680
0.90000.5920 0.6410 0.7050
0.7550 0.7570
0.8030
Table 4. Recall and MRR values at the second level
RECALL
@1
@2
@3
Title
Abst
Title
Abst
Title
Abst
0.5970 0.6470 0.7440
0.8060 0.7920
0.85800.5160 0.6040 0.6360
0.7090 0.6640
0.75500.5810 0.6090 0.7350
0.7370 0.7810
0.80200.5100 0.5570 0.6440
0.6520 0.6860
0.7140
Table 5. Recall and MRR values at the third level

EngineeringSocial ScienceAbst
0.83460.9500(217/260) (247/260)
0.81920.8962(213/260) (233/260)
Medicine, Dentistry, andPharmacy
Title
Abst
0.9556
0.9333(344/360) (336/360)
0.94440.9333(340/360) (336/360)

Title
Abst
0.56670.8833(34/60)(53/60)0.55000.8167(33/60)(49/60)Chemistry

Title
k-NN

RECALL
@1
@2
@3
Title
Abst
Title
Abst
Title
Abst
0.8270 0.8790 0.9260
0.9660 0.9570
0.98600.7390 0.8240 0.8200
0.9070 0.8660
0.93400.8160 0.8510 0.9250
0.9450 0.9590
0.97700.7550 0.7840 0.8630
0.8730 0.8890
0.9170
Table 3. Recall and MRR values at the first level

Title
0.8667(26/30)0.8000(24/30)Abst
0.6333(19/30)0.6000(18/30)Interdisciplinary
FieldsTitle
Abst
0.57140.6000(40/70)(42/70)0.5571
0.5429(39/70)(38/70)
New Multidisciplinary Fields
Title
Abst
0.40000.5000(8/20)(10/20)0.35000.5500(7/20)(11/20)MRR
Title
0.88580.79480.88100.8177Abst
0.92620.87450.90720.8432MRR
Title
0.76620.62950.76300.6658Abst
0.82850.75280.79650.7140MRR
Title
0.68230.58530.67250.5910Abst
0.74280.67180.68900.6251

Humanities

Agriculture

Title
Abst
0.80000.8500(16/20)(17/20)0.70000.7500(14/20)(15/20)Mathematical andPhysical Sciences
Title
Abst
0.82500.8750(66/80)(70/80)0.83750.8750(67/80)(70/80)Title
Abst
0.83750.9500(67/80)(76/80)0.86250.8750(69/80)(70/80)
Biology
Title
0.4500(9/20)
0.5000(10/20)Abst
0.4500(9/20)
0.5500(11/20)

Table 6. Top-1 recall scores for each research field at the first level
Fall, C.J., Torcsvari, A., Benzineb, K. and Karetka, G.
(2003). Automated Categorization in the International
Patent Classification. Proceedings of the ACM SIGIR
Forum, pp. 10-25.
Fukuda, S., Nanba, H. and Takezawa, T. (2012).
Extraction and Visualization of Technical Trend
Information from Research Papers and Patents.
Proceedings of the 1st International Workshop on
Mining Scientific Publications, collocated with JCDL
2012.
Larkey, L.S. (1998). Some Issues in the AutomaticClassification of U.S. Patents. Working Notes for the
AAAI-98 Workshop on Learning for Text
Categorization, pp. 87-90.
Nanba, H., Fujii, A., Iwayama, M. and Hashimoto, T.
(2008). Overview of the Patent Mining Task at the
NTCIR-7 Workshop. Proceedings of the 7th NTCIR
Workshop Meeting, pp. 325-332.
Nanba, H., Fujii, A., Iwayama, M. and Hashimoto, T.
(2010). Overview of the Patent Mining Task at the

NTCIR-8 Workshop. Proceedings of the 8th NTCIR
Workshop Meeting, pp. 293-302.
Salton, G. (1971). The SMART Retrieval System Experiments in Automatic Document Processing.
Prentice-Hall, Inc., Upper Saddle River, NJ.
Sendhilkumar, S., Mahalakshmi, G.S. and Dilip, S.S.
(2012). Enhancement of Co-authorship Networks with
Content-Similarity Information. Proceedings of theInternational Conference on Advances in Computing,Communications and Informatics, pp. 1225-1228.
Xiao, T., Cao, F., Li, T., Song, G., Zhou K., Zhu, J. andWang, H. (2008). KNN and Re-ranking Models forEnglish Patent Mining at NTCIR-7. Proceedings of the
7th NTCIR Workshop Meeting, pp. 333-340.
Zhang, X., Hu, X. and Zhou, X. (2008). A Comparative
Evaluation of Different Link Types on Enhancing
Document Clustering. Proceedings of the 31st Annual
International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, pp. 555-562.