Academic Term Search Support System for
Beginners in Inquiry-based Learning
Yasunobu Sumikawa1 , Ryohei Ikejiri2 , and Yuhei Yamauchi2
1

Depat. of Computer Science, Takushoku University, 815-1 Tatemachi, Hachioji-shi,
Tokyo, Japanysumikaw@cs.takushoku-u.ac.jp
2
Interfaculty Initiative in Information Studies, The University of Tokyo, 7-3-1
Hongo, Bunkyo-ku, Tokyo, Japan{ikejiri,yamauchi}@iii.u-tokyo.ac.jp

Abstract. Previous studies on journal searching have proposed models to assist those with experience in publishing articles in journals. We
propose a search engine to retrieve academic terms using non-academic
terms to support beginner students while searching for academic journals. The proposed search engine supports beginner students with insufﬁcient knowledge regarding academic terms, which limits their ability toﬁnd articles that they want to read. Our search engine uses a classiﬁer
to retrieve the appropriate academic terms from sentences without using
academic terms, and a ranking algorithm that uses the Wikipedia graphstructure.
Keywords: Academic terms searching · inquiry-based learning · digital
library · search engine

1

Introduction

Knowledge discovery is considered as one of the most important skills to be acquired by not only professional academic researchers who structure knowledge
systems, but also by many others, and its importance has been widely recognized.
Research is being conducted on organizing inquiry activities as a learning activity
in schools [4, 8]; in these activities, students formulate their own hypotheses, investigate previous research, and conduct experiments to prove their hypotheses.
According to Pedaste et al. [8], inquiry activities consist of ﬁve phases: orientation, conceptualization, investigation, conclusion, and discussion. The process ofconceptualization includes two sub-phases question generation and hypothesis
generation. Therefore, a literature review of previous studies is considered forconceptualization. However, conceptualization is one of the most diﬃcult phasesof the inquiry activity process. For example, numerous students have diﬃculty
starting a research project, deﬁning a topic, or narrowing a topic [3]. Although
they have been educated to develop literacy in information retrieval for these
problems, it is assumed that students have a certain level of academic knowledge in the research ﬁeld. It has been pointed out that students need support to2

Y. Sumikawa et al.

supplement their academic knowledge [9]. A review of previous studies is an important phase in inquiry activities; it requires a high level of competence because
discovering the appropriate academic articles without academic terms used in
the articles is diﬃcult. Therefore, appropriate support for literature research in
an inquiry activity is required, especially for high school and university students
who do not have any research experience or publication history.
This study proposes a search engine to support high school and university students in conducting literature research. Because the target learners of this study
are high school and university students who have never published a paper, we assume the following: 1) We cannot use the content of previously published articles
or citation-related information, as suggested in previous studies. 2) Users have
insuﬃcient knowledge of academic terms. We assume that a literature search
consists of two stages: selecting the appropriate keywords to search for articles
and using the keywords for the literature search. Our algorithm aims to support
the selection of the appropriate keywords to search for articles. To achieve thisgoal, our algorithm uses a sentence that summarizes the information input by thelearner. The algorithm then extracts named entities and creates feature vectors
using the entire text. These two types of information are used to calculate thesimilarity between the input text and academic terms using two algorithms that
use the graph structure of Wikipedia and a text classiﬁer. Finally, the algorithm
outputs academic terms useful for article retrieval.

2

Related Works

Support for academic article retrieval has been studied in the ﬁeld of information
retrieval. There are two main approaches for this support: one that intervenes
with professional supporters, and one that improves the ability to search for
information. For the former, an educational model was developed in which librarians ask questions to clarify students’ interests, provide knowledge on the
topic, and support information retrieval [9]. For the latter, there are studies
about query expansion based on data sources, applications, and expansion techniques over the past few decades [1]. Recent research has shown that various
data sources are divided into categorized into four categories: documents used in
the retrieval process, hand-built knowledge resources, external text collections
and resources, and hybrid data sources. Furthermore, there are three types of
expansion approaches: manual, automatic or interactive [1]. As a type of automatic approach, there are numerous techniques using word and topic models
and citation contexts to recommend citation, see the details in the survey paper[2]. One of the closest studies to this study supports appropriate article retrieval
for young researchers who have published only a few articles [10]. These studies recommend articles based on previously published papers; however, diﬀerent
information is required to provide recommendations to users who have not yet
published academic articles, such as high school and university students, for the
initial stages of inquiry activities.

Academic Term Search Supporting System

3

Fig. 1. System overview

3

Ranking Algorithm

Fig. 1 shows an overview of the proposed algorithm3 . This search engine uses a
database (DB) including journals downloaded in advance from digital libraries(DL). The current version uses J-STAGE4 because it is one of the most widelyused digital libraries in Japan. We collected all 5,293,761 articles stored in the
DL and extracted all the nouns from the titles. We also extract nouns from the
input text of the user. The nouns are used 1) to create the feature vectors FVec
for user text and AT FVec for academic terms that are used in the classiﬁcation
algorithm Clf , 2) to collect named entities Entity for user text and AT Entity foracademic terms and their corresponding Wikipedia articles to measure similarity
using a Wikipedia-graph-based algorithm WG, and 3) to perform a simple text
analysis ST that counts the number of words that have the same characters. We
describe the two algorithms, Clf and WG, in the remainder of this section.
3.1

Document Classiﬁcation Based Ranking Algorithm

The Clf algorithm calculates the probability of the learner’s question text being
related to each academic term by applying a classiﬁer. To train the classiﬁer,
we ﬁrst created a labelled dataset that included combinations of questions andacademic terms. We asked 1,000 high school and university students without
research experience or publication history to propose questions as an initial stepin inquiry-based history learning. We then annotated the questions by assigning
academic terms stored in the encyclopedia of historiography [7]. Note that allannotators had a Ph.D. degree in machine learning or education. Before annotation, we ﬁltered the academic terms for which corresponding academic articles
could not be found in the DL database. Consequently, we decided to use 943
3
4

The current search engine supports only Japanese. We will support other languages such as
English.
https://www.jstage.jst.go.jp/browse/-char/ja/

4

Y. Sumikawa et al.

sentences for constructing our algorithms and 51 academic terms to present theoutput results of the algorithm.
As the current version uses texts written in Japanese, it ﬁrst applies morphological analysis. The algorithm uses JUMAN++[13] for user text and MeCab[5]for journal text. After removing the stop words, we create feature vectors. Because this algorithm classiﬁes sentences entered by high school students, it isassumed that sentences are relatively short. To create eﬀective feature vectors
for short sentence classiﬁcation, we use the following seven that are validated
in the previous studies [11, 12] that classify events written in short sentences:
TF-IDF to use words from the user and academic term texts (F1 ), LSA (F2 ),LDA (F3 ), and Doc2Vec (F4 ) to capture latent semantic structures, noun context(F5 ) to capture the semantic meanings of nouns by replacing the top k closest
words on Word2Vec space, Wikipedia title (F6 ) and Wikipedia category (F7 ) touse a knowledge collection by mapping the short sentence by explicit semantic
analysis (ESA) [6]. We use the top 5 Wikipedia articles as the results of ESA forthe last two features. Finally, we trained the classiﬁer on the feature vectors.
3.2

Wikipedia Reference Graph Based Ranking Algorithm

This algorithm analyzes the term context, indicating a term set that is often
used together to determine the rankings. If two diﬀerent academic terms are used
together with the same term or if the same term is used to explain them, the twoterms are considered to be similar. For this algorithm, we used Wikipedia because
we need the large number of academic terms. Wikipedia not only provides adetailed description of each article in the text but also provides references toother related articles.
Wikipedia article collection. To apply this algorithm, we manually collected
Wikipedia articles corresponding to the academic terms output by the algorithm
in advance. We apply ESA to collect Wikipedia articles that corresponded to
the learner texts. The ESA outputs concepts if it analyzes them to be the mostappropriate for the input sentence. Because ESA considers Wikipedia articles asconcepts, it is possible to retrieve the relevant Wikipedia articles from sentences.
Scoring based on the similarity of reference graphs. Under the assumption that articles with matching reference relations are similar, we deﬁne thesimilarity WG between the user’s sentence Lt and the academic term sentence
Wt on the Wikipedia reference graph using the following equation:
WG(Lt , Wt ) =Ref (Lt , Wt ) ∩ Refed (Lt , Wt )Ref (Lt , Wt ) ∪ Refed (Lt , Wt )(1)

Function Ref returns the number of Wikipedia articles cited in both Wikipedia
articles provided in the argument. Function Refed returns the number of Wikipedia
articles that cite both Wikipedia articles provided in the argument. In the above

Academic Term Search Supporting System5equation, the numerator calculates the similarity between the user text and textof the academic term. Because both the number of references and non-references
to the Wikipedia articles vary from article to article, the denominator normalizes
this number such that the number of references does not aﬀect the result.
3.3

Integrated Algorithm

To integrate all results of the three algorithms mentioned above, we use thefollowing equation to calculate ranking scores of academic terms.
rank(Lt , Wt ) = α ∗ Clf (Lt , Wt ) + β ∗ WG(Lt , Wt ) + γ ∗ ST (Lt , Wt )(2)α, β, and γ are hyper-parameters. We assume that the sum of the three hyperparameters is 1.0. In our search engine, we set the hyper-parameters as 0.333,
0.333, and 0.334, respectively.

4

Experimental Evaluations

4.1

Research Questions

In this study, we performed experimental evaluations according to the following
two research questions:
RQ 1 Can we discover learners’ text that can present appropriate academic
terms?
RQ 2 How accurately can we ﬁnd appropriate academic terms to text?
For the ﬁrst research question, we evaluated the classiﬁers trained on texts
as inquiry-based history-learning topics. First, we deﬁned each academic term
as a category. We also deﬁned a category None that indicates that there is no
academic term that can be assigned. We then evaluated the accuracy of theclassiﬁer in classifying each category. As 51 academic terms were used in thisevaluation, we trained classiﬁer algorithms to classify them into 52 categories,
including None.
For the second research question, we evaluated the accuracy of our algorithm
using only test data to which academic terms could be assigned.
4.2

Experimental setting

Dataset As described in Section 3.1, this study uses 943 crowdsourced texts
that summarize the topics in which high school and college students currently
have an interest to conduct inquiry learning that they actually ﬁlled out for thisexperiment.

6

Y. Sumikawa et al.

Algorithms The classiﬁers used for the evaluation of RQ1 were: random forests(RFs), SVM with linear kernel (SVM-Lin), and SVM with RBF kernel (SVMRDF).
The following algorithms were used in the evaluation of RQ2:
– Jaccard: This uses only nouns in all sentences to measure similarity scoresbased on letter agreement alone.
– Clf : This ﬁnds the similarity using only the classiﬁer described in Section
3.1.
– WG: This algorithm uses the similarity of Wikipedia’s reference graphs described in Section 3.2.
– P roposed: The algorithm uses Clf , WG, and ST , as described in Section
3.3.
This evaluation requires the use of Wikipedia, which corresponds to an academic
term. In this evaluation, 77 learner data met this condition.
Evaluation criteria As the evaluation of RQ1 can be regarded as a multi-classclassiﬁcation problem, we used three types of evaluation criteria for this problem:
precision, recall, and F1-score.
As the evaluation of RQ2 can be regarded as a problem of information retrieval, we used the mean average precision (MAP), which is widely used for theevaluation of search engines.
All evaluation results in this study were the average of the results obtained
from a 10-split cross-validation test.
4.3

Results

Q. Which feature vectors achieved the highest accuracy for RQ1?
A. The best results were obtained by combining ﬁve feature vectors:
Wikipedia title, TF-IDF, Doc2Vec, Wikipedia category, and noun context.
First, we analyzed the features that should be used for classiﬁer training.
Tab.1 shows the results of training an RF using each of the features listed in
Section3.1. The overall trend shows that the precision value is high; however,
the recall is relatively small. Next, we checked whether the combination of thesefeatures would improve the accuracy, as previous studies of short sentence classiﬁcation on events found that combining several features improve accuracy [11,12]. In this study, we combined each feature individually in the order of their
F1 scores. After combining the features, we used random forests to extract only
the top 500 elements that were important for classiﬁcation and used them totrain the classiﬁer. As shown in Tab. 1, combining them provided a better result
than using them individually. Especially, we can see that the combination of
F6 , F1 , F4 , F7 , and F5 is the best. Therefore, in the following sections, the classiﬁer is trained using only a combination of the F6 , F1 , F4 , F7 , and F5 features.

Academic Term Search Supporting System

7

Table 1. Scores in feature selection with random forests

F1
F2
F3
F4
F5
F6
F7
F6 + F1
F 6 + F1 + F 4
F6 + F1 + F4 + F7
F6 + F 1 + F4 + F7 + F5
F6 + F1 + F4 + F7 + F5 + F2
F6 + F1 + F 4 + F7 + F 5 + F2 + F3

Precision
77.2%
80.3%
70.6%
78.1%
77.9%
78.1%
78.3%
79.1%
78.8%
78.7%
80.2%
80.4%
80.1%

Recall
68.9%
63.1%
62.8%
67.9%
67.1%
68.6%
67.4%
68.4%
67.7%
67.7%
68.5%
67.5%
67.4%

F1
72.8%
70.7%
66.3%
72.6%
72.1%
73.0%
72.4%
73.4%
72.8%
72.7%
73.8%
73.3%
73.2%

Table 2. Scores in classiﬁer selection

RFs
SVM-lin.
SVM-rbf
NB

Precision
80.2%
78.2%
76.9%
53.4%

Recall
68.5%
69.9%
68.5%
68.6%

F1
73.8%
73.8%
72.4%
59.7%

Table 3. Comparison of unsupervised learning and classiﬁers

Jaccard
WG
Clf

Precision
20.4%
6.5%
80.2%

Recall
33.1%
36.7%
68.5%

F1
22.4%
10.4%
73.8%

Q. Which classiﬁcation algorithm achieved the highest accuracy for RQ1?
A. Random forests provided the best accuracy.
Tab. 2 shows the precision, recall, and F1 scores obtained using the fourclassiﬁers. This result also shows that the precision was high for all algorithms,
whereas recall was low. Because the best F1 score was obtained by RFs, we used
RFs in the following evaluations as a classiﬁer.
As our algorithm is a search engine system, Tab. 3 shows the results of thecomparison with the algorithms we planned to use in RQ2, which uses pure word
matching (Jaccard) and the Wikipedia graph structure (WG) with classiﬁcation.
As a characteristic of the results, we can conﬁrm that unsupervised learning has
low precision but relatively high recall, whereas the classiﬁer tends to have highprecision but low recall.

8

Y. Sumikawa et al.
Table 4. Comparison of the top 10 search results
MAP
26.6%
12.7%
3.5%
36.2%

WG
Jaccard
Clf
Jaccard +WG + Clf

No result in TOP10
54.7%
54.7%
75.3%
39.3%

Table 5. Distribution of ranking where correct answer data existed in the top 10 of
the proposed algorithm.
Rank
Num.

1
17

2
10

3
1

4
4

5
2

6
3

7
2

8
1

9
2

10
2

Q. Which is the best search algorithm among the 4 algorithms?
A. The proposed algorithm is the best.
Next, we evaluated the accuracy of the academic terms listed as the top 10
results for each algorithm to verify its eﬀectiveness as a retrieval system. As
mentioned above, we used 77 data points in this evaluation because we used
only those academic terms from Wikipedia that could be assigned to learners’
sentences in the test data.
Tab. 4 shows the results. Focusing on the MAP score, using only the classiﬁer obtained the worst result. However, the best result was obtained when we
combined all the algorithms. As shown in the evaluation in Tab. 3, the classiﬁer provided a high precision score, whereas the other algorithms provided highrecall scores. It is likely that combining these factors would have improved theoverall results.
The number of correct answers that were not in the top 10 is shown in thethird column of Tab. 4. This result indicates that the proposed algorithm is thebest. Our algorithm did not provide the correct answer for only 40% of the test
data, whereas the other algorithms failed to obtain the correct answers for morethan half of the test data.
4.4

Error Analysis

Q. What was the distribution regarding the location of the correct answer
among the top 10 results output by the proposed algorithm?
A. Rank 1 was the highest.
A. Most data provided the correct answer using the top two numbers.
Tab. 5 shows where the correct answer was obtained in the top 10 cases when
the proposed algorithm was applied. First, 44 (57%) data had correct answers in
the top 10. The location with the highest number of correct answers was ranked

Academic Term Search Supporting System

9

Table 6. Academic terms for which correct data were not available and their frequencies

Academic term
辺境
世界システム論
社会進化論／社会ダーウィニズム
昭和史論争
文化圏
オーラルヒストリー
社会集団
発展段階（解釈／）解釈学
社会変動
産業革命論争オリエンタリズム
文化伝播
文化財保護
明治維新論争
実証主義
日本文化論
マルクス主義（歴史学）
共同体

English name
Periphery
Theory of the world-system
Social evolutionism/social Darwinism
Showa history controversy
Kulturkreis
Oral history
Social group
Stages in development
Hermeneutics
Social change
Industrial revolution
Orientalism
Trans-cultural diﬀusion
Law for the protection of cultural properties
Meiji restoration
Positivism
Culture of Japan
Marxism
Community

Num.
4
3
3
3
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1

1. Rank 2 also had a high number of correct answers. Therefore, the majority ofcorrect answers were found in the top two cases.
Q. What were the academic terms that the proposed algorithm tends to getwrong most often?
A. “Periphery” was the most diﬃcult word to assign correctly.
Tab. 6 shows the academic terms that were not ranked in the top 10. The
reason why “periphery” and “theory of the world-system” were listed at the
top of the error list would be that these academic terms could be regarded astheories that explain various historical phenomena from a broad spatio-temporalperspective; thus, the accuracy of the search results was lower than that of terms
related to speciﬁc periods or phenomena. In contrast, “social evolutionism/social
Darwinism” and “Showa history controversy” were also listed as the top error
terms because they were both academic terms with a narrow range of applicability although the words included in these terms have a wide range (for example,“Showa” is the name of a period in Japanese history).

5

Conclusions

In this study, we proposed an academic term search engine to assist high school
and university students in conducting literature research. In particular, our algo-

10

Y. Sumikawa et al.

rithm aims to support the selection of appropriate keywords to search for articles
in the literature.
In future work, we aim to make high school students use the search engine
with our algorithm and analyze how beneﬁcial it is for inquiry-based history
learning.
Acknowledgments. This work was supported in part by MEXT Grant-in-Aids(#20H01717).

References
1. Azad, H.K., Deepak, A.: Query expansion techniques for information retrieval: A survey. Information Processing & Management 56(5), 1698–
1735(2019).
https://doi.org/https://doi.org/10.1016/j.ipm.2019.05.009,
https://www.sciencedirect.com/science/article/pii/S0306457318305466
2. Färber, M., Jatowt, A.: Citation recommendation: approaches and datasets. Int J
Digit Libr (2020)
3. Head, A., Eisenberg, M.: Truth be told: How college students evaluate and use
information in the digital age. SSRN Electronic Journal (01 2010)
4. Keselman, A.: Supporting inquiry learning by promoting normative understandingof multivariable causality. Journal of Research in Science Teaching 40(9), 898–921(2003)
5. Kudo, T., Yamamoto, K., Matsumoto, Y.: Applying conditional random ﬁelds
to japanese morphological analysis. In: Proceedings of the 2004 Conference on
Empirical Methods in Natural Language Processing , EMNLP 2004, A meetingof SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL
2004, 25-26 July 2004, Barcelona, Spain. pp. 230–237. ACL (2004)
6. M. Chang, M.W., L. Ratinov, L., Roth, D., Srikumar, V.: Importance of semantic
representation: Dataless classiﬁcation. In: AAAI (7 2008)
7. Ogata, I., Kato, T., Kabayama, K., Kawakita, M., Kishimoto, M., Kuroda, H.,
Sato, T., Minamizuka, S., Yamamoto, H.: Encyclopedia of historiography. koubundou (1994), http://ci.nii.ac.jp/ncid/BN10236869
8. Pedaste, M., Mäeots, M., Siiman, L.A., de Jong, T., van Riesen, S.A., Kamp, E.T.,
Manoli, C.C., Zacharia, Z.C., Tsourlidaki, E.: Phases of inquiry-based learning:Deﬁnitions and the inquiry cycle. Educational Research Review 14, 47–61 (2015)
9. Scharf, D., Dera, J.: Question formulation for information literacy:
Theory and practice. The Journal of Academic Librarianship 47(4),102365 (2021). https://doi.org/https://doi.org/10.1016/j.acalib.2021.102365,
https://www.sciencedirect.com/science/article/pii/S0099133321000562
10. Sugiyama, K., Kan, M.Y.: Scholarly paper recommendation via user’s recent research interests. In: Proceedings of the 10th Annual Joint Conference on Digital
Libraries. pp. 29–38. JCDL ’10, Association for Computing Machinery, New York,NY, USA (2010)
11. Sumikawa, Y., Jatowt, A.: Classifying short descriptions of past events. pp. 729–
736. ECIR’18 (2018)12. Sumikawa, Y., Ikejiri, R.: Feature selection for classifying multi-labeled past events.
Int. J. Digit. Libr. 22(1), 63–83 (2021)
13. Tolmachev, A., Kawahara, D., Kurohashi, S.: Juman++: A morphological analysis
toolkit for scriptio continua. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 54–59.
Association for Computational Linguistics, Brussels, Belgium (2018)