九州・沖縄地区

教育システム情報学会 2023年度学生研究発表会

Integrating Object Recognition and WordNet for Japanese
Thesaurus Acquisition
Meihua Xue 1, Wen Gu 2, Koichi Ota 3, Shinobu Hasegawa 4
1,2,3,4 Graduate School of Advanced Science and Technology, JAIST
2,3,4 Center for Innovative Distance Education and Research, JAIST

Ishikawa, Japan
Email: {s2210099, wgu, ota, hasegawa}@jaist.ac.jp
Abstract: The primary goal of this research is to develop an innovative approach to Japanese thesaurus acquisition
by integrating object recognition technology with the comprehensive lexical database of WordNet. We developed asystem in which learners can photograph objects, and the system recognizes and displays Japanese thesaurus for theseobjects. This approach is hypothesized to improve vocabulary retention by associating words with tangible visuals.
The methodology combines technology with language learning, presenting a unique, context-rich educationalexperience. Preliminary results from learner trials indicate increased engagement and retention compared toconventional methods. This paper outlines this innovative approach, emphasizing the potential of blending technology
with traditional educational practices to enhance language learning efficiency.
Keyword: Japanese thesauruses learning, object recognition, learning system, YOLO, WordNet.

1.

Introduction

database is used to find thesauruses for the identified word,aiding the user in learning related vocabulary.

To many foreigners, learning Japanese is not only apursuit of linguistic proficiency for education but holds a
key to unlocking a fuller experience within Japan.
However, complicated languages like Japanese takes
more effect to reach appropriate fluency. This reportpresents a study delving into the development of anengaging system to aid learners expanding their Japanese
thesaurus through the utilization of visual cues fromsmartphones. We aim to investigate the potential of using
smartphone-captured images of everyday objects toenhance language learning. Next, we explore strategies for
integrating thesaurus learning into this framework,
enriching learner's vocabulary. Lastly, we compare theeffectiveness of this pictorial-based approach against thetraditional flashcard methods. Our objective is to assess if
this method can improve learning outcomes and offer anefficient way to master Japanese vocabulary, which canbenefit lives of international students in Japan (1).

2.

Figure 1. Design of PICSU Application Process Flow
3.1 Proposed Method
Our application boasts an intuitive, user-friendly
interface that provides visual references through captured
photos, displays the identified word with its common
usage in English and Japanese, and offers a list of related
thesauruses to enhance vocabulary learning, as shown inFigure 2.

Related Works

Shadiev et al.'s system enhances vocabulary acquisition
through an image search of user-taken photographs,
demonstrating the value of authentic visual contexts inlearning (2). Kikuchi et al. have created W-DIARY, anapplication that facilitates English vocabulary retention byassociating words with personal photographs, leveraging
the natural human capacity for visual memory (3).
Lastly, Safitri et al. have employed augmented reality todevelop child-friendly tools for learning Japanese
vocabulary and hiragana, with their approach showing
promise in boosting learning outcomes (4).
The originality of our study is the integration ofphotographic object recognition and WordNet to enhance
the acquisition of Japanese thesauruses.

3.

Figure 2. Flow of PICSU System
3.2 System Components
YOLO: YOLO, an acronym for "You Only Look Once(5)," is a cutting-edge object detection algorithm that ourPICSU system employs with its latest version, YOLOv7,
to recognize and classify objects in smartphone photos
and link them to relevant WordNet thesauruses efficientlyand accurately.
WordNet: Japanese WordNet (6), created in 2006 by
Japan's National Institute of Information andCommunications Technology, is a comprehensive
vocabulary database. It supports applications by allowing
the importation of a library, accessing synsets for JapaneseResearch Design & Methodology

The overview in Figure 1 depicts the PICSU Application
process flow, where a user takes a photograph of an object
using a smartphone, which is then identified by the YOLO
algorithm as a specific word. Subsequently, the WordNet

185

九州・沖縄地区

教育システム情報学会 2023年度学生研究発表会

RP) in Table 1 (orange part). Further observation revealed
that under our experimental settings, there is no noticeable
difference in the memory retention result between PICSU
method and Flashcard method. This indicates the PICSU
method, in this experiment, might be more effective forenhancing student scores.

words (like " 車 " for car), and providing users with anetwork of synonyms to enrich their vocabulary.
3.3 System Development
The platform is a web-based mobile application using
Python for backend logic and HTML, CSS, and JavaScript
for a dynamic front-end user interface, as shown in Figure
3. It utilizes the Django framework for efficient andorganized web development, with PostgreSQL as thedatabase for robust data management. It is hosted onHEROKU cloud services for reliable performance andeasy deployment.

Table 1. Friedman-Nemeyi Test Result of Correct
Answers by Participants
MeanStandard
Error
Standard
Deviation
Minimum
Maximum
CountGroup 1FFFF
FF
RFRF

Figure 3. User Interface of PICSU System

4.

5.

ExperimentFF
81.8132.6187

RF
81.5932.7271

FP
90.1972.2947

RP
89.6101.9717

11.711

12.196

10.262

8.817

60
56.79
98.33
99.42
20
20
Post-hoc TESTGroup
R
q-stat2sum
RF8
1.385641
FP
24.5
4.243524RP
29.5
5.10955FP
32.5
5.629165RP
37.5
6.495191

67.9
99.44
20

60.89
98.33
20

p-value
0.7610052
0.014467
0.0017567
0.0004121
2.732E-05

Conclusion

This study introduced an innovative language learning
approach by combining YOLO object recognition
technology with the WordNet Japanese language database,demonstrating that a picture-based learning strategy
significantly enhances vocabulary acquisition compared
to traditional flashcards. Future research will focus onrefining the system, exploring long-term retention, andincorporating elements like gamification and auditory
components to further enhance learning efficacy andengagement.

4.1 Overview
The study aims to evaluate the effectiveness of PICSU
against conventional flashcards in learning Japanese
vocabulary to 20 JAIST international students with varied
Japanese language proficiency (JLPT levels N3 or above),consisting of 19 Chinese and 1 Vietnamese students,
equally divided by gender, aged between 21 and 30 years
old, with an average age of 26.
4.2 Conditions
The process was a two-days study protocol. On the firstday, spanning two hours, participants underwent an
introductory session, a pre-test to filter out unknown
words, followed by alternating learning and recall tests
between PICSU and flashcards, with the order of thesessions counter-balanced and breaks included. The firsttest (FF for flashcards, FP for PICSU) assessed immediate
recall post-learning. Two days later, an one-hour delay test
(RF for flashcards, RP for PICSU) evaluated retention,using the same content as the first test. The study
concluded with a questionnaire, aiming to understandlong-term retention and compare the two learning
methods.
4.3 Results
Participants’ scores are computed by percentage of theircorrect answers over total correct ones. The statisticalanalysis of test scores from 20 participants wassummarized in Table 1 (blue part). Using the Friedman(7)Nemenyi Post-hoc Test , scores from four tests (Firsttest and Retention-test for both Flashcard and PICSUmethods) were compared. Significant differences wereidentified between two specific pairs: (FF, FP) and (RF,References(1) マスデン, 眞理子, Mariko, Masden-「留学生の相談から見た日本語学習の必要性」，熊本大学留学生センター
紀要 12 65-70, 2008-12-26(2) R. Shadiev, T.-T. Wu, and Y.-M. Huang, “Using image-totext recognition technology to facilitate vocabulary acquisitionin authentic contexts,” ReCALL, vol. 32, no. 2, pp. 195–212,May 2020, doi: 10.1017/S0958344020000038.
(3) 菊池開，長谷川忍："過去の写真を利用する日記スタ
イル単語学習アプリケーション W-DIARY の開発"，信学
技法，ET2015-61, pp.15-19, (2015)．
(4) Safitri R, Muslima RT, Herlina S. Mobile Augmented Reality
for Japanese Vocabulary and Hiragana Letters Learning with
Mnemonic Method. In2022 Seventh International Conference on
Informatics and Computing (ICIC) 2022 Dec 8 (pp. 1-7). IEEE.
(5) CY Wang, A Bochkovskiy, HYM Liao 「, YOLOv7: Trainablebag-of-freebies sets new state-of-the-art for real-time objectdetectors」，arXiv preprint arXiv:2207.02696, 2022
(6) H Isahara, F Bond, K Uchimoto, M Utiyama, K Kanzaki ,「 Development of the Japanese WordNet. 」， 2008 cs.brandeis.edu(7) Pohlert, T. (2014). The pairwise multiple comparison ofmean ranks package (PMCMR). R package, 27(2019), 9.

186