1
Abacus Manipulation Understanding by
Behavior Sensing Utilizing Document
Camera as a Sensor

Yuki MATSUDA 1
Nara Institute of Science and Technology
Nara, Japan

Abstract
The abacus (also known as Soroban) is a numerical calculation tool that is traditionally used in East Asian countries. With the advancement of information
technologies, the abacus is no longer used as a standard calculation tool. However, abacus learning is garnering global attention due to the secondary skills
it can foster, e.g., mental arithmetic ability. Numerical calculation using an
abacus requires learning numerical expressions using the beads of the abacus
and manipulating beads in multiple ways and in different orders. Due to thiscomplexity, a long period of repeated learning is usually required to acquire
the skill of using the abacus. However, the teaching method of the abacus
mainly relied on lecturers’ observation through finding errors and poor beadmanipulations and pointing them out, and there is no other way but to rely onhuman labor at this moment. In this study, we aim to realize an ICT-basedlearning support system for arithmetic with a common abacus. This paper
proposes a method of estimating input values on an abacus based on image
recognition captured by a document camera. Through the evaluation experiments, we have confirmed that the proposed method showed an accuracy of
95.0% in the estimation of 7-digit number input on an abacus. Additionally,
this paper will provide discussions to realize the proposed method with othercameras such as wearable camera devices, and to design the coaching system
of abacus learning.
1 yukimat@is.naist.jp

1

2

1

Introduction

A numerical calculation tool, the abacus (Soroban), is still used today as amethod of acquiring calculation skills. It can assist in various calculations,
including the four arithmetic operations, by expressing numbers with the positions of five beads which can be moved vertically. The combination of thephysical manipulation of beads and the visual representation of numbers on
an abacus enables calculations in a different way from general calculations.
Although the number of situations in which one directly uses an abacus as a
calculation tool has been decreasing in recent years, the abilities and effects
that can be acquired through the study of the abacus have been garnering
attention [1, 2, 7, 8, 11, 13, 14]. However, calculation with an abacus requires
understanding the numerical representation using beads and manipulatingthem in multiple ways and orders, necessitating long-term repetitive learning
for mastery.
In general school education and abacus school, instruction often be provided based on the correct/incorrect determination of answers to problems.
Depending on the learner’s situation, instructors encourage understanding bydemonstrating the calculation procedure, and find and instruct on mistakes orpoor bead manipulations based on the lecturers’ observations. However, thiscoaching relies primarily on the manual efforts of lecturers. It can be considered there are regularities and patterns in calculation mistakes and poor beadmanipulations (points where learners struggle), however, the knowledge is accumulated as tacit knowledge within the instructors, making it unavailable
for reuse by others.
Based on these backgrounds, this study aims to realize the ICT-basedlearning support system using a common abacus. In this paper, we propose
an abacus input value estimation method based on image recognition captured
by a document camera to understand the learner’s “abacus manipulation behavior” required for providing learning support. As a result of the evaluation
experiments, we confirmed that it is possible to estimate the inputted 7-digit
numbers on the abacus with an accuracy of 95.0%.
The rest of this paper is organized as follows. In Section 2, we summarize
the preliminary knowledge of the abacus and describe the motivation of thisstudy. Section 3 provides a literature review of existing work and services.
Then, we will propose a system for estimating the input value of the abacusin Section 4. Finally, Section 5 provides the evaluation of the proposed methodand discussion, and we conclude this paper with Section 6.

3

   





FIGURE 1: Components of the abacus.

2

Preliminary

The abacus is an assistive tool for various calculations such as the four arithmetic operations by expressing numbers with the positions of five beads which
can be moved vertically. In this section, we will summarize the preliminary
knowledge of the abacus and describe the motivation for this study.

2.1

Numerical Representation Using AbacusFirst, the components of the abacus are shown in Figure 1. A “digit” corresponds to one digit in base 10, and the placement of five beads can represent
numbers from “0∼9.” Each “digit” is divided into an upper and lower section
by a “beam,” and the single bead in the upper section (five-bead) represents
the base-10 number “5,” while the four beads in the lower section (one-beads)
represent base-10 numbers “1” each. A specific example of numerical representation is shown in Figure 2. There is a space half the height of the beads inboth the upper and lower sections, and the state of each bead is represented bymoving the beads up and down. In the upper section, the bead in the raised
position represents OFF (0), while the lowered position represents ON (5);conversely, in the lower section, the raised position represents ON (1), andthe lowered position represents OFF (0). The sum of the beads in the same
digit (the total of the numerical values represented by the beads in the ON
state) represents the value of that digit. The “position marker” in Figure 1 isindicated on the beam every three digits and is used as a marker to determine
the ones place. The digits to the right of the determined position marker areused to represent decimals.

4



0 1 2 3 4 5 6 7 8 9
FIGURE 2: Numerical representation on the abacus.

2.2

Calculation Methods with the Abacus

Calculations using the abacus are characterized by the extensive use of complementary combinations of 5 and 10. Specifically, there are three types ofmanipulations. In the following, we explain the procedure for each manipulation with addition as an example, but subtraction can also be represented ina similar manner by performing the inverse manipulation of it.
1. Calculations involving either one-beads or five-beadThis is the simplest manipulation, which can be completed by moving either one-beads or five-bead. The following are examples ofaddition. First, for the calculation of 1 + 2, starting from the statewith only one one-bead raised (1), a manipulation to raise two more
one-beads (+2) can be performed, resulting in the state representing 3. For the calculation of 3 + 5, with three one-beads raised (3),the manipulation of lowering the five-bead (+5) can be performed,
resulting in the state representing 8.
2. Calculations involving one-beads and five-beadThis manipulation requires simultaneous movement of one-beads
and five-bead. Specifically, it corresponds to the manipulation
where the state changes from 0∼4 to 5∼9 (or the reverse). The
following are examples of addition. The calculation of 1 + 6 can beobtained by combining manipulation (1), adding 6 (i.e., 5 and 1)
to the state with one bead raised (1), resulting in the calculated
result of 7. On the other hand, the calculation of 2 + 4 requires
adding 4 to the state with two beads raised (2), but since there
are only four one-beads per digit, this calculation cannot be represented by the combination of only one-beads. Therefore, the concept of the complement of 5 is introduced. Since 4 can be expressedas 5 - 1, performing the manipulations of “subtracting 1 (lowering
one one-bead)” and “adding 5 (lowering the five-bead)” simultaneously from the current input value of 2 results in the calculated
result of 2 + 4, which is 6.

5
3. Calculations involving two digits (carrying and borrowing)This manipulation requires the movement of beads over two digits. Specifically, it corresponds to the manipulation where the statechanges from 0∼9, which can be represented in one digit, to a number of 10 or greater (or the reverse), i.e., carrying or borrowing. The
following are examples of addition. In the calculation of 3 + 8, 8 will
be added to the state with three beads raised (3), however since itcannot be represented within a single digit, the concept of the complement of 10 is introduced. Since 8 can be expressed as 10 - 2,
performing the manipulations of “subtracting 2” and “adding 10”
simultaneously from the current input value of 3 results in the calculated result of 3 + 8, which is 11. In addition, there are cases
where the complement of 5 must also be considered. For example,
in the calculation of 6 + 8, although 8 requires considering the complement of 10 as mentioned above, and subtracting 2 from the one’s
place, subtracting 2 from 6 requires considering the complement of
5 (5 - 3). Thus, to obtain the calculated result of 6 + 8, which is14, it is necessary to perform the three manipulations of “adding3”, “subtracting 5”, and “adding 10” simultaneously.

2.3

Challenges to Realize Abacus Learning Support System

Calculations using an abacus, as mentioned earlier, require constant consideration of the complements of 5 and 10 while performing manipulations, which
is significantly different from general calculations on paper. It supposes thatin order for beginners to acquire abacus calculation skills, repetitive and continuous abacus learning is necessary. However, the common approach of abacus coaching relies on the observation of learners, especially discovering andteaching mistakes and poor bead manipulations, which consume much effort
from lecturers. Therefore, providing effective learning and coaching support
based on the sensing of learners’ abacus manipulation behaviors is considerednecessary to solve this problem.

3

Related Work

Kitagawa et al.2 proposed an abacus learning support system consisting ofa board estimation system using a camera and a projection mapping systemthat overlays images on the beads to convey manipulation methods. To estimate the board, they used a method that captures the abacus with an RGB
2 This paper is published only in Japanese: http://www.interaction-ipsj.org/procee
dings/2022/data/pdf/6D04.pdf

6camera from the backside of a transparent table and detects objects using thebrightness difference caused by the shape of the abacus beads. To emphasize
the brightness difference, an LED light needs to be installed on the undersideof the table, similar to the camera. This method relies on vision-based object
detection, all beads of one digit should be detected precisely to perform theestimation of the input number on the abacus.
Arakawa et al. [3] proposed a Learning Management System (LMS) for
abacus education, managing learning software such as flash mental arithmetic,
reading aloud arithmetic, and quick-view arithmetic on the LMS, and combining it with individual grades and learning progress to enable learning anywherewith a PC equipped with the software.
There are also approaches to support abacus learning by reproducing theabacus on the screens of smartphones and tablet devices. Saito et al. [10] proposed an electronic abacus function as one of the plugins for a learning support system on smartphones. By reproducing the abacus on the screen, basic
abacus manipulations can be performed, and the process of calculation using
the electronic abacus can also be displayed as a formula. Baharudin et al. [4]proposed an interactive abacus learning application, which was implemented
as PC software, for beginners. Digika offers a service called SoroTouch [5],which provides mental arithmetic learning instruction based on the abacus UI
and manipulation methods. The system reproduces an abacus-like interface
on tablet devices and adopts a calculation method that operates buttons corresponding to beads with both hands. Beside, Tokuda et al. [12] propose amethod for estimating abacus learners’ performance by utilizing matrix factorization on the student-generated learning data with the Sorotouch app.

4

Proposed System - AbaCaaS

We propose a novel system, named AbaCaaS (Abacus Manipulation Sensing Using Camera as a Sensor), that realizes abacus manipulation behavior
sensing using a document camera, targeting general abacus learning using acommercially available abacus. In this paper, we focus on the particularly
important aspect of abacus manipulation behavior sensing, “what numbers
are being input on the abacus board?,” and propose an abacus input value
estimation method using the AbaCaaS system.

4.1

System Configuration

The system configuration is shown in Figure 3. The AbaCaaS system consistsof a commercially available abacus with pasted AR markers3 , a document
3 Placements of AR markers on the abacus will be explained in Section 4.2.1 and Figure 5.

7    

     

FIGURE 3: System configuration of AbaCaaS.
camera that captures an overhead view of the abacus board, and a mini PC
that performs image processing and abacus input value estimation. The document camera is assumed to be placed at an angle parallel to the desk tocapture the image directly below, however, the position of the abacus can be
freely set as long as it fits within the camera’s field of view (no need to fix theabacus position).
In this paper, we employed the following devices — the document camera:
Okio Labs OKIOCAM S2 Pro (resolution: 1920 × 1080 px)4 ; the abacus: Unshudo SORO-MATIC USM-103 (23-rod, boxwood beads)5 ; and the mini PC:
Intel NUC8i5BEH (CPU: Intel Core i5-8259U, RAM: 16 GB, GPU: Intel Iris
Plus Graphics 655, OS: Ubuntu 22.04.1 LTS)6 . The system is developed using
Python (version 3.10.6), and TensorFlow7 is used for building and executing
the recognition model8 .
4 https://www.okiolabs.com/product/okiocam-s2-pro/
5 https://www.unshudo.co.jp/global/product/124/
6 https://www.intel.co.jp/content/www/jp/ja/products/sku/126148/intel-nuc-kit-nuc8i5beh/specifications.html
7 https://www.tensorflow.org/
8 The major libraries and these versions for developing the proposed system are following:
numpy==1.23.5,opencv-python=4.6.0.66,opencv-contrib-python=4.6.0.66,Pillow==9.3.0,keras==2.11.0,tensorboard==2.11.0,tensorboard-data-server==0.6.1,tensorb
oard-plugin-wit==1.8.1,tensorflow==2.11.0,tensorflow-estimator==2.11.0,tensorf
low-io-gcs-filesystem==0.29.0.

8(2) obtaining one-digit images(1) Abacus position identification

perspective transform& image cropping

detecting AR markers(3) bead image recognition & input value estimation
bead image classifier(CNN model)

Ubead imageor?

sequence of recognition result

U R

U U U R L

upper section

lower section

dictionaryL

templatematching

orR
extract 7 bead images
from one-digit image

estimation result =3

repeat this procedure
for all digits

FIGURE 4: Workflow for estimating input values on the abacus.

4.2

Abacus Manipulation Behavior Sensing Method

Here, we describe a method for detecting the abacus area and estimating the
input values on the abacus board using video data obtained by the document
camera. The sensing procedure is shown in Figure 4 and below.
(1) Abacus position identification using a document camera and AR
markers(2) Obtaining 1-digit images by perspective transformation and image
cropping(3) Bead image identification and input value estimation using templatematching
In the following, we describe each step in detail.

9
id=0

1

2

3

4

5

6

7

8

9

10

11

12

13

FIGURE 5: AR markers and corresponding IDs to be pasted on the abacus.
4.2.1

Abacus Position Identification using a Document Camera
and AR Markers

Normally, the abacus position is moved on the table during calculation, hence
the abacus area must be detected from the video frames obtained by the
document camera. However, to avoid interference with abacus manipulation,
modifications/customization to the abacus must be minimized. Also, taking
actual learning situations into consideration, it is not desirable to require theuse of desks with special shapes or materials. Therefore, we adopt a methodof identifying the position by detecting AR markers [6] pasted to the abacus
frame from the video frames.
The appearance of the abacus with pasted AR markers is shown in Figure 5. We generated 14 distinct AR markers (dictionary size: 3 × 3, print size:
5 × 5 mm) using the ArUco [9] AR marker generation and detection module
in OpenCV. These markers were then pasted to the abacus frame, above andbelow the digits with position markers. Note that since the size and shape ofthe beads in commercially available standard abacuses are generally the same,the proposed method can be applied to other abacuses too. Since abacus users
mainly move the beads of the abacus, AR markers pasted on the frame will
not negatively affect natural abacus manipulations.
4.2.2

Obtaining 1-digit Images through Perspective Transformation and Image Cropping

Next, we extract the abacus region from the detected AR markers. Due to
occlusion caused by the hand using the abacus and the influence of lightreflection, not all AR markers may be detected. Therefore, we first complete
the missing AR marker positions based on the detected AR marker’s positional
relationship. The area enclosed by these AR markers is determined as theabacus region. Since the angle and position within the image differ, we apply
a perspective transformation9 to convert the image to one taken from a normal
position. By dividing and cropping the resulting image, we obtain the 1-digit
images.
9 A function cv2.warpPerspective() in OpenCV is used.

Output

Softmax

Dense

Dropout

Dense

FlattenMax Pooling 2D

Conv 2D & ReLU

Dropout

Max Pooling 2D

Conv 2D & ReLU

Input

10

U
L
R

FIGURE 6: Building blocks of bead image recognition model.
4.2.3

Bead Image Recognition and Input Value Estimation by
Template Matching

Finally, by recognizing the obtained 1-digit images, we estimate the input
values for each digit. We focus on the characteristics of the abacus structureand simplify the problem to achieve high-performance input value estimation.
As mentioned earlier, an abacus represents numbers by moving five beads(four one-beads and one five-bead) up and down for each digit, with thespace for moving beads being approximately half the height of a bead. Thismeans that, as shown in the bottom left of Figure 4, seven bead images canbe obtained from a 1-digit image. Each bead image corresponds to either the“upper half of the bead,” “lower half of the bead,” or “digit rod,” and thesequence of the seven bead images is unique for each input value.
Therefore, we construct an image recognition model that classifies eachof the seven bead images into the three classes mentioned above, and apply
template matching between the sequence of the recognition results and a dictionary of bead image sequences for each numeric value (0∼9), resulting theestimated input value for the target digit. We estimate the input value on
the abacus by repeating this procedure for all digits. In this paper, the image
recognition model is a five-layer CNN model, as shown in Figure 6, trained
only with a custom dataset of 4,845 samples (upper half: 1,925 samples, lower
half: 1,540 samples, digit rod: 1,386 samples). For training, we used TensorFlow and the Keras library10 , and constructed model is quantized to int-8
and converted to TensorFlow Lite format using TensorFlow Lite Converter
for lightweighting.

5

Evaluation

We conducted the experiment to evaluate the performance of the proposed
method of abacus input value estimation.
10 https://keras.io/

11

FIGURE 7: Example images in our dataset for evaluation.

5.1

DatasetFirst, we constructed the evaluation dataset, assuming an actual usecase. The
author inputted the 7-digit numbers which are randomly generated between
0000000∼9999999 on the abacus, and captured images using a document
camera. Example images in the evaluation dataset are shown in Figure 7.
As the requirements of images for the dataset, we set the following three
conditions: 1) at least two AR markers are included in both the upper andlower sections, 2) the entire input part of the abacus is included in the image,and 3) there is no occlusion by hand. Insofar as filling these conditions, theangle and position of the abacus were arbitrary. In this dataset, a position
marker between id = 4, 11 shown in Figure 5 is always set as the ones place.
Note that the images in this dataset are not included in the training dataset.

5.2

Evaluation Result

The evaluation results are shown in Table 1.1. Out of the 300 total samples
in the evaluation dataset, the proposed method correctly estimated the input values for 285 samples, confirming that our method can achieve highly
accurate estimation (Accuracy: 95.0%). Among the 15 samples that failed, 11
samples were due to the misidentification of AR markers, which causes failurein the perspective transformation of step (2). In the current method, we apply
a completion process only for undetected AR markers assuming all detectedmarkers are correctly identified. We are sure that adding a filtering process for
AR marker misidentification will address this issue. If the samples with failed
perspective transformation are excluded, the accuracy improves to 98.6%. The

12

TABLE 1.1
Evaluation results of 7-digit input value estimation.
Item

Value

Number of evaluation dataset samples

300

Number of samples estimated correctly

285
(a)

Number of samples including digit(s) estimated as “X”– Due to failed projection transformation– Due to failed bead image recognition
Number of samples misestimated as different input numbers

15
11
4
0

Accuracy (correct answer rate) across all samples (b)95.0 %

Accuracy (correct answer rate) when excluding (b)failed projection transformation samples

98.6 %

Average of inference timeStandard deviation of inference time

0.176 sec
0.013 sec(a) A digit that is not matched to any numbers in a dictionary of bead image sequences.
(b) Precision, Recall, and F-measure are the same as their Accuracy due to no confusion

with different input numbers.

TABLE 1.2
Samples with failed bead image recognition. The letters “U,” “L,”“R,” and “ ” represent the upper half of the bead, the lower half ofthe bead, the digit rod, and the beam, respectively.
Input value

Estimated value

05041
.52
.

1 ⇒ UR2 ⇒ UR

URL.
LL
UUR.
LL

UR
UR

URL.
RL ⇒ XUUR.
RL ⇒ X

0504X
.5X
.

0159
.7
.8
.1

9 ⇒ LR
7 ⇒ LR
8 ⇒ LR

UUU
.U
.R
U.
URLL
U.
UURL

LR
LR
LR

UUL
.L
.R ⇒ XU.
LRLL ⇒ X
U.
LURL ⇒ X

015X
.X
.X
.1

0156797. 7 ⇒ LR

UUR.
LL

LR

UUR.
RL ⇒ X015679X.

0937390. 0 ⇒ UR

RLL.
LL

UR

RLL.
RL ⇒ X

093739X.

remaining 4 samples had failed with bead image classification, however, theestimation results were “no matching value (X)” through the template matching process as shown in Table 1.2. For more detailed analysis, we derived theconfusion matrices, as shown in Figure 8, of 1-digit input number estimationand single bead image recognition with the evaluation dataset excluding failed
projection transformation samples (= 289 samples). Figure 8 (a) shows that all

13(a) one-digit input value estimation(b) single bead image recognition

FIGURE 8: Confusion matrices of 1-digit input number estimation andsingle bead image recognition.
misestimated digits were output as X, with no confusion occurring between thenumbers 0 to 9. Consequently, we obtained Precision, Recall, and F-measurevalues of 100%, 99.7%, and 99.8% respectively. Figure 8 (b) shows the recognition performance of bead images is highly accurate, with 14,692 out of 14,161
bead images (= 289 samples × 7 digits × 7 bead images) correctly recognized.
Consequently, Precision, Recall, and F-measure values all achieved 99.9%. It
suggests it is possible to eliminate misestimated input values (outputs withno matching values, X) by integrating the results of recognition for temporally
adjacent frames. However, we could not find specific patterns or conditions inthis experiment, regarding the cause of the failed bead image recognition for
8 samples.

5.3

Discussion and Limitations

In the following sections, we will engage in a comprehensive discussion andidentify the limitations of this study by considering the above results frommultifaceted perspectives.
5.3.1

Factors adversely affecting estimation performance

To strengthen the estimation method, e.g., improving robustness, we should
identify the factors adversely affecting estimation performance.
Firstly, lighting conditions might affect adversely because our proposed
method relied on an image-based approach. In the aforementioned experiments, we evaluated the system using the dataset from a typical learning
environment with a ceiling light on. In addition to this, we considered a morechallenging environment of a dark room with only a desk light. We built a14
    

FIGURE 9: Experimental setup with the dark environment (with thecondition that only built-in LED light of document camera is turned on, andany other light sources are turned off), and example images in the dataset.
dataset of 200 images under the environment shown in Figure 9, following thesame procedure. We evaluated the proposed method using this dataset andconfirmed that 145 out of 200 samples were estimated correctly (Accuracy =72.5%). As this environment was not included in the training dataset, the estimation performance became lower than the assumed environment. However,no confusion occurred with other input values (i.e., the Precision of 1-digit
input values estimation was 100%), suggesting that our method might be applicable in darker conditions. Regarding AR marker detection, we find thelighting condition does not affect adversely performance (6 out of 200 samples
are misidentified, however, there was no big difference from the case with theassumed environment).
Secondly, the position and angle of the abacus in the captured image might
affect adversely because our proposed method allows users to change them.
Regarding the position of the abacus, Figure 10 (a) shows the center position(midpoint between AR markers with id = 3, 10) of the abacus in the image
and its estimation results with evaluation dataset of Section 5.1. As the abacus deviates from the center of the captured image, the small bead images
differ from those captured from a frontal view, suggesting a higher likelihoodof misestimation. Regarding the angle of the abacus, Figure 10 (b) shows ahistogram of the abacus’s angle and its estimation results with the evaluation dataset. While estimation fails in cases where the abacus is extremely
tilted (> 20 deg), there are also several samples of failure even when the angle is small (< 10 deg). This suggests that factors other than angle may givesignificant effects. As the limitation of our proposed method, it can not estimate when the abacus is outside of the area to be captured by a document

15(a) Center position of abacus(b) Angle of abacus (histogram)

FIGURE 10: Analysis results of effects in position and angle of the abacus.
camera. To address this problem, we need to design a new interface to avoid
drastic abacus movement, e.g., dynamically changing the position to display
calculation problems.
Finally, the occlusion caused by the hand manipulating the beads, which
was not dealt with in this paper, should be considered. In our proposed
method, it’s unable to recognize the bead image where hand occlusion occurs,
making it hard to extract a bead image sequence that matches the dictionary,i.e., the estimation result will be X, in most cases. Besides, the document camera is assumed to be positioned in front of the abacus user, capturing images
close to the user’s sights. As the hand moves gradually from left to right during
bead manipulation, we believe that information for all digits can be obtained
after inputting the rightmost digit. As a future task, we plan to construct a
dataset in the wild settings, i.e., a time-sequentially continuous dataset, andwe will evaluate the performance of our proposed method.
5.3.2

Computation performance, and device alternatives

As the result of evaluation in this paper, the average computational time for
estimating input values of 19 digits (including the 7 digits of the evaluation
target) is 0.176 seconds, as shown in Table 1.1. This indicates that continuous
estimation is possible at about 5–6 fps with the performance of the machineused as mentioned in Section 4.1. Also, the computational complexity of thesystem reaches almost maximum with 19 digits, because the maximum digits
number of a common abacus is 23, and several digits in the leftmost are always
covered with a hand for holding an abacus. If we consider the input speedof a beginner using an abacus, it might be possible to be used even lower-

16
performance and more affordable devices (e.g., Raspberry Pi11 ) by reducing
estimation frequency. As future work, we will evaluate the user experienceand effectiveness of the abacus learning support system which combines our
proposed method in this paper and feedback functions.
In this paper, we use a document camera to capture images of the abacus
board, but we are also considering replacing it with a different camera device. For example, by using smart glasses, represented by Google Glass12 and
XREAL light13 , which are equipped with a camera that captures the user’ssights, there is a possibility that image capturing and applying our input estimation method can be performed only in wearable devices. Furthermore, since
many smart glasses are also equipped with information provisioning functions,
we believe they can be used as one of the abacus learning support approaches,such as overlaying images using Augmented Reality (AR) or providing guidance through audio.

6

Conclusion

Abacus learning requires understanding the numerical representation using
the beads of the abacus and mastering the manipulation techniques, necessitating long-term, repetitive learning. In this study, we aim to realize ICTbased learning support for abacus learning using a conventional abacus. In
this paper, we proposed an input value estimation method for an abacus basedon image recognition obtained by a document camera to grasp the learner’s“abacus manipulation behavior” required for providing learning support. By
detecting the abacus with AR markers attached from the image and recognizing the bead images that make up the digits, we confirmed that it is possible
to estimate the 7-digit numbers input on the abacus with an accuracy rateof 95.0% through template matching. In the future, we will extend our approach to continuous recognition under situations where occlusion occurs dueto the manipulating hand and develop an information presentation systemthat provides appropriate guidance tailored to abacus behavior.

Acknowledgment
This study was supported in part by the 2021 KDDI Foundation Research
Grant Program.
11 https://www.raspberrypi.org/
12 https://www.google.com/glass/
13 https://www.xreal.com/light/

Bibliography[1] Amaiwa, S.: The Effects of Abacus Learning on Solving Arithmetic Problems: A Comparative Study of Elementary / Junior High School Students
at Upper Level and Inexperienced Students. Journal of the Faculty of
Education, Shinshu University 96, 145–156 (1999)[2] Amaiwa, S., Hatano, G.: Effects of abacus learning on 3rd-graders’ performance in paper-and-pencil tests of calculation. Japanese PsychologicalResearch 31, 161–168 (1989)[3] Arakawa, K., Kawasaki, K., Sawada, K., Futatsuishi, Y., Kakehi, M.,
Watanabe, I.: Web Learning Support System Development in Abacus
Education. In: The journal of the Faculty of Science and Technology,
Seikei University, vol. 48, pp. 75–79 (2011)[4] Baharudin, S., Rias, R.M., Ahmad, M.F.: Designing an interactive abacus
learning application for beginners: A prototype. In: 2010 International
Conference on User Science and Engineering (i-USEr), pp. 89–92 (2010).
DOI 10.1109/IUSER.2010.5716729[5] Digika Co., Ltd.: Sorotouch. https://www.sorotouch.jp/. (accessed2023-04-27)[6] Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Marı́nJiménez, M.J.: Automatic generation and detection of highly reliablefiducial markers under occlusion. Pattern Recognition 47(6), 2280–2292(2014). DOI 10.1016/j.patcog.2014.01.005[7] Hu, Y., Geng, F., Tao, L., Hu, N., Du, F., Fu, K., Chen, F.: Enhanced
White Matter Tracts Integrity in Children With Abacus Training. Human
Brain Mapping 32(1), 10–21 (2011). DOI 10.1002/hbm.20996[8] Lu, Y., Li, M., Cui, Z., Wang, L., Hu, Y., Zhou, X.: Transfer Effectsof Abacus Training on Cognition. Current Psychology 42, 6271–6286(2023). DOI 10.1007/s12144-021-01968-1[9] OpenCV: ArUco markers and boards detection for robust camera pose
estimation. https://docs.opencv.org/4.7.0/de/d67/group__objdet
ect__aruco.html. (accessed 2023-04-10)

17

18

Bibliography[10] Saito, K., Sasaki, H., Mizuno, K.: Development of Learning Support Tools
using the Mobile Phone. In: Proceedings of Forum on Information Technology, FIT’09, vol. 8, pp. 653–654 (2009)[11] Stigler, J.W.: “mental abacus”: The effect of abacus training on chinese children’s mental calculation. Cognitive Psychology 16(2), 145–
176 (1984). DOI 10.1016/0010- 0285(84)90006-9. URL https://www.sciencedirect.com/science/article/pii/0010028584900069[12] Tokuda, K., Kaschub, D., Ota, T., Hashimoto, Y., Fujiwara, N., Sudo,
A.: Prediction of student performance in abacus-based calculation using
matrix factorization. In: Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’20
Adjunct, pp. 114–118 (2020). DOI 10.1145/3386392.3399309. URL
https://doi.org/10.1145/3386392.3399309[13] Wang, C.: A Review of the Effects of Abacus Training on Cognitive Functions and Neural Systems in Humans. Frontiers in Neuroscience 14(913),
1–12 (2020). DOI 10.3389/fnins.2020.00913[14] Wang, C., Xu, T., Geng, F., Hu, Y., Wang, Y., Liu, H., Chen, F.: Training on Abacus-Based Mental Calculation Enhances Visuospatial Working
Memory in Children. Journal of Neuroscience 39(33), 6439–6448 (2019).
DOI 10.1523/JNEUROSCI.3195-18.2019