IEEE ICTMOD 2023 1570956145

2023 IEEE International Conference on Technology Management, Operations and Decisions (ICTMOD) | 979-8-3503-1335-2/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICTMOD59086.2023.10438119

1	
  
2	
  
3	
  
4	
  
5	
  
6	
  
7	
  
8	
  
9	
  
10	
  
11	
  
12	
  
13	
  
14	
  
15	
  
16	
  
17	
  
18	
  
19	
  
20	
  
21	
  
22	
  
23	
  
24	
  
25	
  
26	
  
27	
  
28	
  
29	
  
30	
  
31	
  
32	
  
33	
  
34	
  
35	
  
36	
  
37	
  
38	
  
39	
  
40	
  
41	
  
42	
  
43	
  
44	
  
45	
  
46	
  
47	
  
48	
  
49	
  
50	
  
51	
  
52	
  
53	
  
54	
  
55	
  
56	
  
57	
  
60	
  
61	
  
62	
  
63	
  
64	
  
65	
  

Study of Measuring Motor Function in Dementia by
Tapping Task Using IoT and Image Recognition
3rd Masahide Nakamura 1,2

1st Sinan Chen

2nd Atsuko Hayashi

Kobe University,
1-1 Rokkodai-cho, Nada,
Kobe, 657-8501, Japan
chensinan@gold.kobe-u.ac.jp

Kobe University,
7-10-2 Tomogaoka, Suma,
Kobe, 654-0142, Japana-hayashi@pearl.kobe-u.ac.jp

1

2

Kobe University,
RIKEN Center for Advanced Intelligence Project,Tokyo, 103-0027, Japanmasa-n@cs.kobe-u.ac.jp

task using methods such as magnetic sensors or touch panels[3]. However, these methods have mainly focused on assessing
only the ﬁrst and second ﬁngers, with limited attention given
to evaluating all ﬁve ﬁngers. Additionally, a challenge withthese methods is that the instruction-giving aspect and the
input aspect are independent, making it difﬁcult to conduct
evaluations and analyses within an application.
The goal of this study is to address these challenges bydeveloping a web service for measuring ﬁne motor skills using
the tapping task. The proposed service automates the executionof the tapping task through simple interactions and records andevaluates the results automatically. Speciﬁcally, concerning
hardware, we employ a USB ﬁxed camera and a generalpurpose computer. On the software front, we designed a web
interface for conducting tapping tasks. The system corresponds
to numerical and Japanese kana characters presented every two
seconds, allowing it to record data of buttons pressed on thetouch panel. Concurrently, utilizing a pre-trained model basedon images from the USB camera, the system can also capturechanges in ﬁnger recognition features.
Based on our proposed methodology, we proceed withsystem implementation. As a key technological aspect, we
coordinate a web server built with node.js and a MongoDB
database to collect and store the two-dimensional coordinatesof 21 ﬁnger key points obtained from the ﬁnger recognition
framework called MediaPipe. Furthermore, we organize theexperimental scenarios and provide examples of visualizing
feature changes through graphs. The outcomes of this researchenable the measurement of ﬁne-grained variations in ﬁnger
motor function, holding the potential for cognitive function
assessment and prevention. The remainder of this paper isorganized as follows. Section II introduces the preliminaries of
the related work and technilogies for tapping task. Section III
produces a complete description of the proposed method. The
implementation and discussion of the proposed Web service
for measuring motor function are presented in Section IV,
followed by conclusions in Section V.

Abstract—Dementia has emerged as a signiﬁcant issue incontemporary Japanese society. Assessing ﬁnger motor function
is pivotal for diagnosing and rehabilitating individuals withdementia. In this study, we design a web application to gauge
manual dexterity through tapping exercises. Our fundamental
concept combines ﬁnger identiﬁcation using a web camera andtactile panel input to quantify ﬁnger actions. In our proposed
methodology, we amalgamate real-time visuals captured by a
web camera into the hand.js module within the MediaPipe
framework, autonomously generating 21 critical ﬁnger data
points. Concurrently, we establish touch panel control buttons
on the web interface. In the context of the operating interface,participants manipulate their ﬁngers in sync with numbers orfeatures displayed every two seconds and utilize touch panel
input to log button presses. This approach facilitates an allencompassing assessment of ﬁnger motor aptitude.
Index Terms—motor function, tapping task, ﬁnger dexterity,
IoT, image recognition, Web service

I. I NTRODUCTION
In recent years, Japan has been experiencing a rapid aging
population, leading to concerns about the increasing numberof dementia patients. Dementia is a condition characterized bya decline in memory and cognitive functions, which hinders
everyday life and social interactions. The number of dementia
patients in Japan is projected to rise in the future. Against
this societal backdrop, there has been a growing interest in
healthcare for dementia. Dementia not only leads to a decline
in cognitive functions but also affects motor functions. In
particular, Alzheimer’s disease is known to be associatedwith a noticeable decline in motor function [1]. Research has
been conducted to diagnose dementia in its early stages bymeasuring changes in motor functions such as ﬁne motor skills
and walking abilities in dementia patients [2].
A method commonly employed to detect a decline in ﬁne
motor skills is the tapping task. The tapping task involves
the experimenter providing instructions, often using numbers
or other cues, and assessing whether the subject can movetheir ﬁngers appropriately in response to these instructions.
Traditional methods relied on providing instructions on paperor other physical mediums, with the experimenter visuallyassessing the results. However, this approach posed signiﬁcantburdens on both the experimenter and the subjects. Efforts
have been made to automate the evaluation of the tapping

II. P RELIMINARIES
A. Tapping Task for Measuring Motor Function
One of the methods used to measure motor function quantitatively is the tapping task. The tapping task involves tapping

Authorized licensed use limited to: Kochi University of Technology. Downloaded on March 05,2024 at 01:51:59 UTC from IEEE Xplore. Restrictions apply.

1

footage, such as from cameras, is undesirable from a privacy
perspective. Methods utilizing touch panels offer the advantageof recording digital information. However, they face the issueof being unable to record ﬁnger movements when the subject
fails to touch accurately or when a touch goes unregistered.
The second challenge is the requirement for special equipment and procedures. As mentioned earlier, the methods
used in the preceding research in Section II-B allow forrecording ﬁne-grained data on ﬁnger movements but demand
dedicated sensors. In conducting tapping tasks, subjects, andexperimenters should be able to operate without requiring
in-depth knowledge of digital devices. Moreover, the device
setup should be straightforward, such as simply turning on theterminal’s power.
The third challenge is to facilitate result viewing andassessment. Making tapping task accuracy rates and the distribution of results for each ﬁnger easily visualizable would bedesirable. It would enable both experimenters and subjects toreview the results and make them useful for future reference.
For this, proper data recording and processing are necessary.

Fig. 1. Examples of presenting number and Japanese Kana in a tapping task.

a pad or a similar surface with one’s ﬁnger in response toperiodic stimuli, often visual. Cognitive and motor functions
can be evaluated by assessing how quickly and accurately
individuals respond to the presented stimuli [4]. In this study,
we propose an overview of a tapping task conducted througha web service. Examples of presenting number and Japanese
Kana in a tapping task are illustrated in Figure 1.
Before performing the tapping task, numbers or features
are assigned to the ﬁngers of both hands. The web service
then presents random numbers or features every two seconds,
and participants move the ﬁnger corresponding to the presented number to touch a speciﬁc area on the screen. The
tapping task consists of two variations: the normal task,
where participants touch the presented number or character
immediately, and the n-back task, where participants respondto the number or feature presented one step before the current
one.

D. Advantage of IoT Device and Web Server
In this study, we contemplate conducting and evaluatingtapping tasks as a web service. There are primarily three advantages to measuring cognitive function using a web service.
The ﬁrst advantage is leveraging Internet of Things (IoT)
devices as input sensors. We use web cameras and touch
panels to recognize ﬁnger movements. This approach canbe implemented inexpensively since no special equipment or
facilities are required.
The second advantage is the convenience of operation as itruns within a web browser. We can perform measurements
and record data efﬁciently through simple interactions byconducting stimulus presentation and measurement within asingle web application.
The third advantage is managing measurement results using
a web server. Storing measurement results on a web server
allows for easy comparison of results and facilitates utilizingthese results in other services.

B. Related Work
There are several related works on devices used for measuring tapping tasks. One method explored in research involves
using devices equipped with magnetic sensors to measureﬁnger movements during tapping tasks [5]. Magnetic sensors
offer the advantage of accurately measuring ﬁnger distancedata; however, they require specialized equipment, which
presents a challenge.
Another measurement method investigated is based on touch
panels [6]. When using touch panel devices, there are several
challenges to consider. Many touch panels determine input as abinary response (touched or not), and there are limited options
for continuously assessing input strength. In addition, thesemethods do not record movements if a touch is unresponsive.
Furthermore, it is worth noting that these previous studies
have primarily focused on evaluating only the ﬁrst and second
ﬁngers rather than assessing all ﬁve ﬁngers simultaneously.

E. Collecting Data with Touch Panel
To conduct the assessment of tapping tasks within a web
application, touch panel recognition is employed. This involves
connecting a tablet-type monitor device equipped with a touch
panel to a notebook PC running a web browser, utilizing it asboth a display screen and an input device. The recognitionof touch panel inputs is implemented in the HTML using thespeciﬁcations of buttons.

C. Technical Challenges
We will enumerate the current challenges related to the goalof quantitatively evaluating tapping tasks:
The ﬁrst challenge involves the difﬁculty of appropriately
recording the measured information. Traditional methods rely
on presenting numbers written on paper, for example, as
stimuli and visual observation conﬁrm the subject’s ﬁngers’
movement. While this method does not require specialized
tools, it has drawbacks, such as the inability to review results
later and reliance on the experimenter’s subjective judgment
for measurement standards. Additionally, using raw video

F. Image Recognition Using Pre-Trained ModelIn the web application developed in this study, we utilize
images of ﬁngers captured by a web camera. To extract
feature points from image data and obtain coordinate data,
we employ image recognition technology using a pre-trainedmodel. The pre-trained model leverages MediaPipe Hands [7],which is publicly available, to automatically output the 2D XY

Authorized licensed use limited to: Kochi University of Technology. Downloaded on March 05,2024 at 01:51:59 UTC from IEEE Xplore. Restrictions apply.

2

Fig. 2. Overall architecture of proposed method.

coordinates of 21 feature points on ﬁngertip locations from live
images captured in real-time.
III. P ROPOSED M ETHOD
A. Goal and Approach

Fig. 3. The ﬂowchart for experimenting.

The purpose of this study is to simplify the implementationof tapping tasks and enable quantitative measurements. The
requirements necessary to achieve this goal are outlined below.
Requirement 1: No Complex Operations Required
Requiring complex operations during the task, not only on
the side of the participants but also for the experimenters,
can lead to stress during the implementation and decrease
experiment speed and measurement accuracy. Hence, it isessential to prominently display only the necessary buttons andinput ﬁelds, making the operation methods as straightforward
as possible.
Requirement 2: Clear Presentation of Results
It is necessary to analyze the obtained results and present
them in an easily understandable format. The information to be
displayed includes whether the given stimuli match the actionsof the participants and the reaction times.

2) Module 1-2: Experiment Execution Screen: Once theusername input is completed on the User Registration Screen,
the screen automatically transitions to the Experiment Execution Screen. Next, the experimenter conﬁgures informationsuch as the number of times numbers or characters are displayed and whether to perform the normal or n-back task.
Subsequently, adjustments are made to position the subject’s
hand in front of the webcam to record it. During this process,the experiment can begin once the webcam correctly recognizes the hand. This precaution is taken to avoid starting theexperiment without hand recognition, which would result inno data being recorded. Once the hand is correctly recognized,
the start button becomes operational. Upon pressing the start
button, the tapping task commences. After completing thespeciﬁed number of displays, the tapping task automaticallytransitions back to the initial screen.

B. Overall Architecture

D. Module 2: Integrating Image Recognition and Touch Panel

The overall architecture of the proposed service is illustrated
in Figure 2. The proposed service consists of three main components: user registration, experiment execution, and result
viewing. It is assumed that only the participants operate thesystem during the experiment execution phase, while the othercomponents are intended to be operated by experimenters.

1) Module 2-1: Real-Time Replication of Feature Points
through Image Recognition: During the experiment, hand
image data is captured by a webcam. The captured image data
is processed using the hands.js library, which extracts only
the feature points and records them as coordinate data. Whilethe coordinate data can be overlaid on top of the image datafor visualization, it is set to be hidden by default during theexperiment.
2) Module 2-2: Algorithm Design for Touch Panel Measurements: The touch panel measurements are performed
using ten buttons at the bottom of the experiment execution
screen. Participants place their ﬁngers on the buttons andconduct measurements by touching the corresponding button
in response to stimuli. The web application determines whether
the touched button corresponds correctly to the stimulus anduses this information when displaying results. It is designed to

C. Module 1: Initial and Experiment Implementation Screen
1) Module 1-1: User Registration Screen: The ﬂowchart forexperimenting is depicted in Figure 3. Upon starting the service, the experimenter can choose between conducting the experiment or viewing the results. If they choose to experiment,
they input a username and complete the registration process.
After registration, the screen transitions automatically. Whenviewing the results, the experimenter selects the username theyregistered during the experiment and displays a list of dates
and times.

Authorized licensed use limited to: Kochi University of Technology. Downloaded on March 05,2024 at 01:51:59 UTC from IEEE Xplore. Restrictions apply.

3

Fig. 5. A screenshot of the service’s starting screen.

Fig. 6. A screenshot of the username registration screen.
Fig. 4. The ﬂowchart for assessing the input accuracy.

be considered correct only when the given stimulus and button
correspond accurately and when the button is not pressed
multiple times. If it is correct, the time taken until the button ispressed is recorded as the reaction time. The ﬂowchart of thealgorithm for assessing the input accuracy through the touch
panel is depicted in Figure 4.
E. Module 3: Viewing Experiment Results
From the initial screen, we can navigate to the screen
to view the experiment results. By selecting the username
registered during the experiment execution, we can access theresults of tasks performed by that user. After selecting a user,
we can display the results in JSON format, view a bar graphof accuracy rates, and observe the accuracy trends for eachsession as a line graph.

Fig. 7. A screen for experimenting

B. Function Explanation and Workﬂow During Usage
To begin with, in the initial screen, the experimenter selects
whether to conduct an experiment or view results throughbutton input, triggering a branching of scenarios.
Scenario 1: Conducting the Experiment
Figure 5 illustrates a screenshot of the service’s starting
screen. Initially, the experimenter enters the subject’s username
and registers it. The registered username is necessary forlater reviewing records. Figure 6 displays a screenshot of theusername registration screen. Next, the experimenter speciﬁes
the format of the experiment and the number of times thestimuli will be presented. Subsequently, after conﬁrming thatthe subject’s hand is correctly recognized, the experimenter
presses the Start button to initiate the tapping task. Once thetapping task is completed, the screen automatically transitions
back to the initial screen, enabling the next steps. Figure 7
depicts the screen for experimenting, and Figure 8 illustrates
the data collection process within the proposed service.
Scenario 2: Viewing Results
In the results viewing section, an example of the user

F. Building and Linking Databases
The data recorded through measurements is sent to a MongoDB database [8]. The transmitted data is divided into hand
image and touch panel recognition data, each stored separately.
An example format for retrieving data from the database ispresented in Table I.
IV. I MPLEMENTATION
A. Technologies Employed
The service proposed in this study is executed in a web
browser using node.js [9]. We utilized the “Hands” feature
within the MediaPipe [10] framework to recognize hand
images. Furthermore, we incorporated MongoDB to store
information in a database for data recording. Table II comprehensively lists the technologies and frameworks utilized in
the proposed service.

Authorized licensed use limited to: Kochi University of Technology. Downloaded on March 05,2024 at 01:51:59 UTC from IEEE Xplore. Restrictions apply.

4

TABLE I
A N EXAMPLE FORMAT FOR RETRIEVING DATA FROM THE DATABASE .

Functions

URL Parameters in node.js

Get All UpdateTime

app.get(’/userN=:userN/dbN=:dbN/itemN=:itemN/getAllUpdateTime’

Get All Data

app.get(’/userN=:userN/dbN=:dbN/itemN=:itemN/getAllData’

Get Speciﬁc Data

app.get(’/userN=:userN/dbN=:dbN/itemN=:itemN/updateTime=:updateTime/get’

TABLE IIL IST OF TECHNOLOGIES AND FRAMEWORKS USED .

Name
Google Chrome

Version113.0.5672.127

Node.js

11.0.0

MongoDB

3.5.9

MediaPipe

Fig. 9. An example of the user selection screen in result viewing mode.

Fig. 8. The data collection process within the proposed Web service.

selection screen in result viewing mode is shown in Figure 9.
Users can choose a username and display a data list for eachuser ID. From this list, they can select the data they wish
to view, allowing them to examine input data in JavaScript
Object Notation (JSON) format. Figure 10 shows an example
JSON ﬁle containing touch panel input data, and Figure 11
illustrates an example JSON ﬁle containing coordinate data
recorded through ﬁnger recognition.
Scenario 3: Visualization via Graphs
When viewing results, after selecting a username, users
can choose the graph display checkbox to view bar graphs
displaying accuracy for each measurement or line graphs
showing the accuracy trends. Multiple graphs can be comparedside by side by selecting graphs for multiple measurements.
Figure 12 displays a screenshot of the graph display screen.

Fig. 10. An example JSON ﬁle containing touch panel input data.

C. Discussion

current service has some challenges. Results are output in
JSON format, which lacks readability. Furthermore, there isno direct association between the image recognition results
and the correctness determination of tapping tasks.
To address the above challenges, several methods can beconsidered. One approach is to enhance the presentation of
results, such as displaying them in easily understandable
formats like graphs, which would facilitate comparisons. An-

The service proposed in this study has enabled the rapid
execution and recording of experiments. Speciﬁcally, input
through a touch panel allows real-time correctness assessment during execution, making determining accuracy fromthe records easy. Additionally, managing user names in the
database facilitates the comparison of results from multiple
tapping tasks conducted by the same subject. However, theAuthorized licensed use limited to: Kochi University of Technology. Downloaded on March 05,2024 at 01:51:59 UTC from IEEE Xplore. Restrictions apply.

5

V. C ONCLUSION
In this study, we propose a web service to streamline theexecution of tapping tasks. Our proposed service involves
recognizing input through a touch panel, recording ﬁnger
movements through image recognition, and automatically calculating accuracy. It is expected to reduce the workload forexperimenters and participants due to its user-friendly features,such as simpliﬁed experiment execution and recording. As
improvements to our proposed service, we suggest presenting
the results of tapping tasks in a more visually accessible format
using graphs and image recognition to evaluate tapping task
performance. In addition to these enhancements, conducting
experiments with participants and incorporating their feedback
will reﬁne the service.
ACKNOWLEDGMENT
This research was partially supported by JSPS KAKENHI
Grant Numbers JP19H01138, JP20H05706, JP20H04014,
JP20K11059, JP22H03699, JP19K02973, and Young Scientists (No.23K17006).
R EFERENCES

Fig. 11. An example JSON ﬁle containing coordinate data recorded.

[1] A. Kluger, J. G. Gianutsos, J. Golomb, S. H. Ferris, A. E. George,
E. Franssen, and B. Reisberg, “Patterns of motor impairment in normal
aging, mild cognitive decline, and early alzheimer’disease,” The Journalsof Gerontology Series B: Psychological Sciences and Social Sciences,vol. 52, no. 1, pp. P28–P39, 1997.
[2] K. E. Kobayashi-Cuya, R. Sakurai, N. Sakuma, H. Suzuki, M. Yasunaga,
S. Ogawa, T. Takebayashi, and Y. Fujiwara, “Hand dexterity, nothandgrip strength, is associated with executive function in japanesecommunity-dwelling older adults: a cross-sectional study,” BMC geriatrics, vol. 18, no. 1, pp. 1–8, 2018.
[3] S. Suzumura, A. Osawa, T. Nagahama, I. Kondo, Y. Sano, and A. Kandori, “Assessment of ﬁnger motor skills in individuals with mild cognitive impairment and patients with alzheimer’s disease: Relationshipbetween ﬁnger-to-thumb tapping and cognitive function,” Japanese
Journal of Comprehensive Rehabilitation Science, vol. 7, pp. 19–28,
2016.
[4] K. Hesseberg, G. G. Tangen, A. H. Pripp, and A. Bergland, “Associations between cognition and hand function in older people diagnosed
with mild cognitive impairment or dementia,” Dementia and GeriatricCognitive Disorders Extra, vol. 10, no. 3, pp. 195–204, 2021.
[5] J. Sugioka, S. Suzumura, Y. Kawahara, A. Osawa, N. Maeda, M. Ito,
T. Nagahama, K. Kuno, K. Shiramoto, S. Kizuka et al., “Assessment ofﬁnger movement characteristics in dementia patients using a magnetic
sensing ﬁnger-tap device,” Japanese Journal of Comprehensive Rehabilitation Science, vol. 11, pp. 91–97, 2020.
[6] S. Makai-Bölöni, E. Thijssen, E. M. van Brummelen, G. J. Groeneveld,
and R. J. Doll, “Touchscreen-based ﬁnger tapping: Repeatability andconﬁguration effects on tapping performance,” Plos one, vol. 16, no. 12,
p. e0260783, 2021.
[7] R. K. Veluri, S. R. Sree, A. Vanathi, G. Aparna, and S. P. Vaidya,“Hand gesture mapping using mediapipe algorithm,” in Proceedingsof Third International Conference on Communication, Computing andElectronics Systems: ICCCES 2021. Springer, 2022, pp. 597–614.
[8] K. Saundariya, M. Abirami, K. R. Senthil, D. Prabakaran, B. Srimathi,
and G. Nagarajan, “Webapp service for booking handyman using mongodb, express js, react js, node js,” in 2021 3rd International Conferenceon Signal Processing and Communication (ICPSC). IEEE, 2021, pp.
180–183.
[9] X. Huang, “Research and application of node. js core technology,” in2020 International Conference on Intelligent Computing and HumanComputer Interaction (ICHCI). IEEE, 2020, pp. 1–4.
[10] M. Harris, A. S. Agoes et al., “Applying hand gesture recognition foruser guide application using mediapipe,” in 2nd International Seminarof Science and Applied Technology (ISSAT 2021). Atlantis Press, 2021,
pp. 101–108.

Fig. 12. A screenshot of the graph display screen.

other approach is representing accuracy, making signiﬁcant results more easily discernible. Additionally, leveraging machinelearning techniques may enable the use of hand motion recognition results for correctness determination. By implementing
these methods, the service can overcome its current limitations
and offer improved functionality for experiment execution anddata analysis.

Authorized licensed use limited to: Kochi University of Technology. Downloaded on March 05,2024 at 01:51:59 UTC from IEEE Xplore. Restrictions apply.

6