「様式３」P-41

Development of a Novel Tabletop and Mobile Imaging
Device Based System to Facilitate Learner-CentricEducation［１］Organization
Project Leader: Bill Kapralos (Faculty of Businessand Information Technology, University ofOntario Institute of Technology, Canada)
Representative at RIE: Hiroshi Inokawa(Research Institute of Electronics, Shizuoka
University)
Participants:

Michael Jenkin (Department of ElectricalEngineering and Computer Science, Lassonde
School of Engineering, York University,Canada)
Robert Allison (Department of Computer Scienceand Engineering, Faculty of Engineering, YorkUniversity, Canada)
Julita Vassileva (Computer Science Department,University of Saskatchewan, Canada)
Hirokazu Taki (Faculty of Systems Engineering,Wakayama University, Japan)
Hidetoshi Nonaka (Graduate School ofInformation
ScienceandTechnology,Hokkaido University, Japan)
Karen Collins (Canadian Centre of Arts andTechnology at the University of Waterloo,Canada)
Adam Dubrowski (Divisions of Emergency
Medicine and Pediatrics, Faculty of Medicine,
Memorial University, Canada)
Noriyuki Matsuda (Faculty of Systems
Engineering, Wakayama University, Japan)
Magdalina Todorova (Department of Computer
Science, Faculty of Mathematics andInformatics, Sofia University, Bulgaria)
Michael Vynnycky (Division of Metals Casting,
Department of Material Science Engineering,
Royal Institute of Technology, Sweden)
Patrick Hung (Faculty of Business andInformation Technology, University of Ontario

Institute of Technology, Canada)

Miguel Vargas Martin (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Andrew Hogue (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Sanshiro Sakai (Graduate School of Informatics,Shizuoka University)
Kamen Kanev (Graduate School of Informatics,Shizuoka University)
Alvaro Uribe Quevedo (Multimedia andIndustrial Engineering, Nueva Garanada Mil.
University)
Robert Shewaga (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Robert Codd-Downey (Department of ElectricalEngineering and Computer Science, Lassonde
School of Engineering, York University,
Canada)［２］Research Progress
The purpose of the 2015 Cooperative Research
Project (Research Institute of Electronics,Shizuoka University), titled Development of a

Novel Tabletop and Mobile Imaging Device Based
System to Facilitate Learner-Centric Education,

was to continue and build upon an ongoingcollaborative research initiative between
researchers from several institutions in Japan,
Canada, United States, and Colombia, that is
examining various aspects of novel multi-modal
display technologies including tabletop computing
displays that will be used to facilitate unique andnovel education, e-learning, and training
applications. In addition to their use in educationand training applications, the novel multi-modal
displays will also be used to examine the effect ofmulti-modal interactions on engagement,immersion, knowledge transfer, and retention.
Within the scope of the funded project, our work
has focused on improving and building upon ournovel display approach that couples a tabletop
display with mobile devices to allow forlearner-centric education and content delivery.
Below is a summary of the progress made during
this funding period: Replication of the novel tabletop-tablet displaythat was previously developed at Shizuoka
University (March 2015) in collaboration withYork University (Michael Jenkin), andUniversity of Ontario Institute of Technology(Bill Kapralos).
 Members of the team have developed a novel
mobile application (“app”) to support cardiac
auscultation training. Although the app iscurrently intended for mobile devices (e.g.,mobile phones and tablets), in the future, itwill be modified to work with (and takeadvantage of), the novel tabletop-tablet
display.
［３］Results(３－１) Research results
Replicating the Novel Tabletop-Tablet Display at
Multiple Locations: Tabletop computers/displays
when coupled with a mobile device (e.g., tablet or
mobile/smart phone) provided to each of the users
may provide an effective physical infrastructure
for promoting learner-centered medical education
whereby the information delivered to users iscustomized for each user to account for theindividual levels of learning (e.g., beginner,intermediate, advanced) and is presented on aas-needed basis. This provides for novel learning
opportunities in areas such as teaching humananatomy to medical students and medical-based
virtual interprofessional education training[Dubrowski et al., 2015]. Recently, at Shizuoka
University, researchers from Japan (KamenKanev), Canada (Michael Jenkin, and Bill
Kapralos), and Colombia (Alvaro Uribe Quevedo),
developed a framework that coupled a tabletop
display and mobile devices (tablets and/orsmartphones), to facilitate multi-user teaching
and collaborative education and training[Uribe-Quevedo et al., 2016]. An initial

application of this framework was for facilitating
eye anatomy education. An article outlining thenovel framework and its application to eye
anatomy education was prepared, submitted to
an international conference, and is currently
under review (see [Codd-Downey et al., 2016]). Inaddition, Bill Kapralos was invited to present thiswork as part of the Department of Anatomy andCell Biology Weekly Seminar Series, Schulich
School of Medicine and Dentistry, University ofWestern Ontario, London, Canada, on October 2,
2015.
In order to ensure greater flexibility with thework on the tabletop-tablet display, we havereplicated it at York University (in MichaelJenkin’s laboratory) and the University of Ontario
Institute of Technology (in the laboratory of Bill
Kapralos and Andrew Hogue) so that now, wehave three such displays and can work on eachdisplay simultaneously.
Development of the Cardiac Auscultation
Application (app): Auscultation refers to the act oflistening, either directly or through aninstrument (commonly, a stethoscope) to sounds
within the body as a method of medical diagnosis
to evaluate the frequency, intensity, duration,
number, and quality of sounds associated withvarious parts of the body (typically the heart,lungs, and intestines). Cardiac auscultation
allows for the examination of the mitral, tricuspid,
pulmonary, and aortic valves to obtain
information regarding the heart’s rate, rhythm,
location timing, intensity, quality, and shape.
Identifying heart anomalies on time can help
reduce the number of annual cardiac related
deaths (according to the World healthOrganization, 17.5 million estimated in 2012). A
variety of audio libraries, websites and mobile
apps that help familiarize trainees with thesounds of the heart under normal and diseased
conditions are available. More recently, virtual
reality and manikin-based simulators havebecome widely used for training as they allow
trainees to practice in controlled scenarios where
several procedures can be configured. After
analyzing cardiac auscultation tools, we identified
a gap between available tools limited topresenting concepts and sound in a digital form
(including websites, apps and VR), and thetraining provided by physical simulators wherepractice and assessment takes place. Attempting
to close this gap, we believe that a game-based
mobile application to learn auscultation skills willconnect the traditional information presented
with videos and animations promoting only thememorization, and hands-on practice withmanikins where they develop and test theirauscultation skills. Within the scope of thisfunded project, team member Bill Kapralos,Alvaro Uribe Quevedo, and Adam Dubrowski(along with other researchers including cardiac
surgeon Fuad Moussa from Sunnybrook health
Sciences Centre in Canada and Norman Jaimes
from the Faculty of Medicine, University Mil.
Granada in Colombia), have developed a novel
game-based mobile application (“app”) for
auscultation training. Our app takes advantage of
the widespread use and user familiarity withsmartphone devices to provide a low fidelity
multimodal VR-based auscultation examination
tool. The app uses low fidelity graphics, andpresents a realistic auscultation setup where thetrainee is required to diagnose a virtual patient.
The app allow us to examine the effects of VR
environments (i.e., multimodality, fidelity, andgamification), on the acquisition and developmentof auscultation skills.

Fig. 1. GUI, virtual patient, and sound colliders.

The app was developed using the Unity3D gameengine and incorporates a 3D model patient withheart sounds (see Fig. 1). The first step is tocharacterize the cardiac examination. The
procedure requires the trainee to inspect, palpate,and auscultate the chest overlying the heart. The
examination allows visual inspection andauscultation by moving a virtual stethoscopearound the chest. Collisions with the points ofinterest result in the reproduction of heart sounds
and pop-up questions about it. The goal in thisearly stage is for the user to auscultate the virtual
patient by identifying rate, rhythm, and soundlocation, and to determine whether it’s a normal

or pathological sound. To challenge and engage
the user, game elements (e.g., challenge,procedure, score and time) are employed toincrease difficulty and reward the user. Healthyand diseased sounds are randomly presented
across the chest to encourage users to pay
attention rather than just memorize the same
sound in the same position with time constraints
associated to the sound intensity level. An article
outlining the auscultation app will be presented
at the 4th International Conference on Serious
Games and Applications for Health in May 2016(see [Uribe-Quevedo et al., 2016]). We have also
recently conducted a preliminary study withhuman participants (medical students inColombia) to examine the effectiveness of the app.
The results are being analyzed and an article will
be prepared and submitted to a journal shortly.
(３－２) Ripple effects and further developments
We are continuing to build-upon and expand theinnovative work and progress that was madeduring this particular funding period. Work willcontinue simultaneously at three locations where
the displays are currently available: Shizuoka
University, York University, and University of
Ontario Institute of Technology. The work to bedone will involve continuing to improve the novel
tabletop-tablet display that we have developed.
More specifically, we will examine theincorporation of spatial sound to the display,taking advantage of our previous work that hasdeveloped various spatial sound techniques fortabletop displays relying on amplitude panning.
We have previously shown (via human based“ground truth” experiments) that localizing a real
(actual) sound source on a horizontal plane isdifficult and error prone [Lam et al., 2015]. Herewe will examine whether the inherent soundlocalization errors can be reduced with theaddition of graphical cues (a review article
outlining audio-visual interactions and fidelity
was recently published by team members Bill
Kapralos, Karen Collins and Adam Dubrowski[Kapralos et al., 2016]). We will also examine
methods to couple and synchronize the spatial
sounds on the tabletop display to the mobiledevice.
Future work will also include continuing thedevelopment of the auscultation app. This will
include conducting a Delphi study with thestakeholders of the mobile application (e.g.,medical educators) to gather information of the
key components/elements that should be included
to make the application a useful learning tool.
Once we have reached agreement consensus
through our Delphi process, we will use thisinformation to refine the development of the app.
Once the application has been developed, it willgo through a validation process to verify that theuse of the newly developed tool provide theknowledge we intended to reach. This will include
conducting formal usability experiments withmedical students.
［４］Achievements (List of Publications)（１） R. Codd-Downey, R. Shewaga, B.
Kapralos, A. Uribe-Quevedo, K. Kanev, and M.
Jenkin. A novel tabletop and tablet-based display
system to support learner-centric ophthalmic
anatomy education. Submitted to 3rdInternational Conference on Augmented Reality,
Virtual Reality and Computer Graphics(SALENTO AVR 2016), Otranto, Lecce, Italy,
June 15-18, 2016 (currently under review).

（２） A. Dubrowski, B. Kapralos, K. Kanev,and M. Jenkin. Interprofessional critical caretraining:
Interactive
virtuallearning
environments and simulations. In Proceedings ofThe Sixth International Conference on
Information,Intelligence,SystemsandApplications (IISA 2015), Corfu, Greece, July 6-8,2015.

（３） B. Kapralos, F. Moussa, K. Collins, and A.
Dubrowski. Levels of fidelity and multimodal
interactions. In P. Wouters and H. van

Oostendorp (Eds.) Techniques to Improve theEffectiveness of Serious Games, Springer
Advances in Game-based Learning book series (toappear 2016).
（４） J. Lam, B. Kapralos, K. Kanev, K. Collins,
A. Hogue, and M. Jenkin. Sound localization on ahorizontal surface: Virtual and real soundlocalization. Virtual Reality, Special Issue on
Spatial Sound in Virtual and Augmented Reality,
19(3):213-222, 2015.
（５） A. Uribe-Quevedo, B. Kapralos, A. Hogue,
K. Kamen Kanev, M. Jenkin, and R. P. Barneva.
A multi-user tabletop display with enhanced
mobile visuals for teaching and collaborative
training. In Proceedings of the Consortium forComputing Sciences in Colleges — Northeastern
Region 2016 Conference, Clinton, NY, USA, April
29 – 30, 2016 (to appear).

（６） A. Uribe-Quevedo, D. Rojas, M. Usman,B. Kapralos, F. Moussa, A. Dubrowski, and N.
Jaimes. Cardiac auscultation serious gameapproach. In Proceedings of the 4th International

Conference on Serious Games and Applicationsfor Health, Orlando, FL, USA, May 11-13, 2016
(to appear)（７） Barneva, R., Kanev, K., Kapralos, B.,
Jenkin, M., Enhanced student engagement using
cell phones and tabletop computers or smart
boards, In Proceedings of 25th CIT 2016
Conference on Instructional Technologies,Potsdam, NY, USA, May 31-June 3, 2016 (toappear).

Travelling Report（Mention each travel by CRP budget.）Name：
Affiliation：
Period of time：
Destination：
Purpose：Name of receiver：