「様式３」課題番号 ：P-4

Promoting Collaborative Interprofessional Education for
Critical Care Teams with a Table-Top Computer-Based
Virtual E-Learning Environment Engaging Advanced
Imaging Devices[1] Organization
Leader: Bill Kapralos (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Representative at RIE: Hiroshi Inokawa (ResearchInstitute of Electronics, Shizuoka University)
Participants:
Hirokazu Taki (Faculty of Systems Engineering,
Wakayama University, Japan)
Hidetoshi Nonaka (Graduate School ofInformation
ScienceandTechnology,
Hokkaido University, Japan)
NoriyukiMatsuda(Faculty of
Systems
Engineering, Wakayama University, Japan)
Michael Jenkin (Department of Computer Scienceand Engineering, Faculty of Engineering, York
University, Canada)
Robert Allison (Department of Computer Scienceand Engineering, Faculty of Engineering, York
University, Canada)
Patrick Hung (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Julita Vassileva (University of Saskatchewan,
Canada)Magdalina Todorova (Department of Computer
Science, Faculty of Mathematics andInformatics, Sofia University, Bulgaria)
Michael Vynnycky (Department of Mathematics
and Statistics, University of Limerick, Ireland)
Karen Collins (Canadian Centre of Arts andTechnology at the University of Waterloo,
Canada)Adam Dubrowski (The Hospital for Sick Children
Learning Institute, and Department ofPediatrics, University of Toronto, Canada)
Andrew Hogue (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Lennart Nacke (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Boris Brimkov (Department of Mathematics,
University at Buffalo)
Sanshiro Sakai (Graduate School of Informatics,

Shizuoka University)
Kamen Kanev (Graduate School of Informatics,
Shizuoka University)[2] Progress of the research
The purpose of the 2013 Cooperative Research
Project (Research Institute of Electronics,
Shizuoka University), was to continue and expand
upon an ongoing research collaboration between
researchers from Japan and Canada that is
examining various aspects of tabletop computing
displays and virtual simulation for e-learning
applications. Within the scope of the current
funding, our work focused on the development of
an innovative, unique, strategy-based, learnercentered, interactive, virtual learning environment(VLE) for IPE in pediatric critical care (see (2,6)
for a discussion on the use of virtual simulations
and serious games for health professions educationand training). The VLE will provide a safe
environment whereby trainees apply and practiceIPE-specific skills in a remote (online), multi-user
setting thus eliminating the need, and problems
associated with, bringing together healthprofessionals of various disciplines to train at one
location. In addition to its use as a training tool for
health professionals, the VLE will be developed asa research tool that will permit the altering ofvarious simulation parameters (e.g., audio andvisual fidelity). This will allow us to examine
multi-modal interactions and simulation fidelity on
learning and explore questions such as thefollowing: Do higher fidelity visual and auditory
cues lead to a greater transfer of knowledge? How
do changes in audio/visual fidelity affectstudents’/trainees’ ability to perform specifictasks? Is a scaffolding approach whereby fidelity is
increased in proportion to trainee level, effective?
What effect does stereoscopic 3D viewing have on
immersion, engagement, and ultimately learning(see (8))? What effect does spatial sound have on
learning? Answering such questions anddeveloping a greater understanding of multi-modal
interactions has implications for virtual reality

applications in general (including virtual
simulations) given that despite the great computing
hardware and computational advancements overthe last decade, real-time, multi-modal renderingof complex virtual environments is still beyond ourcurrent reach; having a good understanding ofmulti-cue interaction will allow us to takeadvantage of perceptual-based rendering whereby
the rendering parameters are adjusted based on theperceptual system (typically vision), to limit
computational processing.
Although this is a long-term goal, significant
progress has been made towards this ultimate goal
within this funding period. More specifically thefollowing milestones were reached:• A scenario outlining a pediatric patient andcritical care response team was developed.
• A series of experiments that examined audiovisual interactions and more specifically, theeffect of contextual auditory cues on visual
fidelity perception were conducted (4,7).
• A novel game-based calibration method thatallows users to customize/individualize thevirtual environment before they begin using
was developed (4).
• Continuation and expansion of the JapanCanada research collaboration with additional
research members.
Greater details regarding each of the above
milestones is provided is the following section.
[3] Results(3.1) Research results
Below, greater details regarding the four areas ofprogress over the last project year are provided.
(3.1.1) Pediatric Patient Scenario
Patient history: Johnny is a 7 year old, malepatient admitted to an emergency room in a tertiarychildren’s hospital. Upon admission, his parents
report that he has been complaining of dizzinessand nausea. While at the emergency room,
Johnny loses consciousness and stops breathing.
Simulation Protocol: Within this scenario, acomputerized mannequin, that is known as the
Human Patient Simulator (HPE), plays the role ofJohnny. The emergency nurse calls “code blue”
and the Critical Care Team (CCT) is mobilized.
The CCTs are fundamental components of most
hospital patient safety infrastructure designed toassist with critically unstable patients. The CCTs
consist of multi-professional teams consisting ofnurses, respiratory therapists and physicians.

The team is expected to respond according to strict
protocol that dictates actions, sequences, timing
and coordination with other team members. The
HPS-Johnny is capable of providing the team withvital signs, such as breathing, oxygen saturation,
heart rate, just the way a real patient would. His
chest rises as he breaths and he can communicate
with the team using voice. The simulation event issimilar to role-playing exercise, where the teacher
plays the role of the patient. She can adjust thevital signs, rhythms etc. and talk to the team. She
follows an algorithm developed based on realclinical experiences.
(3.1.2) Audio-Visual Interaction Experiments
Previous work has shown that sound can affect theperception of visual fidelity. This work expanded
upon our previous work by examining the effect ofcontextual sound cues (i.e., sounds that are related
to the visuals) on visual fidelity perception througha user-based study. Results suggest that contextual
sound cues do influence visual fidelity perception
and, more specifically, our perception of visual
fidelity increases with contextual sound cues.
These results have implications for designers ofmultimodal virtual worlds and serious games that,with the appropriate use of contextual sounds, canreduce visual rendering requirements without acorresponding decrease in the perception of visual
fidelity. Greater details regarding this study can befound in (7).
(3.1.3) Game-Based Calibration Method
Given the large variation observed acrossindividuals with respect to the perception of audiovisual fidelity and with the respect to the effect ofaudio cues in the perception of visual fidelity (see(4)), we have recently begun examining thecustomization (individualization) of a virtualenvironment by each individual user through theuse of a game-based calibration method.
Customization of the audio and visual interface isaccomplished using a simple game-based approach,making the process interactive and far moreengaging. Our approach is inspired by standard
testing methodologies employed by optometrists todetermine the optimum properties of corrective
lenses in order to overcome a variety of visual
deficiencies. The calibration game presents theuser with a split screen with the same gamerunning in each window but under different
fidelity/realism settings with a single backgroundsound. The player chooses the screen they preferby clicking a button just above the corresponding
window. Their choice will be registered and theaudio-visual fidelity of the game running in theother window will change (increase or decrease).
This process is repeated over a number of cycles(the total number of cycles can be easily modified),until the optimal fidelity level is reached (greater
details are provided in (4)).
(3.1.4) Expanding and Building-Upon theJapan-Canada Research Collaboration
Recently, an international team of interdisciplinary
researchers across several institutions in Japan, (Dr.
Kamen Kanev from Shizuoka University), andfrom Canada (Dr. Michael Jenkin from York
University, Dr. Bill Kapralos from the Universityof Ontario Institute of Technology, Dr. Adam
Dubrowski from Memorial University, and Dr.
Karen Collins from the University of Waterloo),
have established an international collaboration toinvestigate various problems and issues related totabletop computers. Their independent andcollaborative efforts have examined a range ofissues related to the development of effective
tabletop computer display technology. Emphasis
has so far been placed on spatial sound generationand localization and our work to date has focusedon the generation (and localization) of spatial
sound for tabletop computers and has led to somenovel results and innovations including thefollowing: i) development of two spatial sound(amplitude panning) techniques for tabletop
computers, ii) the use of video games forevaluating human-computer interaction and inparticular, for conducting usability tests to
examine various loudspeaker configurations andamplitude panning techniques for tabletop
computers, iii) distributed audio over mobile
devices, and iv) developing novel hardware andusing the hardware to conduct a series ofexperiments that investigated sound localization
for both virtual and actual sound sources
positioned on a horizontal surface.
Overall, our work has shown that soundlocalization on a horizontal surface (inherent witha tabletop computer), is erroneous irrespective ofthe techniques used to generate the sound or the
loudspeaker. Developers and designers ofapplications for tabletop computers must accountfor this and not necessarily rely on accurate soundlocalization as a primary method conveying
information. Rather, they may exaggerate
placement when sound source positions
correspond to positions with large error. It has
been shown that sound source localization varies
with frequency and changes in frequency, sounds
that have more formants/overtones are easier tolocalize than sine waves, and reverberation also
aids sound source localization. These are all

potential areas that warrant further investigation.
Furthermore, images tend to “magnetize” sounds
to a specific location and it can therefore be
assumed that some errors can be corrected throughthe visualization of sounds.
Our most recent efforts have focused on building
upon and expanding our current progress and areprimarily focused on i) collection of reference data
for audio-visual table top displays, ii)
incorporating visual cues in audio-visualinteractive displays (including tabletop computers),
and examining the interaction of audio and visualcues in enhancing the user experience, and iv)
development of an innovative, unique, strategybased, learner-centered, interactive, virtual
learning environment (VLE) for interprofessional
education (IPE) in pediatric critical care that
employs a table-top display to leverage its inherent
collaborative nature (described further below). The
funding received from the Cooperative Research
Projects fund (Research Institute of Electronics,
Shizuoka
University)
titled“Promoting
Collaborative Interprofessional Education for
Critical Care Teams with a Table-Top ComputerBased Virtual E-Learning Environment Engaging
Advanced Imaging Devices” was used to directly
support our ongoing work. More specifically, thefunds were used to support the visit to Shizuoka
University by three Canadian researchers in
December 2013 to meet with their Japanesecolleagues/partners with regards to their ongoingcollaboration. The researchers included existing
collaborators Dr. Michael Jenkin, and Dr. Bill
Kapralos in addition to a (new) researcher, Dr.
Patrick Hung from the University of Ontario
Institute of Technology.
During the visit to Shizuoka University inDecember 2013, the visiting Canadian researchers
participated in a Cooperative Research Workshop
titled “Promoting Collaborative Interprofessional
Education for Critical Care Teams with a TableTop
Computer-Based
VirtualE-Learning
Environment Engaging Advanced Imaging
Devices” held on Friday December 13, 2013. The
workshop was organized by Dr. Kamen Kanev andheld at RIE, Shizuoka University. It included thefollowing presentations by the Canadian visiting
researchers to describe their current work andfurther collaboration (exchange) opportunities forstudents and faculty members between Shizuoka
University and the University of Ontario Instituteof Technology.
• Dr. Michael Jenkin: A Model for Interaction
through Augmented Surfaces.

• Dr.
Patrick
Hung:
Evidence-based
Technology: MaxIt Systems.
• Dr. Bill Kapralos: Interprofessional Critical
Care Training – Interactive Virtual
Environments• Dr. Patrick Hung: Possibilities for Joint
Research Collaboration and Exchanges
between University of Ontario Institute ofTechnology and Shizuoka University.
In addition, the visit by the Canadian researchers
coincided with the International Conference on
Humans and Computers 2013, Hamamatsu, Japan,
December 17, 2013. Dr. Kapralos presented aninvited presentation titled “Virtual Simulation and
Serious Gaming: Audio-Visual Interactions,
Perceptual-Based Rendering, and the Implications”.
In addition, the project members presented thefollowing works:• Dr.
Patrick
Hung:
Evidence-based
Technology: Case Studies and Interactions• Adam Dubrowski, Bill Kapralos, Michael
Jenkin and Kamen Kanev: Interprofessional
Critical Care Training: Interactive Virtual
Learning Environments and Simulations.
• Kamen Kanev, Itaru Oido, Patrick Hung, Bill
Kapralos and Michael Jenkin: Augmented
Reality Toys: Applications and Future Works.
Additional activities included “Japan-Canada meetand greet” on Friday, December 13, 2013
(Takayanagi Museum Building from 18:00 –20:00) and organized by Dr. Valerie Wilkinson.
The three Canadian researchers attended the “meetand greet” where they had the opportunity to talkto various students and faculty from Shizuoka
University and other guests.
(3.2) Future WorkWe are continuing to build-upon and expand ourJapan-Canada collaboration. More specifically, Dr.
Kapralos submitted a proposal to the Japan Society
for the promotion of Science (JSPS) January 15 tosupport a visit to Japan (Shizuoka University) for
60 days (January 15 2015 – March 15 2015) tocontinue ongoing work that is examining multimodal interactions and tabletop computers(decision to be made in May 2014). In addition,
researchers from the University of Ontario Instituteof Technology (UOIT) and Dr. Kamen Kanev arein the process of preparing and submitting a multimillion grant proposal to the Canadian Foundationof Innovation (CFI) 2015 Innovation Fund forinfrastructure to support a User-Centred
Informatics Research Space (UCIRS). The UCIRS
will contains state-of-the-art visualization,

computing, and sensing capabilities layered in amodular manner in order to design and test
evaluate user interaction with medical, simulation,
training and entertainment informatics via brainbody sensing and natural user interfaces forcollaboration and affective user sensing, two-way
holographic/volumetric video conferencing, in the
NICU and user-centered informatics context . The
facility will incorporate high quality stereoscopic
3D projection, full body motion capture andtracking, computer vision systems for creating 3D
representations of the environment in real-time,and state of the art observational recording for
behavioral and physiological analysis. One of the
projects to be investigated includes developing thecontrol room of the future. The CFI application
has already passed the internal UOIT review
process and was selected (amongst two otherproposals) to proceed with the next stage(development of the Notice of Intent) CFI. The
Notice of Intent is due Friday March 28, 2014
while the full proposal is due Friday June 27, 2014.
With respect to publications, currently Drs. Hung,
Kanev, Jenkin, and Kapralos are completing anarticle (book chapter) that will appear in the book“Mobile Services for Toy Computing” that will be
published by Springer in late 2014. The article istitled “Augmented reality (AR) toys” (1).
Furthermore, Drs. Dubrowski, Kapralos, Jenkin,
and Kanev are preparing an article titled“Interprofessional
CriticalCare
Training:
Interactive Virtual Learning Environments andSimulations”.
Kapralos and Kanev are also
preparing an article titled “Spatial Sound for
Virtual and Augmented Reality Environments:
Problems, Limitations, and Implications for
Mobile Devices” that will also appear in the book“Mobile Services for Toy Computing” (3). The
article will provide an overview of spatial soundwith an emphasis on the generation of spatial
sound for mobile devices. The article will be
submitted to the Medicine Meets Virtual Reality
2015 conference (deadline is July 2014) andsummarizes progress to-date regarding the work
done by the authors that is examining the use of avirtual learning environment (simulation) forinterprofessional education IPE for pediatriccritical care teams using a tabletop computing
platform coupled with novel image-based sensing
technologies. This will allow for collaboration andinteraction amongst a group of trainees while
promoting a learner-centric learning approach
where the simulation is tailored specifically to theneeds of each trainee.

[4] Publications(1) P. Hung, K. Kanev, M. Jenkin, and B.
Kapralos. Augmented reality (AR) toys. In
Technologies of Inclusive Well-Being. In P.
Hung (Ed.), Mobile Services for Toy
Computing, Springer, to appear 2014.
(2) B. Kapralos, F. Moussa, and A. Dubrowski.
An overview of virtual simulations andserious games for surgical education andtraining. In Technologies of Inclusive WellBeing. In A. Brooks, S. Braham, and L. Jain
(Eds.),
Springer
SeriesStudiesinComputational Intelligence, Heidelberg,
Germany, Chapter 14, pp. 289-306, 2014.
(3) Kapralos, and K. Kanev. Spatial sound for
virtual and augmented reality environments:
Problems, limitations, and implications formobile devices, In P. Hung (Ed.), Mobile
Services for Toy Computing, Springer, toappear 2014.
(4) B. Kapralos, R. Shewaga, and G. Ng. Serious
games and virtual simulations: Customizing
the audio-visual interface. Well-Being,
Rehabilitation, and Healthcare: Serious
Games, Alternative Realities, and Play
Therapy Parallel Session at the 6th Int.
Conference on Virtual, Augmented and Mixed
Reality, HCI International 2014, Crete,
Greece, Jun. 22-27, 2014 (to appear). Invited
paper and presentation.

(5) J. Lam, B. Kapralos, K. Collins, A. Hogue,
and K. Kamen. Amplitude Panning-BasedSound System for a Horizontal Surface
Computer: A User-Based Study, ACM
Computers in Entertainment (to appear, 2014).
(6) S. de Ribaupierre, B. Kapralos, F. Haji, E.
Stroulia, A. Dubrowski, and R. Eagleson.
Healthcare training enhancement throughvirtual reality and serious games. In M. Ma, L.
Jain, and P. Anderson (Eds.), Virtual,
Augmented Reality and Serious Games for
Healthcare 1, Springer-Verlag (to appear
2014)(7) D. Rojas, B. Cowan, B. Kapralos, K. Collins,
and A. Dubrowski. The effect of contextual
sound cues on visual fidelity perception.
Medicine Meets Virtual Reality 2014,
Manhattan Beach, CA, USA, Feb. 20-22,
2014. Also appears in Studies in HealthTechnology and Informatics,196:346-352,
2014.
(8) M. Tawadrous, A. Hogue, B. Kapralos, and K.
Collins. An interactive in-game approach touser adjustment of stereoscopic 3D settings.
In Proceedings of Stereoscopic Displays andApplications XXIV, February 3-7, 2013, San
Francisco, CA, USA, pp. 1-6.

「様式３」
Traveling Report
Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Dr. Bill Kapralos
Faculty of Business and Information Technology
December 8 – 19, 2013
Shizuoka University, Japani) to discuss the ongoing Japan-Canada research collaborations and project
organization, ii) to schedule future work, iii) to participate (provide a presentation) at
the “Promoting Collaborative Interprofessional Education for Critical Care Teams
with a Table-Top Computer-Based Virtual E-Learning Environment Engaging
Advanced Imaging Devices” workshop held on Friday December 13, 2013,
and iv) to participate (present research and an invited presentation) at theInternational Conference on Humans and Computers 2013 held on December 17,
2013 (Shizuoka University).
Dr. Kamen Kanev
Dr. Patrick Hung
Faculty of Business and Information Technology
December 10-20, 2013
Shizuoka University, Japani) to discuss the ongoing Japan-Canada research collaborations and project
organization, ii) to schedule future work, iii) to participate (provide a presentation) at
the “Promoting Collaborative Interprofessional Education for Critical Care Teams
with a Table-Top Computer-Based Virtual E-Learning Environment Engaging
Advanced Imaging Devices” workshop held on Friday December 13, 2013,
and iv) to participate (present research and an invited presentation) at theInternational Conference on Humans and Computers 2013 held on December 17,
2013 (Shizuoka University).
Dr. Kamen Kanev
Dr. Michael Jenkin
Department of Computer Science and Engineering
December 8-14, 2013
Shizuoka University, Japani) to discuss the ongoing Japan-Canada research collaborations and project
organization, ii) to schedule future work, and iii) to participate (provide apresentation) at the “Promoting Collaborative Interprofessional Education for Critical
Care Teams with a Table-Top Computer-Based Virtual E-Learning Environment
Engaging Advanced Imaging Devices” workshop held on Friday December 13, 2013.
Dr. Kamen Kanev