「様式３」課題番号 ：P-7

Multimedia Interaction Interfaces for Table-Top
Computers in Collaborative E-learning Environments[1] Organization
Leader: Bill Kapralos (Faculty of Business andInformation Technology (UOIT), University of
Ontario Institute of Technology, Canada)
Representatives at RIE: Kamen Kanev & Hiroshi
Inokawa (Research Institute of Electronics,
Shizuoka University)
Participants:
Michael Jenkin (Department of Computer Scienceand Engineering, Faculty of Engineering, York
University, Canada)
Julita Vassileva (Computer Science Department,
University of Saskatchewan, Canada)
Hirokazu Taki (Faculty of Systems Engineering,
Wakayama University, Japan)
Hidetoshi Nonaka (Graduate School ofInformation
ScienceandTechnology,
Hokkaido University, Japan)
Karen Collins (Canadian Centre of Arts andTechnology at the University of Waterloo,
Canada)Adam Dubrowski (The Learning Institute, Hospitalfor Sick Children, University of Toronto,
Ontario, Canada)
NoriyukiMatsuda(Faculty of
Systems
Engineering, Wakayama University, Japan)
Magdalina Todorova (Department of Computer
Science, Faculty of Mathematics andInformatics, Sofia University, Bulgaria)
Michael Vynnycky (Department of Mathematics
and Statistics, University of Limerick, Ireland)
Andrew Hogue (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Lennart Nacke (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Boris Brimkov (Department of Mathematics,
University at Buffalo, USA)
Sanshiro Sakai (Department of Computer Science,
Faculty of Informatics, Shizuoka University)[2] Progress of the researchThis cooperative research project continues tofoster, expand upon, and further develop the workof an international research team that isinvestigating the design and development ofinnovative interfaces and environments for e-learning based on natural interaction paradigms
that are adjustable to the needs of specific usergroups. A large part of this work has examinedinteraction with tabletop computers (also known as
surface computers and smart tables), where users
position themselves around a horizontal computer
screen in a manner similar to sitting around a“traditional” table and a combined audio/visionbased interface is employed for sound source anduser tracking in real-time. Although tabletop
computers have yet to be primarily designed asconsumer models, with the growing popularity ofmulti-touch mobility devices (e.g., iPhone/iPads,
smartphones), the move to multi-user touch
screens and a horizontal surface is a likely
trajectory of the technology. Moreover, thesedevices may well become a part of social
entertainment, where families and friends caninteract with each other around a table-like surface.
Given that tabletop computers are intended to havemultiple users seated around the computer display
interacting with the display itself, they naturally
promote interaction amongst users providing an
engaging environment to promote a collaborativeeducation.
Despite the potential benefits of tabletop
computers, there are a number of open issues thatmust be addressed before they become more
widely used, particularly for interactive e-learning
educational applications. One critical questionthat must be addressed in order to fully
operationalize tabletop displays is the user’s ability
to integrate multimodal display cues within thedevice. Of particular interest in this project has
been the ability of the user to integrate spatializedvisual and audio cues into a coherent perception ofa spatialized event. The problem of spatial soundgeneration for such computing platforms iscomplicated by the fact that headphones (as used
in “traditional” computing platforms to deliver
spatial sound) are not an option as they will restrict
and interfere with the ability of users to interact
with each other. Therefore, spatial sound must be
delivered via a number of loudspeakers positioned
around the tabletop computer.
To this end, ourprior work within the scope of this project has
investigated the generation of spatial sound via aseries of loudspeakers surrounding the tabletop
computer. We have developed various amplitude

panning methods to allow for sound to bespatialized to some point on a horizontal surface
and we have experimented with various
loudspeaker configurations. We have also
examined (through user-based experiments), theability to localize a sound on a horizontal surface
using our amplitude panning method although it isprone to error (see Lam et al, 2013).
Although we have made great progress, plenty ofanswers remained unanswered. More specifically,
despite the fact that localizing a (virtual) sound on
a horizontal surface is error-prone, we do not haveany “ground truth” data to compare this to. In
other words, just how accurately are we able tolocalize a sound on a horizontal surface when there
is a physical sound at the corresponding soundlocation? Within the scope of this particular
project, we developed a novel hardware setup toaddress this particular question. In addition to thebasic research questions regarding multimodal
interactions and sound localization on a horizontal
surface (of a tabletop display), within the scope of
this project, we have recently begun working on anovel application of a tabletop computer thatinvolves its use in the interprofessional training ofhealth professional teams (see (Dubrowski et al.,2013) for greater details). Although there aresome interesting and perhaps difficult technicalchallenges to be addressed in order to accomplish
this, the use of an interactive tabletop “simulator”
for training of health professionals presents aninteractive, engaging, platform making it an ideal
environment for such training where multiple teammembers interact with the patient and with eachother. Furthermore, such a platform is very costeffective when compared to traditional simulation
centers (that are costly to build and maintain),where most of this training takes place.
A number of research exchanges took place during
the past year. More specifically, visits to Shizuoka
University by several of the Canadian members of
the research team took place in January andFebruary 2013 (Professor Kapralos visited fromJanuary 31 to February 14, Professor Dubrowski
from February 7 to 14, and Professor Jenkin fromFebruary 8 to 14). As will be outlined throughout
the remainder of this report, during their visit, they
devised experiments that will be conducted in
Canada upon their arrival regarding soundlocalization on a tabletop (horizontal) display, anddiscussed, potential solutions and future plans forthetabletop
computer“simulator”
forinterprofessional training of health practitioner
teams. Professor Kanev undertook two trips toOntario to visit the University of Ontario Instituteof Technology (UOIT), University of Toronto, andYork University in 2012. The first trip in early fall
was to conduct groundwork and set up the basic
parameters for a more extended work with a
specialized experimental apparatus throughout thefall and winter. During his second visit andafterwards the experimental apparatus (see results,
below) constructed at York University andinstalled at UOIT was used for further experiments.
3 Results
During the past year, work within the project has
focused on table-top, touchscreen displays and two
topics in particular: i) developing a greater
understanding on how humans perceive spatialized
audio cues in a tabletop environment, and more
specifically, localization of a sound source
emanating from a horizontal surface (relative tothe listener), and ii) employing the tabletop,
touchscreen display to support interprofessional
education amongst teams of medical professionals.
Greater detail regarding each of these two research
topics is provided in the sections below.
3.1 Sound Localization on a Horizontal Surface(“Ground Truth” Experiments)
As previously described, our work has examined
the localization of a virtual sound source (that was
generated using several amplitude panning
methods; see Lam et al. (2013)) on a horizontal
surface, measured sound localization of virtual
sound sources on a horizontal surface, andcompared different panning methods or different
loudspeaker configurations. However, “groundtruth” data to compare these results with, waslacking. In other words, just how accurately can
we localize a sound on a horizontal surface when
the sound is emanating from an actual soundsource at the corresponding location?
This
particular work addressed this question. A novel
soundverification
hardwaresetup
andmethodology was constructed (by Professors
Jenkin, Kanev, Kapralos, and Hogue) and used tocollect “ground truth” data in order to allow formeaningful comparisons with our previous results
to be made. It allowed a single physical soundsource to be moved to 36 pre-defined locations(positioned on a grid with x- and y-axis separationsof 0.15 m) in a simple and efficient manner. This
grid of 36 locations was identical to the grid weused in our previous experiments that saw a virtual
sound spatialized to one of the 36 locations.

3.1.1 Experimental Procedure
The experiments took place in an Eckel
audiometric room at the University of Ontario
Institute of Technology (room dimensions of 2.3 m
× 2.3 m × 2.0 m). The Eckel audiometric room
provides (frequency dependent) noise reduction
across a wide range of frequencies (e.g., 19 dB at
125 Hz and 60 dB at 4 kHz). Five volunteers
participated in the experiment and their average
age was 29. Participants were seated on a chair
0.51 m from the surface at a height of 1.36 m andinstructed to look forward at the green marker
located at the center of the box. In an effort to limit
deviations from their intended positions,participants were asked to line up the tip of theirnose with a thin piece of string (with a weight on
its bottom) hanging from the ceiling of theaudiometric room. For each trial, the loudspeaker
was physically moved to one of the 36 soundsource positions, the sound stimulus (white noise)
was presented and the participant’s task was toindicate which of the 36 positions they believed
the sound was emanating from. Participants
indicated their choice by stating the corresponding
row and column to the experimenter who thenrecorded their choice. The sound was turned off
and the next trial began. A total of 36 grid
positions (spatial sound sources) were consideredand each position was repeated two times leading
to a total of 72 trials (i.e., 36 grid positions × 2
repetitions). Each of the 72 sound source positions
was considered in random order. Prior to the startof the experiment, participants were presented withthe auditory stimulus at each of the four corner
positions of the surface (individually, one after theother) to provide them with a reference.
In addition to the collection of ground truth datawith respect to a horizontal surface, the experiment
was repeated with the box positioned vertically(i.e., flipped 90 degrees).
3.1.2 Experimental Results
The results of this experiment indicate that soundlocalization on a horizontal surface with actual
sound sources is erroneous albeit to a lesser degree
than sound localization of virtual sound sources.
The average error Euclidean distance (thedifference between the actual and perceived soundsource positions, measured in meters) and standard
deviation for each of the 36 virtual sound source
positions (averaged across each of the five
participants) for the horizontal configuration
ranged from 0.02 m to 0.32 m with an average of
0.18m ±0.07 m. Given the grid spacing of 0.15 m

× 0.15 m, participants were able to localize thesound source to within approximately two
positions of the actual virtual sound source. The
average error (Euclidean distance) and standard
deviation for each of the 36 virtual sound source
positions (averaged across each of the five
participants) for the vertical configuration ranged
from 0.02 m to 0.23 m with an average of 0.13m ±0.05 m. Given the grid spacing of 0.15 m × 0.15m, participants were able to localize the soundsource to within approximately one position of theactual virtual sound source. A graphical summaryof the results for the horizontal and verticalconfigurations are provided in Figures 1 and 2
respectively.

Figure 1. Sound localization results (average
error) for the horizontal configuration.

Figure 2. Sound localization results (average
error) for the vertical configuration.
Although smaller than the errors arising from theamplitude panning methods, even in the presenceof a physical sound source at the correspondingposition, localizing the sound source is still prone
to error hence, one should not expect to eliminate
the errors when employing amplitude panning
methods to spatialize a sound source to a position
on a horizontal surface.
3.2 Tabletop Simulator for Interprofessional
Education of Health Professional Teams
Simulation in health professions education (HPE)
is defined as the replication of a task or an eventfor the purpose of training.
The act itself
demands reduction of a very complex and often
overwhelming event, such as stabilizing a criticallysick patient, to its basic elements.
These
elements are subsequently learned in less complex
environments designed to optimize the learning
process (Guadagnoli, Morin, Dubrowski, 2012;
Matsuda et al 2012).
Although simulation strives for a learner-centric
approach, where information is provided to thelearner on an as-need basis, the current simulation
technologies do not meet this goal. The firstproblem is the problem of the simulator.
Typically, the simulators often referred to asmodels, have a limited number of learning
objectives that they can deliver. For example, asynthetic bench-top simulator, such as arepresentation of an arm, may be well suited forteaching clinical technical skills of intra-venous(IV) catheter placement, but the same simulator is
not well suited for teaching the underpinninganatomy. The second problem is that of thelearner. Simulation is often used to train teams ofhealth professionals not only to develop theclinical skills within their own domain, but also tounderstand what skills need to be performed byother members of the team, as well as develop
skills that need to be performed together as a team.
Because multi-professional teams consist ofmembers with various backgrounds, as well as
skills, knowledge and attitude (SKA) levels,
learning styles and needs, the same-for-all mode of
information delivery may not be adequate for
optimal learning.
Tabletop computers where, as described above,users position themselves around a horizontal
computer screen in a manner similar to sitting
around a “traditional” table, promote collaboration
amongst the users. Tabletop computers provide
the opportunity for the development of innovative,
engaging, interactive, and highly collaborative
pedagogic applications, lending themselves nicely
to interprofessional training of health professionals.
This particular joint Canada-Japan research projectis examining the use of a multiuser interactive
tabletop touch-screen computer in conjunction
with novel image-based sensing and positioning
technologies to allow for virtual simulators that
promote embedding many learning objectives in asingle simulation modality and tailor the learning
experience to the needs of each of the learners
involved while promoting interprofessional
education amongst a team of health professionals
and thus improve patient care. To allow for robust
and accurate touch-sensing positioning on thesurface with multiple users/participants as may berequired within the scope of a virtual medical
simulation, our approach will incorporate theCluster Pattern Interface (CLUSPI) invented by
Kanev and Kimura. CLUSPI-based technologies
will be employed to permit the customization of
the information presented to each of the users
depending on their level (i.e., novice, intermediate,or advanced) and to their actions within thesimulation. As described further in Section 4(“Future Work”), this particular work presents
some interesting technical challenges that must be
solved including the ability to present
individualized views to each of the users (ideallygreater than two) which will require some form ofuser tracking and incorporating a display with alarge refresh rate and may involve multiplexing of
the rendering using shutter glasses and rendering
the scene for each individual separately (work on
solving these problems has recently begun). An
overview of this work, including a detailed
scenario, was presented at the 15th International
Conference on Humans and Computers, (HC2012) February 11-12, 2013, in Hamamatsu, Japanby Professor Dubrowski.
Greater details
regarding this work including an overview of the
scenario and technical details are provided byDubrowski et al. (2013).
4. Future Work
During the visit to Shizuoka University byProfessors Jenkin, Kapralos, and Dubrowski in
February 2013, together with Professor Kanev,
they devised a plan to continue working on andmove forward with both research topics. With
respect to sound localization on a tabletop display,a research article outlining the results of the“ground truth experiment” is being prepared andwill be submitted to the 2013 ACM International
Conference on Interactive Tabletops and Surfaces(ITS’2012) to be held in St. Andrews Scotland,
October 6-9, 2013 (the full paper submission
deadline is June 14, 2013). In addition, a newexperiment was devised. Rather than presenting

participants with a virtual sound and then having
them determine the location on the surface, in thisnew experiment, participants are told the locationof the sound source on the surface and then theymust adjust (via a graphical user interface) the gain
applied to each of the four loudspeakers
surrounding the tabletop computer display so thatthe virtual sound source is perceived to them asemanating from the given location. The UOIT
student, Mr. Jonathan Lam, was hired to conduct
the experiment. Currently, Mr. Lam has developed
all the necessary software to support theexperiment and is in the process of testing thesoftware and hardware setup. It is anticipatedthat we will begin conducting the experiment inearly April after approval from the UOIT Research
Ethics Board has been granted. Although ourwork has focused primarily on spatial soundgeneration and sound localization, future work will
also examine the interaction of audio visual cues
and more specifically, we will examine whether
visual cues can be used to reduce some of theinherent error associated with localization of asound source on a horizontal surface. Kapralos,
Collins, and Dubrowski have conducted various
experiments that have examined audio-visualinteraction within multi-modal environments (seeCowan et al., 2013; Cullen et al., 2012; Rojas et al.,2012) providing a good base to begin from.
With respect to the interprofessional healthprofessional team training project, we are currently
in the process of further developing the scenariothat was originally described by Dubrowski et al.
(2013) at the HC 2012 conference in Hamamatsu,
Japan. In addition, we are currently focusing on
various technical issues and in the process ofdeveloping a method to allow multiple users(greater than two) situated around the table to view
the scene from their perspective and be presented
with individualized views. One approach we are
exploring is to present individualized monoscopic
views to the users (maximum of four users) bytaking advantage of the high refresh rates available
with current consumer televisions. Essentially,
employing shutter glasses (as employed in active
stereoscopic 3D viewing), during each refresh
cycle, each of the four shutter glasses will be
activated once and the other three will be closed
thus allowing the displays refresh to be
multiplexed across each of the four users (this willof course require a four-fold increase in renderingof the scene). We are aiming to have this setup
completed this spring and planning to prepare aresearch article describing our solution to the 2013
ACM International Conference on Interactive

Tabletops and Surfaces (ITS’2012). This work is
taking place both at Shizuoka University in Japanand at York University and UOIT in Canada whereseparate tabletop touch-screen displays areavailable.
Further exchanges are planned in order to support
the research. Professor Kanev plans extended visits
to York University, UOIT, and the Learning
Institute at the Hospital for Sick Children in thesummer and fall of 2013. Initial planning isunderway to prepare and submit a tutorial proposal
for spatial sound generation for tabletop
touchscreen displays at the 2013 ACM
International Conference on Interactive Tabletops
and Surfaces (ITS’2012) to be held by members of
this research team. Finally, we are also seeking
further funding from funding agencies in Canada
to support this work.
[4] Publications
1. B. Cowan, D. Rojas, B. Kapralos, K. Collins,
and A. Dubrowski. Spatial sound and its
effect on visual quality perception and task
performance within a virtual environment.
21st International Congress on Acoustics,
Montreal, Canada, June 2-7, 2013.
2. B. Cullen, D. Galperin, K. Collins, B.
Kapralos, and A. Hogue. The effects ofaudio on depth perception in S3D games.
Audio Mostly 2012, Corfu, Greece, September
26-28, 2012.
3. A. Dubrowski, B. Kapralos, M. Jenkin, and K.
Kanev. Collaborative, interactive smart-tablebased simulations for interprofessional
education?, The 15th International Conferenceon Humans and Computers (HC-2012),
Electronic Abstracts Collection (Hamamatsu
sessions), Paper No ID10, pp. 17. [11-12
February2013,
Aizu-Wakamatsu&
Hamamatsu, Japan; Düsseldorf, Germany].
4. M. Guadagnoli, M. P. Morin, and A.
Dubrowski. The application of the challenge
point framework in medical education.
Medical Education 46(5):447-53, 2012.
5. Lam, J., Kapralos, B., Collins, K, Hogue, A.,
Kanev, K., Jenkin, M., Sound Localization on
Table-top Computers: A Comparison of Two
Amplitude Panning Methods. ACM Computers
in Entertainment (to appear 2013).
6. D. Rojas, B. Kapralos, S. Cristancho, K.
Collins, A. Hogue, C. Conati, and A.
Dubrowski.
Developing effective serious
games: The effect of background sound on

visual fidelity perception with varying texture
resolution. Studies in Health Technology andInformatics, 173:386-392, 2012.
7. N. Matsuda, H. Kano, H. Taki, L. Cui, K. Seta,
and M. Ikeda. Reviewing Map to Represents
Tutor’s Intention for Nursing Report. The 15th

International Conference on Humans andComputers (HC-2012), Electronic Abstracts
Collection (Hamamatsu sessions), Paper No
ID05, pp. 7. [11-12 February 2013, AizuWakamatsu & Hamamatsu, Japan; Düsseldorf,
Germany].

「様式３」
Traveling Report
Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:

Noriyuki Matsuda
Faculty of Systems Engineering, Wakayama University, Japan
February 10, 2013 – February 12, 2013
Shizuoka University, JapanTo discuss the research and project organization, to schedule future work, and toparticipate at the 15th International Conference on Humans and Computers,
February 11-12, 2013.
Prof. Kamen Kanev
Michael Jenkin
York University, Toronto, Ontario, Canada.
February 7, 2013 – February 14, 2013
Shizuoka University, JapanTo discuss the research and project organization, to plan publications, and toparticipate and report obtained results at 15th International Conference on Humans
and Computers, February 11-12, 2013.
Prof. Kamen Kanev
Bill Kapralos
University of Ontario Institute of Technology, Oshawa, Ontario, Canada.
January 31, 2013 – February 13, 2013
Shizuoka University, JapanTo discuss the research and project organization, to plan publications, and toparticipate and report obtained results at the 15th International Conference on
Humans and Computers, February 11-12, 2013.
Prof. Kamen Kanev
Adam Dubrowski
The Learning Institute, Hospital for Sick Children, University of Toronto, Toronto,
Ontario, Canada.
February 6, 2013 – February 13, 2013
Shizuoka University, JapanTo discuss the research and project organization, to plan publications, and toparticipate and report obtained results at the 15th International Conference on
Humans and Computers, February 11-12, 2013.
Prof. Kamen Kanev