課題番号P-7

Multimedia Interaction Interfaces
in Collaborative E-learning Environments[1] Organization
Leader: Noriyuki Matsuda (Faculty of SystemsEngineering, Wakayama University, Japan)
Representatives at RIE: Kamen Kanev & Hiroshi
Inokawa (Research Institute of Electronics,Shizuoka University)
Participants:
Hirokazu Taki (Faculty of Systems Engineering,Wakayama University, Japan)
Hidetoshi Nonaka (Graduate School ofInformation
ScienceandTechnology,Hokkaido University, Japan)
Julita Vassileva (University of Saskatchewan,Canada)
Michael Jenkin (Department of Computer Scienceand Engineering, Faculty of Engineering, YorkUniversity, Canada)
Magdalina Todorova (Department of ComputerScience, Faculty of Mathematics andInformatics, Sofia University, Bulgaria)
Karen Collins (Canadian Centre of Arts andTechnology at the University of Waterloo,Canada)Bill Kapralos (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Andrew Hogue (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Sanshiro Sakai (Department of Computer Science,Faculty of Informatics, Shizuoka University)[2] Progress of the ResearchThis cooperative research project explores
possibilities for combining expertise fromdifferent research and fostering the design anddevelopment of innovative interfaces andenvironments for e-learning based on naturalinteraction paradigms that are adjustable to theneeds of specific user groups. Smart tablecomputer interactions have been identified as amain vehicle of this project and sound interface
design for smart tables was addressed in (1) whereusers position themselves around a horizontal
computer screen and a combined audio/visionbased interface is employed for sound source anduser tracking in real-time.
We are currently focusing on the application ofhorizontal versus vertical-screen placement forcollaborative activity and our research includes thefollowing four areas: i) the use of video games to
conduct usability tests on several spatial sound(amplitude panning) techniques and thus examine
their effectiveness, ii) the collection of reference
data for audio-visual table top displays, iii)
incorporating visual cues in audio-visualinteractive displays and examining the interactionof audio and visual cues in enhancing the userexperience, and iv) localization (head tracking) for
auralization in acoustic tabletop displays.
In two experiments conducted by the CanadaJapan research team, virtual sound sources werespatialized over a tabletop display surface using abi-linear interpolation (see [3]). In the first of thetwo experiments, four loudspeakers were arranged
one at each at one of the four table corners in astandard
Quadraphonicarrangement(theloudspeakers faced inwards towards the surface ofthe table). Sounds were spatialized to a location on
the surface of the table using a bi-linear
interpolation amplitude panning method whereby
sound was panned between loudspeaker pairs (see[3]). For each trial of the experiment, the soundwas spatialized to correspond to one of 25 predefined locations on the surface and theparticipants' task was to localize the sound.
Results of the experiment revealed that the method
is prone to varying error across individuals
particularly for the virtual sound source positions
that are closest to the participant (user). For thepositions resulting in the largest error (thoseclosest to the participant), it was hypothesized thatthe error was due to the fact that the twoloudspeakers were facing away from theparticipants. The second experiment wasconducted to test this hypothesis and was similar
to the first experiment except that theloudspeakers were “flipped” upwards so that theyemanated sound in an “upwards” direction thus
ensuring that there are no loudspeakers facing
away from the participants. However, the resultsof the second experiment did not support thehypothesis.
A series of further experiments are scheduled tocollect “ground-truth” data with respect to soundsource localization on a surface and such data willallow us to compare the results of our previous
experiments and draw meaningful conclusions (forexample, if the “ground-truth” experiments reveal

that a large sound source localization error then it
may not be worth pursuing other amplitude
panning methods and techniques). To gather the“ground-truth” data, small (physical) loudspeakers
will are placed facing upwards at each of the 25
virtual sound source locations examined in theprevious experiments. The experiment will
involve participants localizing sound source
positions but rather than using amplitude panningto spatial a sound to one of the 25 virtual soundsource positions, a sound will be out from thecorresponding loudspeaker instead.
Since thesound will emanate directly from the position ofinterest via an actual loudspeaker at that position,
any errors cannot be a result of the amplitude
panning method. This will allow us to quantify theerrors that the previous two experiments revealed.
Construction of the physical apparatus isconducted in Japan, led by Dr. Kanev andsupported by Canadian visiting researcher Mr.
Hodge.
An application for an “Invitation
Fellowship Programs for Research in Japan”(offered by the Japan Society for the Promotion ofScience) has also been submitted by Canadian
researcher Dr. Michael Jenkin. If awarded, Dr.
Jenkin will visit Shizuoka University for one
month in the summer to further support theongoing “ground truth” experiments. On theCanadian side, Dr. Karen Collins and Dr. Bill
Kapralos are developing a number of audio-only
video games that can be used to examine theeffectiveness (through user-based experiments) of
the previously developed amplitude panningtechniques for the table-computer. Experiments
will be carried out in both Canada and Japan.
[3] Results(3.1) Research results
The interaction between sound and image plays animportant role in instructional software. For
instance, multimodal information (i.e. sound andimage) can increase or decrease cognitive load,depending on whether or not information isredundant. Furthermore, it is known that
presenting information in multi-modes (when they
do not conflict with each other) helps a user toovercome misinterpretations, improve memory,and respond more quickly. But new technologies
present new problems when it comes to the use ofmultiple modalities. We are exploring ways (see1-4) to test emergent and disruptive technologies
that we anticipate will be particularly important in
future learning experiences (i.e. smart tables).
To enhance this research, two other grants(through Canadian organizations) were applied for,including one large-scale Social Sciences andHumanities Research Council of Canada,Partnership Grant that would include members
from around the world in a larger research
network to explore the role of new technology on
immersion and learning.
(3.2) Future work
We continue to add more members to our projectto enhance our research, and to report obtained
results at international conferences on a regularbasis. We have discussed arranging for exchange
students between Japan and Canada to train thesestudents in cooperative research projects and incommunication skills in addition to the specificsof the technology, methods, and practice. Dr.
Collins has already sent one student over for one
month, and we anticipate further exchange.
We are currently revising our research plan basedon our preliminary findings, including exploringnew methodological approaches to the integrationof sound and image.
Our next stage is to explore the ability to spatialize
sound in relationship to an image on a horizontal
and vertical screen. We have already determined
how to run experiments, and we are in the processof building appropriate screens.
[4] Publications(1) Collins, K., Kapralos, B., Kanev, K., Smart
Table Computer Interaction Interfaces with
Integrated Sound, The Journal of Three
Dimensional Images, Vol. 24, No. 3, 2010, pp.
58-67.
(2) Collins, K., Kanev, K., Kapralos, B., Using
Games as a Method of Evaluation of Usability
and User Experience in Human-Computer
Interaction Design, In Proceedings of the 13thInt. Conf. on Humans and Computers HC’10,
Aizu-Wakamatsu, Japan, December 8-10,
2010, pp.5-10.
(3) Lam, J., Kapralos, B., Collins, K., Hogue, A.,Kanev, K., Amplitude Panning-Based Sound
System for a Horizontal Surface Computer: A
User-Based Study, In Proceedings of theInternational Symposium on Haptic AudioVisual Environments and Games, Phoenix,
AZ. USA, October 16-17, 2010, pp.174-178.
(4) Collins, K., Kapralos, B., Hogue, A., Kanev,
K., An Exploration of Distributed Mobile
Audio and Games. In Proceedings of the ACM
FuturePlay 2010 Int. Conf. on the Future ofGame Design and Technology, Vancouver,
British Columbia, Canada, May 6-8, 2010,
pp.253-254.

Traveling report
Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:

Karen Collins
Canadian Centre of Arts and Technology at the University of Waterloo, Canada
December 3, 2010 – December 12, 2010
Shizuoka University, Japan(Please see the description in the bottom of the page)Prof. Kamen Kanev

Name:
Affiliation:

Bill Kapralos
Faculty of Business and Information Technology, University of Ontario Institute ofTechnology, Canada
December 3, 2010 – December 12, 2010
Shizuoka University, Japan(Please see the description in the bottom of the page)Prof. Kamen Kanev

Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:

Hidetoshi Nonaka
Hokkaido University, Japan
February 20, 2011 – February 23, 2011
Shizuoka University, JapanTo discuss the research and project organization, to schedule future work, and to
make a presentation entitled “Prototyping of Non-planar Multi-touch Interface”.
Prof. Kamen Kanev
Michael Jenkin
York University, Canada
March 1, 2011 – March 8, 2011
Shizuoka University, JapanTo discuss the research and project organization, to plan publications, and toparticipate in a collaborative research seminar organized at RIE with a presentation
entitled “Building and Interacting with Models of Complex 3D Environments”
Prof. Kamen Kanev
Alexander Hodge
University of Waterloo, CanadaFebruary 1, 2011 – March 11, 2011
Shizuoka University, JapanTo build a smart table for research experiments and to further address various
hardware and software issues.
Prof. Kamen Kanev

The purpose of the visit of Dr. Collins, Dr. Kapralos (December 2010), and Dr. Jenkin (March 2011) was
three-fold: For Dr. Karen Collins and Dr. Bill Kapralos to attend the Humans and Computers 2010 conference
and present two research papers including a paper outlining the research being conducted by Dr.
Karen Collins, Dr. Bill Kapralos (Canada) and Dr. Kamen Kanev (see [2]). In addition, Dr. Collins
provided a keynote address/presentation during at the start of the conference.
 To further build upon and expand the research collaborations between the University of Waterloo, theUniversity of Ontario Institute of Technology (UOIT), and York University which are in Ontario,
Canada and the Research Institute of Electronics, Shizuoka University, Japan. This included i)developing a research plan, ii) discussing the details of future experiments to be conducted both inJapan and Canada, and iii) discussing future grant opportunities. Funding opportunities included
working on a Notice of Intent for a large Social Sciences and Humanities Research Council of Canada
Partnership grant (led by Dr. Karen Collins and includes Dr. Kamen Kanev and Dr. Bill Kapralos
amongst other investigators), and also on an application for an “Invitation Fellowship Programs for
Research in Japan” (offered by the Japan Society for the Promotion of Science) by Canadian
researcher Dr. Michael Jenkin (this was submitted January 2011 and if awarded, Dr. Jenkin will visit
Shizuoka University once again for one month in the summer of 2011).
To discuss potential collaborations beyond research and more specifically, the development of a
student exchange program between Shizuoka University and the university of Waterloo and UOIT.
To facilitate this, Dr. Kanev has applied for Adjunct Professor status at UOIT. This will allow him toco-supervise Canadian (UOIT) students and makes the process of establishing formal university
relations (exchange programs) easier.