「様式３」課題番号 ：P-15

Multimedia Interaction Interfaces
in Collaborative E-learning Environments[1] Organization
Leader: Noriyuki Matsuda (Faculty of SystemsEngineering, Wakayama University, Japan)
Representatives at RIE: Kamen Kanev & Hiroshi
Inokawa (Research Institute of Electronics,Shizuoka University)
Participants:
Hirokazu Taki (Faculty of Systems Engineering,Wakayama University, Japan)
Hidetoshi Nonaka (Graduate SchoolofInformation
ScienceandTechnology,
Hokkaido University, Japan)
Julita Vassileva (University of Saskatchewan,Canada)
Michael Jenkin (Department of Computer Scienceand Engineering, Faculty of Engineering, YorkUniversity, Canada)
Magdalina Todorova (Department of Computer
Science, Faculty of Mathematics andInformatics, Sofia University, Bulgaria)
Volodymyr Gnatyuk (V.E. Lashkaryov Institute ofSemiconductor Physics, National Academy ofSciences of Ukraine)
Karen Collins (Canadian Centre of Arts andTechnology at the University of Waterloo,Canada)Bill Kapralos (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Andrew Hogue (Faculty of Business andInformation Technology, University of Ontario
Institute of Technology, Canada)
Boris Brimkov (Department of Mathematics,University at Buffalo)
Sanshiro Sakai (Department of Computer Science,
Faculty of Informatics, Shizuoka University)[2] Progress of the researchThis cooperative research project explores
possibilities for combining expertise from different
research and fostering the design and developmentof innovative interfaces and environments for elearning based on natural interaction paradigms
that are adjustable to the needs of specific usergroups. Smart table computer interactions havebeen identified as a main vehicle of this projectand sound interface design for smart tables is being
addressed where users position themselves arounda horizontal computer screen and a combined
audio/vision-based interface is employed for soundsource and user tracking in real-time.
Tabletop displays are a relatively new display andinteraction technology in which a number of users
interact with a large display surface that is placed
horizontally. Such devices have already foundlimited use as general purpose displays, gaming
devices, and have been proposed for a range ofother applications including their use as control
panels for industrial processes. Tabletop displays
present a large number of interesting technical andoperational questions that must be addressed if theunderlying technology is to make inroads into
serious applications such as their use in industrial
operations and emergency planning and response.
One critical question that must be addressed inorder to fully operationalize tabletop displays is
the user’s ability to integrate multimodal display
cues within the device. Of particular interest in thisproject has been the ability of the user to integrate
spatialized visual and audio cues into a coherent
perception of a spatialized event. To take a specificexample of this, suppose that a warning display is
to be presented on a tabletop display (indicatingsome off-nominal condition in an industrial plant,for example). In a physical embodiment of such adisplay a warning buzzer would be located withthe display and when the off nominal condition
occurs the display would indicate the condition
and the buzzer would sound to alert the operator ofthe event. When this physical display is renderedon a tabletop console it is necessary to provide
both the visual display and also a localized audio
cue that directs the operator’s attention to theappropriate visual console. The fundamental
question then becomes, how does one generate aspatialized audio cue within the constraints of atabletop display?
In answering this question it is instructive to lookat how this spatialization is accomplished in amovie theatre with a pair of speakers. Theaudience sits in front of a vertical display withspeakers presented to the left and right of thescreen. When an event occurs on screen it is notpossible to generate a sound at the exact visual
location of the event (there is no speaker there),rather sounds are mixed through the two speakers
to as to generate a sound pattern that is perceived
as though it is at the location of the visual event.
This is a well studied problem and a large number

of solutions have been proposed and the basic
approach is easily generalized to a collection ofloudspeakers that surrounds the listeners. The
tabletop display environment presents a number ofinteresting challenges to the tradition theatre-likeaudio display environment.
First, the users surround the display rather than all
being located on one side of it. This provides aradically different set of constraints in terms ofsound field generation and also opens up someinteresting questions about the location of thespeaker array. Should the speakers be locatedaround the table “in front” of the various
participants, or should they be positioned outsideof the circle of potential operator locations?
Neither solution seems particularly attractive as in
the first listeners are very close to speakers whilein the second the sound sources are behind some of
the listeners. Second, the display surface ishorizontal, rather than vertical. This changes thegeometry substantively in terms of the direction ofviewing and listening. In the movie theatre you
hold your head erect and sounds should appear to
emanate from a vertical surface that is orthogonal
to the viewing direction. In the tabletop
environment the surface is oblique to the viewing
direction. Third, there is the question of the effectof speaker distance on any mixing strategy. In thetheatre model all speakers are the same distancefrom the listener. In the tabletop version this is
likely to no longer be true, especially for twodimensional spatialization over the tabletop
surface. How does this change in speaker distances
influence the various mixing models. Finally, andperhaps most challenging, is how to deal with thefact that all of the listeners should perceive thesame synthetic sound source location. In thetheatre application this task is simplified by having
all of the speakers on one side of the display. For
the tabletop display however, this is not the caseleading to a number of issues in terms ofsynthesizing spatialized audio.
A number of research exchanges took place during
the year. Prof. Kanev undertook two trips toOntario to visit UOIT, University of Waterloo andYork University in 2011. The first trip in thesummer was to set up the basic parameters for amore extended visit throughout the fall and winter.
During this second visit an experimental apparatus(see results, below) was constructed at York
University and installed at UOIT. Prof. Jenkin,funded by a JSPS Short Term Fellowship spentmuch of February and March 2012 at Shizuoka
University. Andrew Speers, a student of Prof.
Jenkin also visited Shizuoka in March in order topresent some of his research results and to observe
the operation of the experimental hardware so that
he can utilize a similar system in Canada upon hisreturn.
[3] Results(3.1) Research results
During this last year work within the project hasconcentrated on understanding how humans
perceive spatialized audio cues in a tabletop
environment and how traditional mixing
algorithms for vertical displays must be adapted
for the generation of effective spatialized audio
cues. Experiments at Shizuoka University and
UOIT have begun to explore the ability of humans
to localize audio cues in a tabletop environment
versus their ability to localize such cues in atheatre-like environment. At Shizuoka University
this work has lead to the construction of a novel
5x5 speaker array that can be used to generate true
sounds at specific locations while at UOIT a 2D
mockup surface has been constructed that allows asingle sound source to be positioned out of thesight of the user over the entire table surface.
Experiments have demonstrated that humans makeconsistent errors in sound source localization fortable top displays (2). Left-right localization isquite good although forward-backward localizationshows systematic errors with sounds being pulled‘towards’ the listener. Although the nature of thispulling is unknown it is an open question if thiseffect is found in the vertical plane. The
experimental apparatus constructed at UOIT isbeing used to answer this question now.
The question of how best to synthesize different
virtual sound source locations is considered in (6).
Two traditional stereo panning approaches wereevaluated in the tabletop environment setting.
Systematic errors were identified using bothtechniques and ongoing work is exploring how toimprove on these previous mixing approaches.
Other work within the project has investigateddifferent educational frameworks (3) andtechnologies for interaction localization on atabletop display using marked surfaces. (1) and (4)describe various approaches to invisibly mark an
interaction surface or object so that the marks
provide a map for decoding later interactions withthe surface. (1) demonstrates the approach
generalized to non-planar surfaces which is ofparticular interest in the application of tabletop
display technology in industrial plant displays
where nonlinear (curved) surfaces may be an
appropriate surface upon which to mount any
display. (5) describes recent work in theacquisition of 3D models for display in tabletop
settings. During the trip of Andrew Speers to Japana number of 3D external structures were captured
suitable for later display on tabletop surfaces(3.2) Future work
During Prof. Jenkin’s stay at Shizuoka University
an experimental protocol was established formeasuring systematic errors in sound source
localization using a pair of speakers. The
experimental apparatus has been assembled andsoftware written to allow the experiments to beconducted. A student is currently collecting data
and we expect to be able to report the results ofthis experiment shortly. The experimental setup is
being replicated in Canada although with someslight differences. The setup at Shizuoka
University is designed to work with the existing
speaker array system built there. The test will be tocompare a real sound from a number of different
locations with a spatialized signal within a
psychophysical testing regime. The system at York
University will utilize the active display surface
which will display a localized visual cue on thetabletop with a similar acoustic panning systemand the same experimental protocol.
Further exchanges are planned in order to support
the research. Prof. Kanev plans to visit York
University and UOIT in the fall of 2012, and Prof.
Kapralos is planning to visit Shizuoka in the springof 2013. Initial planning is underway for a joint
graduate course in tabletop display technology totake place in Japan in 2013. In 2011, anapplication for a Social Sciences and Humanities
Research Council Partnership Grant was submitted
and we are awaiting results. It is planned that infall of 2012 we will apply for a Natural Sciencesand Engineering Research Council grant with Dr.
Collins as Principle Investigator.

[4] Publications(1) Kanev, K., Mizeikis, V., Gnatyuk, V.,
Localization Encoding in the Bulk of PhysicalObjects by Laser-Induced Damage, In
Proceedings of The Joint Int. Conf. on HumanCentered Computer Environments HCCE2012, Aizu-Wakamatsu, Japan, March 8-13,
2012, pp.93-98.
(2) Nakano, D., Lam, J., Kapralos, B., Kanev,K., Collins, K., Hogue, A., Jenkin, M., A
Frameworkfor
Sound
Localization
Experiments and Automation, In Proceedingsof The Joint Int. Conf. on Human-Centered
Computer Environments HCCE 2012, AizuWakamatsu, Japan, March 8-13, 2012, pp.137144.
(3) Todorova, M., Kanev, K., Educational
Framework for Verification of ObjectOriented Programs, In Proceedings of The
Joint Int. Conf. on Human-Centered Computer
Environments HCCE 2012, Aizu-Wakamatsu,
Japan, March 8-13, 2012, pp. 23-27.
(4) Gnatyuk, V., Kanev, K., Gagarsky, S.,
Features of Transparent Material Marking withNano- and Subnanosecond Laser Pulses, The
9th Int. Conf. on Global Research andEducation InterAcademia2011,
Sicevita,Romania, September 26-29, 2011, pp.26.
(5) Speers, A., Jenkin, M., Tuning Stereo Image
Matching with Stereo Video Sequence
Processing, In Proceedings of The Joint Int.
Conf.
onHuman-CenteredComputer
Environments HCCE 2012, Aizu-Wakamatsu,Japan, March 8-13, 2012, pp.208-214.
(6) Lam, J., Kapralos, B., Collins, K, Hogue,A., Kanev, K., Jenkin, M., Sound Localization
on Table-top Computers: A Comparison ofTwo Amplitude Panning Methods. (acceptedforpublicationbyComputersinEntertainment)「様式３」
Traveling Report
Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:Name:
Affiliation:
Period of time:
Destination:
Purpose:Name of receiver:

Noriyuki Matsuda
Faculty of Systems Engineering, Wakayama University, Japan
March 10, 2012 – March 11, 2012
Shizuoka University, JapanTo discuss the research and project organization, to schedule future work, and toparticipate in The Joint International Conference on Human-Centered Computer
Environments HCCE 2012.
Prof. Kamen Kanev
Volodymyr Gnatyuk
V.E. Lashkaryov Institute of Semiconductor Physics, National Academy of Sciences,
UkraineMarch 10, 2012 – March 16, 2012
Shizuoka University, JapanTo discuss continuing cooperative research, to plan publications, and to participate
and report obtained results at The Joint International Conference on Human-Centered
Computer Environments HCCE 2012.
Prof. Kamen Kanev