International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2016)
D. Reiners, D. Iwai, and F. Steinicke (Editors)

Superimposing Questioner on Presentation Screen Using
Microphone with Whole-Sky Camera
Yuki Kobayashi1 and Kenji Funahashi1
1 Nagoya Institute of Technology

Abstract
Academic conferences are usually held at a large hall. It is difficult that audiences watch the facial expression and gesture ofa presenter there. Sometimes the presenter is shot by video camera and projected onto another screen, but it involves costs. At
the same time someone usually asks a question there. However conference site is large as mentioned above, it is also difficultthat audiences watch the questioner. Of course a conference staff can shoot the questioner and projects onto another screen, butadditional manpower is required. Although it is not easy to install a camera at a conference site, a questioner generally uses amicrophone. Then we consider projecting a questioner onto a screen using the camera built-in a microphone. In this paper we
describe the pilot system to verify the effectiveness of this idea.
Categories and Subject Descriptors (according to ACM CCS): H.4.3 [Computer Graphics]: INFORMATION SYSTEMS APPLICATIONS/Communications Applications—Computer conferencing, teleconferencing, and videoconferencing

1. Introduction
Academic conferences are usually held at a large hall. It is difficult that audiences watch the facial expression and gesture of apresenter there. Sometimes the presenter is shot by video cameraand projected onto another screen, but it involves costs. We alreadyhave developed a presentation support system which superimposesa presenter onto the screen displayed a presentation slide [FN14].
At the same time someone usually asks a question there. However conference site is large as mentioned above, it is also difficult that audiences watch the questioner. Of course a conference
staff can shoot the questioner and projects onto another screen, but
additional manpower is required. There are related studies for thisproblem. One is that a fisheye camera and a lot of microphone areset on the ceiling and a speaker position is estimated using microphone levels [SKY∗ 04]. Another is that infrared ray emitter is installed in a microphone and a camera is paned and zoomed for aspeaker using the infrared ray irradiation [SMK08]. Although it is
not easy to install a camera at a conference site, a questioner generally uses a microphone. Then we consider projecting a questioner
onto a screen using the camera built-in a microphone. In this paper
we describe the pilot system to verify the effectiveness of this idea.

2. Extract Questioner with Whole-Sky Camera
You fundamentally use a microphone near the face, and in unspecified direction. So we tried to install two fish-eye cameras on a microphone, as one is back to back with another one. In this study wec 2016 The Author(s)Eurographics Proceedings c 2016 The Eurographics Association.

used RICOH THETA S that has two fish-eye cameras as a wholesky camera, and has a microphone for videography. Figure 1 showsthe example image that is taken by the camera. After converting
one hemisphere image of figure 1 to figure 2, a person is extracted
by face recognition. We use following equations for the conversion;
X −w)
w
2
x= r+ ,
2X−w
Y−h22
D2 + () +()22Y −hr()
h2+ ,y= r
2X −w 2Y −h 2
D2 + () +()22r((1)(2)where x and y are the coordinate of the pixel before conversion, Xand Y are the coordinate of the pixel after conversion, w and h are
the image width and height, D is the distance from the lens to theprojection surface, and r is the radius (r = w/2 = h/2). Incidentally the questioner is not always in only one hemisphere image. So
any directional hemisphere image is generated from whole spherical image. First the axes x, y and z are defined as shown in Figure 3. We describe y axis rotation by 90 degrees for example. The
left semicircle of left hemisphere and the right semicircle of righthemisphere are integrated to one hemisphere as a circle (Figure 4).
The RGB value at (y, z) in Figure 4 is calculated by the RGB value
at (x, y) in Figure 3, where assuming the point (x, y, z) is projectedby equidistant projection method. This is proceeded by 30 degree

Yuki Kobayashi & Kenji Funahashi / Superimposing Questioner on Presentation Screen Using Microphone with Whole-Sky Camera

Figure 1: Captured image

Figure 2: Converted image

Figure 3: Captured image (2)Figure 4: Rotated image

Figure 6: Result (horizontal 90 degrees)

Figure 7: Result (vertical 60 degrees)

Figure 5: Result (0 degree rotation)

step for a person extraction. When some people are extracted, thelargest face area is determined as the questioner.

Figure 8: Situation of figure 7

3. Experiment and Result
We used PC and whole-sky camera; RICOH THETA S to develop
the system we proposed. It should be noted that the camera wasconnected to PC with USB cable for the experiment. Figure 5
shows the captured image and converted image at the situation
where one lens of the camera was just in front of the face. Figure 6 shows the situation where the microphone (camera) was justupright and in front of the face, but two lenses were oriented 90
degrees from the face. Figure 7 shows the situation as shown in
Figure 8; the microphone was tilted, and a part of head was shotby one lens and other part of face was shot by another lens. The
presentation screen image is shown in Figure 9.

Figure 9: Screen image

References
4. Conclusion
In this paper we proposed new approach to show a questioner toaudiences. The questioner uses camera-attached microphone and issuperimposed onto a presentation screen. It is expected to accelerated their communication since the presenter can see the questioner,
and also audiences understanding. As future tasks, we should havean experiment in the large hall to verify the effect of this system.
The microphone should be also configured with wireless network.
We would like to estimate the front face image from the face that islooked up.

[FN14] F UNAHASHI K., N AKAE Y.: Getting yourself superimposed on apresentation screen. Proceedings of the 2nd ACM symposium on Spatial
user interaction (Sep 2014), 138–138. 1[SKY∗ 04] S ATOSHI N., K AZUHIDE H., Y OSHINARI K., K OH K.,
M ICHIHIKO M.: Audio and visual information integration for speaker’s
localization in automatic shooting of lecture (in japanese). IEEJ Trans.
EIS 124, 3 (Jun 2004), 729–739. 1[SMK08] S ATOSHI I., M ASAKI F., K AZUYA T.: Tracking questioner
with infrared-ray-emitting microphone (in japanese). IPSJ journal 49, 1(Jan 2008), 141–149. 1c 2016 The Author(s)Eurographics Proceedings c 2016 The Eurographics Association.