情報処理学会 インタラクション 2019
IPSJ Interaction 20193B-36
2019/3/6

Observing Semantic Human Expression in a Group
Meeting by using an Omni-directional Camera
Yusuke Soneda1

Stirapongsasuti Sopicha1

Yutaka Arakawa1

Keiichi Yasumoto1

Abstract: In Japan, according to a survey of a company, average of meeting time per a day is 68.2 minutes. To improve the productivity of a group meeting, we need to know and measure the quality of the meeting. On the other hand,
it is hard to record and analyze the conversation because of the reason of compliance. Therefore, we start considering
the camera-based group meeting assessment that utilizes Omni-directional camera. We can observe various kind of
facial expression and body gestures of attendees through the video. However, there is no standardized definition ofhuman expressions. So, in this paper, we try to define the semantic human expression observed in a group meeting.

1.

Introduction

A meeting is widely considered to be the most important keyof achievement at work. Some people, especially oﬃce workers, have to participate at least one meeting in a week. However,some meetings have a low eﬀectiveness due to various factors.
For example, some meetings are hold so long time which cancause some participants lacking interest of the meeting contents.
Also, participants’ contribution in a meeting may be not equal
because not all participants have the same chance to express theirideas. Therefore, it is necessary to develop a system to evaluate
a productivity of a meeting. To fulfill this requirement, the typeof meeting is the first issue which we should consider. Generally,
there are 2 main types of meeting i.e. face-to-face meeting andonline meeting.
In Japan, JR Tokai Agency [1] has surveyed about time andtypes of meeting hold in Japanese companies. The result showsthat 91.5% of companies hold face-to-face meetings and the average of meeting time per a day is 68.2 minutes. Because of thisresult, our research focused on the face-to-face meeting when human’s physical expression is the key point of a system development.
To clarify, we assumed that the quality of meeting can bemeasured by participants’ physical expressions such as smiling,laughing, speaking, nodding and etc. According to Peter’s study[2], nonverbal communication is important for social relationshipand shows an emotion. For instance, nodding has an explicit
meaning of positive feeling and makes people easily understandan agreement during conversation. Another assumption is if wedevelop a system to support participants having more contribution
in a meeting, the meeting time will be decreased due to participants’ brainstorms.
In this paper, we observed the semantic human expression using an omni-directional camera (360 degree camera).We setup an1Nara Institute of Science and Technology, Ikoma, Nara 630–0101, Japan© 2019 Information Processing Society of Japan

Fig. 1 OpenFace image

experiment of a meeting with 4 participants and applied the facialdetection algorithm called OpenFace to get the facial position andmapped them to the labeled data.
The paper is organized as follows. In the section 2 reviews
the related work. Section 3 introduces the camera-based groupmeeting assessment. Section 4 shows how to label data. The lastsection is the conclusion of this paper.

2.

Related Work

In order to evaluate a quality of meeting, some previous studies has been proposed. For example, Yu et al. [3] has proposeda smart meeting system using several cameras setting up in frontof individual participants to record and detect who is talking towhom. They also constructed a meeting browser to provide otherusers understanding the meeting contents as a real time. Although
the proposed method is interesting, but deployment is diﬃcult toapply in the real scenario due to the number of cameras based onparticipants.
Recently, NTT Communication [4] has developed the system
which can analyze the group meeting in real time. The proposed method used 8-channel microphone array with an omnidirectional camera to record a meeting and analyze a meeting byspeech recognition and visual processing. The result showed thatNTT can achieve 80% of spoken words within 3 seconds. However, their system is suﬀered from high cost equipment. More-850

情報処理学会 インタラクション 2019
IPSJ Interaction 20193B-36
2019/3/6

Table 1 definition of facial expressions
label
smilinglaughing
speaking

noddinggazingFig. 2

Fig. 3

Image of meeting

definition
Raise mouse corners.According to Peter’s study[2],smiling is eﬀective way to close interpersonal distanceSmiling with voice. We tried to predict smiling andlaughing as a diﬀerent one.
Say opinions or something. We do not use sound information to define speaking because the topic of meetingis privacy information. We used only mouse movement
data to detect speaking.
Move head up and down. Nodding influences positive
impression for speaker.
To look a participant. Eye contact expresses agreement
for speaker or insists on listening to discussion.

used OpenFace.But body language is very useful for analyzing
meeting too. Arakawa’s research suggested some gestures showemotion[7].openpose[8] can analyze body movement like OpenFace.
When we finish making a model which can detect the important gesture in the meeting, we will apply the model real-time. Ifwe show the gesture of the meeting real time, there is possibilityof behavior change.

OpenFace apply image for meeting

over, the main issue is how to deal with the privacy concern because microphones recorded contents of a meeting.
Obviously, designing such a system has a lot of issues to deliberate such as how to select commercial equipment, how to annotate participants’ physical expression. To solves these issues, weoutline the method in the following section.

3.

Camera-based Group Meeting Assessment

3.1

Meeting recording system by using Omni-directional
cameraWe use a omni-directional camera, RICOH THETA S[5], andMacbook Pro to record the meeting. RICOH THETA S has twolens, so it can record 360 degree images. The resolution recorded
by THETA S is 1920 × 1080 and number of frames for recording
is 30 fps. For detecting the facial expression, we utilize OpenFace[6], Fig. 1 shows the example of output image.
3.2 Data collection
We assume that participants of meeting are 4 people, and meeting time is 10 min. They are same grade and know each otherwell. We put it with tripod and participants sitting around theomni-directional camera.

4.

Some participants said ”I concerned omni-directional camera
because the height of the camera is the same as the height of theline of sight”.We will think about optimum equipment again.
References
[1]
[2]
[3]
[4]

Labeling

We can analyze the faces of multiple persons simultaneously
using omni-directional camera, we think that we can analyze not
only the information from the face but also the influence of facial
expressions in the future.
I described the facial expressions and attitudes identified in Table 1.

5.

Fig. 4 Real time analyze meeting

[5]
[6]
[7]
[8]

JR Tokai Agency. ビジネスパーソンの「社内会議」に関する調査. ビ
ジネスパーソンウォッチング調査 (in Japanese), 2017.
Peter E Bull. Posture and Gesture. Pergamon Press, 1987.
Zhiwen Yu, Motoyuki Ozeki, Yohsuke Fujii, and Yuichi Nakamura. Towards smart meeting: enabling technologies and a real-world application. In ICMI, 2007.
T. Hori, S. Araki, T. Yoshioka, M. Fujimoto, S. Watanabe, T. Oba,
A. Ogawa, K. Otsuka, D. Mikami, K. Kinoshita, T. Nakatani, A. Nakamura, and J. Yamato. Low-latency real-time meeting recognition andunderstanding using distant microphones and omni-directional camera. IEEE Transactions on Audio, Speech, and Language Processing,20(2):499–513, 2012.
Ricoh theta s. https://theta360.com/ja/about/theta/s.html.
Openface. https://github.com/TadasBaltrusaitis/OpenFace.
Ayumu Arakawa. Naoto Suzuki. しぐさと感情の関係の探索的研究.
The Japanese Journal of Research on Emotion, 10(2):56–64, 2004.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser
Sheikh. OpenPose: realtime multi-person 2D pose estimation using PartAﬃnity Fields. In arXiv preprint arXiv:1812.08008, 2018.

Conclusion

In this paper, we defined the labeling of the meeting to evaluate quality of meeting. We focused on the face movement so we© 2019 Information Processing Society of Japan851