CHI 2005 | Late Breaking Results: Posters

April 2-7 | Portland, Oregon, USA

What is Connected by Mutual Gaze? — User’s Behavior
in Video-mediated Communication —Naoki Mukawa, Tsugumi Oka, Kumiko Arai and Masahide Yuasa
School of Information Environment, Tokyo Denki University
2-1200 Muzai-Gakuendai, Inzai, Chiba, 270-1382 Japan{mukawa@,01ie020@ed.,01id002@ed.,yuasa@}sie.dendai.ac.jp
ABSTRACTlaboratory systems have been designed to establish mutualgaze [4,7,8,10]. However, few systems in the marketplace
support eye contact.

Video-mediatedcommunicationsystems
suchasteleconferencing and videophone have become popular. Aswith face-to-face communication, non-verbal cues such asgaze, facial expression, head orientation and gestures invisual systems play an important role. Existing systems,
however, do not support mutual gaze because the lay-out of
the camera and monitor is restricted. Thus, conversations
using visual systems differ from those in face-to-facecommunication. This paper clarifies the problems of thevideo-mediated system, specifically for comparing thesystem with communication using eye-contact and with
communication using no-eye-contact. This study focuses on
the protocol of opening communication, e.g. establishment ofa visual-audio link, person identification and confirmation of
the acceptance of conversation. We conducted experiments
using the two systems. Analysis of recorded video sequences
revealed that the system using communication with eyecontact induced behavior similar to the system using face-toface communication.

Psychologists have revealed the importance of gaze in faceto-face communication, where the human gaze indicates
turn-taking cues in conversation, projects the sense ofintimacy between persons and also represents the focus ofattention [2]. As for existing video-mediated communication
systems that do not support eye contact, Short et al. claimthat the system often misinterprets conversation situations.
They point out that these systems may convey distorted gazesthat differ from the case of face-to-face and of eye-contact
systems. Sending a distorted gaze is sometimes much worse
than sending no visual information at all (like a telephone
conversation). Unfortunately, there are no precise evaluation
reports that compare the two systems, with/without eye
contact from both a psychological and an engineering pointof view.
Three approaches can be applied to an evaluation of thesystems. One is the cognitive psychological approach. Cheninvestigated the sensitivity of the discrepancy of gaze when
the user’s gaze direction shifts from ideal eye contact, andshowed that spatial sensitivity is not symmetric [5]. His
results present a design basis for a practical system ofestablishing eye contact, but he does not compare the twosystems. The second approach is based on questionnaires
about usability [9]. Many results show that a system thatprovides eye contact is preferable to a non-eye-contact
system. However, Grayson et al. point out that even non-eyecontact systems convey some eye contact information if thelay-out of camera and display is carefully chosen. Theirexperiments show that, after some experience of the system,
users can interpret their partner’s gaze as looking at themeven though the gaze is distorted [3]. Their results suggestthat a questionnaire-based approach alone by asking users’
preferences does not provide a solid basis for evaluating thesystems.

Author Keywords

Video-mediated communication,
ethenographical approachgaze,eyecontact,

ACM Classification Keywords

H5.3. Information interfaces and presentation (e.g., HCI):
Group and Organization Interface.
INTRODUCTION

Face-to-face communication is the most important type ofhuman communication style. Human beings exchange words,
facial expressions, gestures, postures and gazes that enrichmutual understanding. To expand human communication tobridge the spatio-temporal gap, electronic communication
media have been developed, and are gaining widespread use
in the community. Video-mediated communication systems
such as videophone (wired or mobile) and videoconferencing have become popular as well as existing
telephone and e-mail[1].

Still another approach is the ethnographic methodology thatanalyzes users’ behavior throughout their conversing
interactions. By analyzing the users’ conscious andunconscious behaviors that are considered to compensate thelack or the distortion of gaze, the usability of the system canbe evaluated from a number of aspects. Even if users areaccustomed to using the systems and can converse fluently

Researchers have pointed out that video-mediatedcommunication systems need to facilitate mutual gaze
between users [9] as in face-to-face communication. SomeCopyright is held by the author/owner(s).
CHI 2005, April 2–7, 2005, Portland, Oregon, USA.
ACM 1-59593-002-7/05/0004.

1677

CHI 2005 | Late Breaking Results: Posters

April 2-7 | Portland, Oregon, USA

after experience, this approach can clarify that the systems
are affected by distorted gaze. However, there are few
research works that compare the eye contact system with the
non-eye-contact system based on the approach..

Communication media
StepThis paper clarifies the feasibility of video-mediated systems,and compares the eye-contact system with the non-eyecontact system by observing users’ verbal and non-verbal
behaviors.

Audiovisuallink

Whatever communication style is chosen, the communication
session can be divided into three stages: opening,maintaining and closing. Almost all human-to-humancommunication requires a specific protocol for starting a
conversation such as the initial finding-out about each other,and salutation. So far, there are few research results on theopening stage of the visual communication link. By
comparing
existing
face-to-faceandtelephonecommunication, our study will clarify the characteristics ofeye-contact and non-eye-contact video-mediated systems.

Personidentification

PROTOCOL OF ESTABLISHING VIDEO-MEDIATEDCOMMUNICATION

Schegloff found that the following confirmation protocol isneeded in telephone conversation [6].

Conversation
acceptance(1) Confirmation of physical connection of telephone channel“Hello!” “Yes.” With these words, the telephone channel
connection is established.
(2) Person identificationFace-to-faceVideomediatedTelephone

Approach,gaze, raise
hand, “Hi!”smileGaze,
raise hand,“Hi!,” smile,“Can you seeme?”“Hello!”(Sameabove)(Sameabove)“Yes.”

asas

Shake hands,bow,“My name’sTaro.”(do not knoweach other)Bow,“My name’sTaro.”(do not knoweach other)“This isTarospeaking.
Is thatHanako?”(Same asabove)(Same asabove)“Speaking.”“Can I ask
yousomething?”“Are you freenow?”“Well.”“Can I ask
yousomething?”“Are you freenow?”“Well.”“Is this agood time
to talk?”

Table 1. Examples of conversation opening protocol“This is Taro speaking. Is that Hanako?” “Speaking.”
With these words, the caller and receiver are identified.
Note that this step can be skipped in the case of cellular
phones that display the phone number or name of thecaller.

research on how participants use non-verbal information
when they use visual communication media e.g. videoconferencing and videophone systems. The role of nonverbal information in the opening of a communication
channel based on observation of users’ behavior is unknown.
Referring to Schegloff’s work in telephone case and our own
empirical findings, the following hypotheses have beenformulated: “Those who use a video-mediated, non-eyecontact system shift their gaze less to confirm the connection
than those who use an eye-contact system,” and “To
compensate distorted gaze, such communicators use moreutterances, gestures and facial expressions than those using
an eye-contact system.” Considering the above three-step
protocol, the hypothesis is reorganized as follows:(3) Confirmation of receiver acceptance“Can I talk to you now?” “Is this a good time to talk?”This confirmation is needed because telephone is amedium where the caller interrupts the receiver.
This kind of protocol observed in telephone conversationmust also exist in other electronic communication media suchas videophone and teleconferencing systems. Examples ofprotocol patterns are summarized in Table 1. By verifying thethree-step protocol, we investigate users’ cognitive burden ofgaze understanding and recognitive burden of distorted gaze
compensation. For this purpose, observations of users’ verbal
and non-verbal behavior should be carried out.

Hypothesis 1: Confirmation of audio-visual connection
Users who utilize an eye-contact system repeatedly gazeand avert their gaze. This is because the audio-video linkconnection is confirmed by mutual gaze. Immediateawareness of connection follows social greetings such assmiles and raises of the eyebrows.

HYPOTHESESThis section formulates hypotheses on human behavior at theopening of communication based on findings in social
psychology and findings from the evaluation results of visualcommunication systems. Short et al. [9] compared two
media: telephone and face-to-face. They point out that
telephone users mainly use verbal and paralinguistic
information, while face-to-face communicators use a varietyof rich non-verbal information. However, there has been littleOn the other hand, users who utilize a non-eye-contact
system gaze more than users of an eye-contact system.

1678

CHI 2005 | Late Breaking Results: Posters

April 2-7 | Portland, Oregon, USA

remunerated after the experiments. Five pair of participants
were intimate friends, and the others were not acquainted
with each other. They debated the “Pros and cons of capital
punishment,” “Which do you prefer, cats or dogs?” andother topics.
They were requested to insist on their own opinion as far as
possible. In addition, they were informed that the purpose of
the experiment was to evaluate the video-mediated systems.
No information was given that the actual purpose of theexperiments was the establishment of communication
channels. Before the debate, the participants were instructed
to answer a questionnaire which did not concern thediscussion topic. One of the participants was requested topush a switch to connect the audio-video channel after
finishing the work. After the debate, the participants wereinstructed to fill in a questionnaire sheet about the usability ofboth systems.

Figure 1. Experiment with eye-contact system.

Experimental Results

Figure 2. Experiment with non-eye-contact system.

The recorded video sequences were analyzed based on theethnographical approach. An example of an analyzed
sequence is shown in Figure 3. From observations of the allvideo scenes with eye contact system, the subjects seem toconsider that visual-audio link was immediately connected
only by repetition of mutual gaze and averted gazeaccompanied by smiling. This resulted in some participants
forgetting to confirm the audio connection at first and having
to confirm the audio link connection later again. On theother hand, the non-eye-contact system needed quite a longtime for confirmation of the connection by attempting
behavior such as saying “Hi!” gestures, and smiling. This
(probably unconscious) behavior can be interpreted asconfirmation that the partner is looking at the subject, which
is one kind of compensation behavior that recognizes
distorted gaze. As a result, in the non-eye-contact system, theaudio-link turned out to be confirmed by unintended behavior
and there was no needed to confirm it.

This is because a non-eye-contact system gives erroneous
gaze direction and causes the compensated behavior of theuser such as, “Can you see me?” and waving his/her hand at
the monitor to confirm the connection. As a result,confirmation behavior requires a continuous gaze.
Hypothesis 2: Personal identification
If the caller and receiver are acquaintances, the confirmation
behavior is not observed. This is because confirmation is
performed by the displayed face only. If they are notacquaintances, salutations and self-introductions areobserved.
Hypothesis 3: Confirmation of the receiver’s acceptanceAn utterance corresponding to “Can I talk now?” isobserved. A speaker who utilizes the eye contact systemglances briefly. A user who utilizes the without-eye-contact
system continuously gazes at his/her partner.

Hypothesis 1 agrees with the experimental results. As for theuse of the non-eye-contact system, participants tried to draw
the partner’s attention by greeting and waving their hands,
which was not observed in the eye-contact system. The users’
behavior in the eye-contact system, that repeatedly gazed andaverted their gaze, meets the intimate equilibrium theory. The
behavior can be interpreted as prolonged gaze in close
proximity sometimes being considered rude.

EXPERIMENTS

Experiments were performed to investigate the above
hypotheses through the conversations of the participants.
Experimental System

An experimental video-mediated communication systemwith eye contact was produced using a half-silvered mirror
with 30-percent transmittance. The system was arranged sothat the camera center axis meets the conversational partner’s
gaze. Figure 1 shows the experimental system with eyecontact. “A non-eye-contact condition” was realized byplacing the camera on the right-hand side of the monitor as
shown in Figure 2. A two-way audio-video link was
established by the user pressing the button. The participants
wore headsets with a microphone. Video and audio wererecorded for analysis.

Hypothesis 2 agrees with the experiments. When theparticipants are known to each other, the partner’s face on amonitor did not draw the confirmation behavior. On the other
hand when the participants were not known each other, theygave salutations and self-introductions.
Hypothesis 3 partially agrees with observations where theparticipants uttered “Well….,” “Let’s begin,” for bothsystems. However, no apparent gaze differences were found.

Procedure

Seven pair of students between the ages of 20 and 23
participated in the experiments. The students were1679

CHI 2005 | Late Breaking Results: Posters

April 2-7 | Portland, Oregon, USA

CONCLUSION AND FUTURE STUDY

2. Argyle, M. and Cook, M. Gaze and Mutual Gaze.
London: Cambridge University Press. 1976.

The experiments indicate that eye-contact systems provide
immediate awareness of visual connection through users’
mutual gaze. These results correspond to the behavior
observed in face-to-face communication. On the other hand,participants using non-eye-contact systems are likely to need
confirmation of opening the conversation by waving theirhands and uttering a greeting. This can be regarded ascompensating for distorted gaze direction.

3. Grayson, D.M. and Monk, A.F., Are You Looking at Me?
Eye Contact and Desktop Video Conferencing, Trans. on
CHI ,SVol.10, No.3, 2003, pp.221-243
4. Ishii, I. and Kobayashi, M., ClearBoard: a Seamless
Medium for Shared Drawing and Conversation with EyeContact. In Proc. CHI92, ACM Press, 1992,pp. 525-532
5. Chen, M. Leveraging the Asymmetric Sensitivity of Eye
Contact for Videoconferencing In Proc. CHI2002, ACM
Press, pp.49-56, 2002

Interestingly, it was observed that a pair of participants using
the non-eye-contact system confirmed the three-dimensional
consistency of the space between the two, such as, “Am I in
the middle (of the monitor)?” This suggests that“confirmation of spatial consistency” can be included inaudio-visual connections. As future work, we will continue
our observation of users’ behavior and clarify the relationshipbetween the behavior and the usability of video-mediatedcommunication systems.

6. Schegloff, E. A., Beginning in the Telephone, ìn J.E Katzand M.A. Aakhus eds., Perpetual Contact: Mobile
Communication, Private Talk, Public Performance,
Cambridge Univ. Press, 2002
7. Okada, K., Maeda, F., Ichikawa, Y. and Matsushita, Y.
Multiparty Videoconferencing at Virtual Social Distance:
MAJIC Design. In Proc. CSCW, pages 385-393, 1994.

ACKNOWLEDGMENTS

This research was partially supported by the Ministry ofEducation, Science, Sports and Culture, Grant-in-Aid for
Scientific Research (B)(2)16300032, 2004 and by the
Research Center for Advanced Technologies, Tokyo Denki
University.

8. Rosenthal, A.H. Two-way Television Communication
Unit. US Pat. 2,420,198, 1947.
9. John Short, Ederyn Williams and Bruce Christie: The
Social Psychology of Telecommunications, John Wiley &Sons, 1976

REFERENCES

10. Vertegaahl, T., Van der Veer, G. C., and Vons, H. Effectsof Gaze on Multiparty Mediated Communication. In
Proc. Graphics Interface 2000 pp.95-102, 2000

1. Finn, K.E., Sellen, A.J., and Wilbur, S.B. (eds.): Videomediated Communication, Lawrence Erlbaum Associates,Inc., 1997.
Item

SubjectHand
gestureA
B

Time line
0 sec

Facialexpression
utterance

Front

10sec
mic
Raise hand

Front

Left

BA
B
A

4 sec
Raise hand

A
Gaze

2 sec
Raise hand

Front

Below left

Front

Below left

Below
Front Below
left
Below
Front
left

Front
Below
Below
Front
left
left

Smiling
Hi,B

Hi,Oh!

Well

Hi
Hi

(a) System with eye contact

Item

SubjectHand
gesture

A
B
A
B
A
B

Gaze
Facialexpression

0 sec

touch mic
touch microphone
Front
Front
CAM

Fron CAM

Front

Below left

18 sec

Below
Front

mic
Front
CAM Front B-Left CAM

Surprised
What'swrong?

Autterance
B

Time line
10 sec2 sec

Hi!

Aside?
You lookaside.

You push meaway

On my left.
Both of us. Here's
a camera.

Well….

(b) System with non-eye-contact
Figure 3 Behaviors of Participants just after the system on.

1680

No.

24sec
mic
Left
Front