Interaction Mediate Agent Based on User
Interruptibility Estimation
Takahiro Tanaka and Kinya Fujita
Graduate School, Tokyo University of Agriculture and Technology
2-24-16 Nakacho, Koganei, Tokyo 184-8588, Japan{takat,kfujita}@cc.tuat.ac.jp

Abstract. In recent years, the scope of users receiving information from information systems has been increasing. However, the timing of interruption is notcontrolled in most of these systems. In this paper, we propose a secretary agent
to mediate interactions between users and others. The agent estimates user interruptibility from PC operation records and head motions. Moreover, the agent
requests interaction from others by controlling the avatar’s gaze, joint attention,and mutual gaze on the basis of the interruptibility. We confirmed the usefulness of the mediation based on the estimation of user interruptibility.
Keywords: interruptibility, estimation, human–agent interaction, application
switching.

1 Introduction
In recent years, the scope of users receiving information from information systems
has been increasing with the popularization of the Internet and ubiquitous computing
environments. Although the frequency of interruptions from systems such as mailalerts, instant messaging tools, and software agents has been increasing, the timing ofsuch interruptions is not controlled in most information systems. Previous studies [1]have indicated that frequent interruption without considering the user’s status fragments the user’s work time and decreases productivity. Therefore, automatic status
estimation and ambient interruption methods are expected to assist in avoiding unintended interruptions.
One potential method for estimating a user’s online status is by monitoring PC operations or by using sensors [2-6]. However, these physical activity indices do not
capture all instances of the intellectual activities that should not be interrupted,
because intellectual activities do not always have observable outputs. Another approach is to estimate the breakpoint of a task [7, 8]. At the breakpoint, the user’s interruptibility is expected to increase instantaneously. However, this approach requires
task-structure analysis to determine the interruptibility level of the breakpoint.
We considered focused application switching (AS) as a breakpoint in PC work. Our
experimental results demonstrated that the interruptions at AS are significantly more
acceptable than those during continuous work. Finally, we proposed a user interruptibility estimation method for AS timing [12].
M.J. Smith, G. Salvendy (Eds.): Human Interface, Part I, HCII 2011, LNCS 6771, pp. 152–160, 2011.
© Springer-Verlag Berlin Heidelberg 2011

Interaction Mediate Agent Based on User Interruptibility Estimation

153

In this paper, we propose a secretary agent to mediate interactions between the user
and other users or systems, in order to start accepting interactions with others, basedon the interruptibility estimation method. The agent gently requests interactions fromthe user by using gazing actions (joint attention and mutual gaze) and leads the user toconfirm the request on his/her own. We experimentally confirmed the usefulness of
the agent for estimation and mediation between users and others.

2 Related Studies
2.1 Estimation of User Business/Interruptibility
There are several related studies regarding user-status estimation by various techniques; e.g., counting the keystrokes or mouse clicks and using various sensors placedin the work space or on the users [2-6]. These methods are expected to adequately
estimate the user’s status when the status has observable physical activity indices.
However, intellectual activity such as deep thinking has no observable output. Moreover, it is practically difficult to place various sensors on the user or in the work
space.
On the other hand, some studies have reported a relationship between interruptibility and breakpoints [7, 8]. At a breakpoint, the user’s interruptibility is expected toincrease instantaneously even if the work requires intellectual activity. Furthermore, it
has been suggested that the kind of the breakpoint affects interruptibility. However,this method requires an analysis of the task structure to determine the level of the
breakpoint. Multi-tasking during PC usage is also a potential difficulty in the application of the task-structure based method. Therefore, it is still a challenge to distinguishthe level of interruptibility at a breakpoint.
Moreover, the interruptibility expression must be intuitive in order to reduce theuser’s cognitive load. Some studies have examined ambient awareness using nonverbal information. Movements of the head, face, and body were determined to be effective in expressing nonverbal behavior [9]. The expression of interruptibility using
nonverbal information through an avatar appears promising for the intuitive recognition of a user’s status.
2.2 Study of Application Switching for User Interruptibility Estimation
We used focused application switching (AS)—transition of the active application
window—as a breakpoint in PC work [12]. AS is considered to be the user’s intentional switching of the working space or working target. Therefore, the user’s concentration at AS is expected to be instantaneously weakened compared to that duringcontinuous work. Moreover, AS commonly and frequently occurs in PC work and iseasily detected. Thus, AS offers scope for information presentation with less risk oftask disturbance.
To examine this assumption, we experimentally collected and analyzed PC operation records and subjective interruption evaluation logs. Experimental results demonstrate that interruptions at AS are significantly more acceptable for users than thoseduring continuous work (p < 0.01, t-test).

154

T. Tanaka and K. Fujita

Therefore, we analyzed the relationship between interruptibility scores and the indices calculated from the operation records, which were expected to reflect the interruptibility at AS. Finally, we defined the estimation rule on the basis of co-occurring
indices at the same AS, and decided to judge the interruptibility using three levels in away similar to the previous works, for practical utility. Figure 1 shows the experimental results of the interruptibility estimation using our proposed method. The results
demonstrate that the proposed estimation method allows information systems to reduce the risk of serious work disturbance due to interruptions.

Fig. 1. Results of interruptibility estimation using three levels

3 Secretary Agent
We proposed an interaction initiation method with acceptable timing by using anagent to mediate the interaction based on the user-interruptibility estimation. In thissection, we explain the concept of agent mediation and details of agent architecture.
3.1 Concept of Interaction Initiation by Agent’s Mediation
The above analysis indicates that interruptions at AS are significantly more acceptablefor users than those during continuous work. Based on this analysis, we proposed theinterruptibility estimation method. This method provides a controlling interruption
timing that is useful to initiate interactions without causing problems of disturbancefrom online users or other information systems. However, as the estimation has anerror, there is a risk of unintentional disturbance. Therefore, we need to consider amethod that is robust against estimation errors.
Interruptions by pop-up windows or sounds may seriously disturb the user’s work.
Therefore, an ambient and natural presentation of the interruption is necessary toenable the busy user to not respond until a break [9-11]. Moreover, it is difficult topredict a highly interruptible time. Therefore, assistance from the information system
is needed for smoothly starting the interaction at a high-interruptibility period.
In this research, the secretary agent estimates user interruptibility and receives interaction requests from a user or system in bulk. After receiving the request, the agent

Interaction Mediate Agent Based on User Interruptibility Estimation

155

starts to convey it at a high-interruptibility period by using an ambient presentation.
The presentation is based on actions in human daily life, “wait and see how the interaction partner works.” In particular, the agent gently presents the request to the user
by gazing actions and leads the user to start the interaction. Therefore, the agent’s
mediation causes a shift of positions between the interrupter and the interrupted user.
Figure 2 shows an example of dialog initiation by the secretary agent. The dialog
initiation has the following steps:
1. Request dialog: If user A wants to start a dialog with user B, A sends a dialog
request to B’s agent through his/her own agent via the network. Then A awaits amessage from B while doing his/her own work.
2. Start appeal: The dialog request by A is sent to the secretary agent of B, and theagent starts to present the dialog request unobtrusively by using gaze controls, joint
attention, and mutual gaze. The agent continues to appeal to B according to his/herinterruptibility.
3. Notice: If B notices the appeal and decides to start a dialog with A, B sends an
instant message using the system to A. On the other hand, B is allowed to not respond to the request until a break, because A is doing his/her own work and is notable to recognize whether B has not noticed the appeal or is ignoring the request.
3.2 Overview of the Secretary Agent
Figure 3 shows the architecture of the secretary agent. The developed agent consistsof two main functional components, the user’s interruptibility estimation component
and the request appeal component.
The interruptibility estimation component monitors the user’s PC operation activity
and head position using a web camera and then estimates user interruptibility using
three levels based on our proposed method [13]. The estimated interruptibility is used
to control the intensity of an appeal using a CG avatar. The request appeal component
expresses the existence of an interaction request from others by the joint attention and

Fig. 2. Example of dialog initiation by the secretary agent’s mediation

156

T. Tanaka and K. Fujita

Fig. 3. Architecture of the secretary agent

Fig. 4. Example of user environment

mutual gaze motions. The presentation strength and timing of the request are modified
on the basis of the estimated interruptibility in order to attain pleasant and acceptabledialog initiation.
Figure 4 shows an example of user environment using the secretary agent. In thisstudy, we propose using a USB sub-monitor to display the agent. Moreover, a web
camera, which is used to capture the user’s head position, is set up at the top of the
sub-monitor. This not only improves the practicality, because it does not encroach
upon a user’s working area, but also reinforces the weak awareness of software
agents. Moreover, a sense of intimacy and reliance on the agent are achieved easily
because of the continuous interaction with the agent existing at the same location. We
expect that a positive impression of the agent reduces the adverse effect due to theestimation effort.
Function I: User interruptibility estimation
The agent estimates the user interruptibility at AS timings in three levels based on our
proposed method [13]. One difficulty in the previous research was that there wereInteraction Mediate Agent Based on User Interruptibility Estimation

157

insufficient opportunities to interrupt at high interruptibility, only twice an hour.
Therefore, in this study, we developed a method to estimate the interruptibility evenduring continuous work (not application switching: NAS).
To improve the accuracy of the estimation and to expand the estimation range ofthe target work, we considered using the user’s head motion because it tends to reflect
the user’s motivation toward the current work [14]. To recognize the user’s headmotion, the agent captures the user’s image using the web camera, and then detects
the user’s facial position and motion using OpenCV.
Function II: Ambient appeal of interaction requests
The agent presents a specific interaction request by using both joint attention andmutual gaze.
− Joint Attention: If the partner is in the NAS state, the agent occasionally observes
the user’s active window, which might be the user’s current work space. The agent
then appeals that the requested user is interested in the partner.
− Mutual Gaze: On the partner’s AS, the agent gazes at the user who might be sitting in front of the monitor. At the time of AS, the interruptibility of the user will
be reduced temporally, and so the agent uses a stronger presentation of the request,“face to partner,” to influence the user to talk.
User interruptibility continually changes during the user’s task. Therefore, the intensity of the appeal should be changed according to user interruptibility, or else theappeal would easily disturb the user’s concentration even if it was ambient. Figure 5
shows a model of appeal intensity control for joint attention and mutual gaze basedon an utterance attitude model [12]. The largeness and frequency of movement are

Fig. 5. Appeal intensity control for joint attention and mutual gaze

158

T. Tanaka and K. Fujita

controlled on the basis of the user’s interruptibility in three levels so as not to disturb
the user’s work. When the user’s interruptibility is low, the agent appeals the request
by using small movements and a low frequency. On the other hand, when the interruptibility is high, the agent frequently appeals the request by using large movements.

4 Experiment
The principal aim of this experiment is to confirm whether our interruptibility estimation method is useful for controlling presentation timing and whether the usage of a
sub-monitor affects the impression of the secretary agent.
4.1 Procedure
In this experiment, the agent interrupts a subject performing given tasks in four experimental conditions. On being interrupted, the subject evaluates the interruptibility
at the time. The evaluation is scaled from 1: “Absolutely uninterruptible” to 5: “Noproblem.” Moreover, the subject evaluates his/her impression of the agent from three
viewpoints by using a five-point scale: accuracy of interruption timing, tolerance ofpoorly timed interruptions, and motivation to use the agent. The subjects were eight
university students who participated in this experiment in an environment shown in
Figure 4. The subjects were assigned two tasks, a paper test and a puzzle. In addition,
they were informed that the agent estimates the interruptibility and interrupts at highlyinterruptible times.
Two factors are crucial in this experiment. One is the control of presentation timing; the agent either interrupts the subject at random or estimates the interruptibility
and interrupts at a highly interruptible time. The other is the position of the secretary
agent; at the corner of the main monitor or sub-monitor. We set four experimental
conditions as follows:− A) Random timing and displaying in main monitor.
− B) Random timing and displaying in sub-monitor.
− C) Estimated timing and displaying in main monitor.
− D) Estimated timing and displaying in sub-monitor.
4.2 Results and Discussion
Figure 6 shows the evaluated interruptibility scores in random and estimation conditions. The average interruptibility of random interruptions was 2.5, whereas the average interruptibility of estimated interruptions was 3.6. Almost all random interruptions
were during low interruptibility. On the other hand, the estimated interruptions were athigh interruptibility. The interruptions at estimated timings were significantly higher
than those at random. Thus, the results indicate the usefulness of the control of presentation timing based on the proposed method.
Figure 7 shows an interaction diagram of the four experimental conditions. The results of two-way factorial ANOVA without replication indicate that the level of interruptibility significantly affects the impression of the agent for all evaluations. Thus, it
is considered that the interruption timing is very important for the impression of theagent because the agent has a well-defined purpose of controlling presentation timing.

Interaction Mediate Agent Based on User Interruptibility Estimation

159

Fig. 6. Evaluated interruptibility: random vs. estimation

Fig. 7. Interaction diagram of evaluated scores

There was no significant difference regarding the display position of the agent.
However, the subjects rated the agent displayed in the main monitor higher than that in
the sub-monitor when evaluating the tolerance of a poorly timed interruption. At thispoint, we additionally interviewed the subjects regarding the reasoning behind thishigh rating. Some subjects assigned low scores for the tolerance because they expected
improved agent performance with the use of particular devices and spaces such as a
sub-monitor, and were disappointed because they did not observe any particular advantage of the device. In this experimental condition, we had turned off the request-appeal
function because we attempted to evaluate the availability of the proposed estimation
method. Therefore, the agent never moved during the experiment, even if it had a dedicated device and space. The user evaluates the agent according to not only its performance but also the cost or impression of the devices that comprise it.

5 Conclusions
In this paper, we proposed a secretary agent to mediate interactions between auser and other users or systems in order to start accepting interactions based on the

160

T. Tanaka and K. Fujitainterruptibility estimation method. We experimentally confirmed that our proposed
estimation method is useful for controlling the agent’s presentation timing. In the
future, we shall experimentally evaluate the ambient appeal method by using gazing
actions to interrupt without causing stress.
Acknowledgements. This research was supported by JSPS KAKENHI (21700130)
and National Institute of Information Technology (NICT).

References
1. Bailey, B.P., Konstan, J.A., Carlis, J.V.: The Effect of Interruptions on Task Performance,
Annoyance, and Anxiety in the User Interface. In: Proc of INTERACT 2001, pp. 593–601(2001)
2. Honda, S., et al.: A Home Office Environment Based on the Concentration Degrees
of Wokers: A Virtual Office SystemValentine. Trans. Info. Processing Society ofJapan 39(5), 1472–1483 (1998) (in Japanese)
3. Minakuchi, M., Takeuchi, T., Kuramoto, I., Shibuya, Y., Tsujino, T.: An Automatic Estimation Method for Busyness at Deskwork. Trans. Human Interface Society 6(1), 69–74(2004) (in Japanese)
4. Fogarty, J., Christensen, J.L.J.: Presence versus availability: the design and evaluation of acontext-aware communication client. Int. J. Human-Computer Studies 61(3), 299–317(2004)
5. Chen, D., Hart, J., Vertegaal, R.: Towards a physiological model of user interruptability.
In: Baranauskas, C., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT 2007. LNCS,
vol. 4663, pp. 439–451. Springer, Heidelberg (2007)
6. Hudson, S.E., Fogarty, J., Atkeson, C.G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J.C.,
Yang, J.: Predicting Human Interruptibility with Sensors: A Wizard of Oz Feasibility
Study. In: Proc. SIGCHI Conf. on Human Factors in Computing Systems, pp. 257–264(2003)
7. Czerwinski, M., Cutrell, E., Hirvutz, E.: Instant Messaging and Interruption: Influence ofTask Type on Performance. In: Proc. of OZCHI 2000, pp. 356–361 (2000)
8. Iqbal, S.T., Bailey, B.P.: Leveraging Characteristics of Task Structure to Predict the Costof Interruption. In: Proc. of SIGCHI Conference on Human Factors in Computing Systems, pp. 741–750 (2006)
9. Lee, J., Marsella, S.C.: Nonverbal behavior generator for embodied conversational agents.
In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS
(LNAI), vol. 4133, pp. 243–255. Springer, Heidelberg (2006)
10. Kendon, A.: Some Functions of Gaze-direction in social interaction. Acta Psychologia 26,
22–63 (1967)
11. Kobayashi, K., Yamada, S.: Informing a User of Robot’s Mind by Motion. In: CIRAS
2005, SS4B-3 (2005)
12. Yuasa, M., Mukawa, N., Kimura, K., Tokunaga, H., Terai, H.: An utterance attitude modelin human-agent communication: from good turn-taking to better human-agent understanding. In: Proc. of CHI 2010, pp. 3919–3924 (2010)
13. Tanaka, T., Fujita, K.: Study of User Interruptibility Estimation Based on Focused Application Switching. In: CSCW 2011 (2011)
14. Kimura, K., Tanaka, T., Fujita, K.: Study of Head Motion Components related Uninterruptibility during Desk Working. Info. Processing Society of Japan 52-4 (2011) (in Japanese)