II-p5

Secretary Agent for Mediating Interaction Initiation
Takahiro Tanaka 1
1Kinya Fujita 1

Graduated School, Tokyo University of Agriculture and Technology

Abstract: In recent years, the possibility that users might be interrupted by information systems
has been increasing with the growing popularity of the Internet and the ubiquity of computing
environments. However, the timing of interruptions is not controlled in most systems. In this study,
we propose a secretary agent to mediate interaction initiation between users or human-computer.
The agent appeals the request of interruption to user from other by controlling the character’s
gaze at the timing of less cognitive load based on the interruptibility estimation. We conﬁrmed
the usefulness of the mediation by the agent.

1Introduction

In recent years, the possibility that users might beinterrupted by information systems has been increasing with the growing popularity of the Internet andthe ubiquity of computing environments. However,
the timing of interruptions is not controlled in mostsystems. Previous studies have suggested that thesuspension and resumption of the problem state andmemory processes relating to a previous task occur asa result of interruption when task is switched in multitasking, thus the switch causes a time lag[1, 2] (Figure1). In particular, it has been pointed out that the time
lag or ”Resumption Lag” can be potentially increased
because of interruption timing, length of task execution time, the type of task being performed, and therelationship among diﬀerence tasks[3, 4]. Therefore,
it has been pointed out that frequent interruptions do
not reﬂect user status, can fragment the user s working time and can decrease intellectual productivity[5,6].
ᵧᶌᶒᶃᶐᶐᶓᶎᶒᶇᶍᶌᴾᵪᵿᶅᶕᵿᶐᶌᶇᶌᶌᶅᴾᶍᶄᴾᶇᶌᶒᶃᶐᶐᶓᶎᶒᶇᶍᶌᵰᶃᶑᶓᶋᶎᶒᶇᶍᶌᴾᵪᵿᶅ
ᵧᶌᶒᶃᶐᶐᶓᶎᶒᵰᶃᶑᶓᶋᶃᶍᶀᶑᶃᶐᶔᶃᶂᴾᵿᶁᶒᶇᶔᶇᶒᶗ

ᵲᵿᶑᶉᴾᵟ

ᵲᵿᶑᶉᴾᵠᶒᵿᶑᶉᴾᶑᶕᶇᶒᶁᶆ

ᵲᵿᶑᶉᴾᵟ

ᶒᶇᶋᶃᶒᵿᶑᶉᴾᶑᶕᶇᶒᶁᶆ

Figure 1: Cognitive load caused by an interruption
during multitasking.
One potential method for estimating a user s online status is by monitoring PC operations or by using
sensors[7, 8, 9, 10, 11]. However, these physical activity indices do not capture all instances of the intellectual activities that should not be interrupted, because
intellectual activities do not always have observable
outputs. On the other hand, previous studies suggested that interruptions at a recognized breakpoint
are preferable to interruptions during an ongoing task
because the Resumption Lag caused by an interrup-tion at a breakpoint is signiﬁcantly short[12, 13, 14,15]. Therefore, real-time breakpoint estimation by an
information system can be considered a method forcontrolling the timing of interruption. Iqbal and Bailey proposed breakpoint detection based on structural
analysis of tasks[18], and there are suggestion that thetiming of application switching during PC work is as
an alternative breakpoint[16, 17]. However, in realworld, variations of applications, tasks, and personalenvironments occur frequently. Therefore, it is difﬁcult to apply the task-structure-based method andrecognize the details of PC operations in each speciﬁcapplication.
In our previous studies, we considered focused application switching (AS) as an alternative of breakpoint during PC work. Our experimental results demonstrated that the interruptions at AS are signiﬁcantly
more acceptable than those during continuous work.
Finally, we proposed a user interruptibility estimation
method during PC work[19]. However, as the estimation essentially has an error, there is a risk of unintentional disturbance. Interruptions by pop-up windows
or sounds based on the error may seriously disturb theuser s work. Therefore, we need to consider a presentation method that is robust against estimation
errors.
In this paper, we propose a secretary agent to mediate interactions between users or human-computer,in order to start interactions smoothly with others.
The agent gently appeals interaction request from otherby using gaze actions at estimated interruptible timing and leads the user to conﬁrm the request on his/herown. We experimentally conﬁrmed that the secretary
agent is useful for smooth interaction initiation andour method reduced a cognitive load by the interruption.

2Mediation of Interaction Initiation

We proposed an interaction initiation method withacceptable timing by using an agent to mediate theinteraction based on the user-interruptibility estima-

tion. Figure 2 shows an example of dialog initiationby the secretary agent. The dialog initiation has thefollowing steps:
1. Request dialog: If user A wants to start adialog with user B, A sends a dialog request toB s agent through his/her own agent via thenetwork. Then A awaits a message from B while
doing his/her own work.
2. Start appeal: The dialog request by A is sent
to the secretary agent of B, and the agent starts
to present the dialog request unobtrusively byusing gaze controls at estimated high interruptible timing.
3. Notice: If B notices the appeal, B will access his/her agent for conﬁrming the request on
his/her own. If B decides to start a dialog withA, B sends a message to A. On the other hand,
B is allowed to not respond to the request until
a breakpoint, because A is doing his/her own
work and is not able to recognize whether B has
not noticed the appeal or is ignoring the request.

3Secretary Agent

Figure 3 shows the architecture of the secretary agent.
The developed agent consists of two main functional
components, the user s interruptibility estimation
component and the request appeal component. Theinterruptibility estimation component monitors theuser s PC operation activity and head motion using aweb camera and then estimates user interruptibility inthree levels. The request appeal component expresses
the existence of an interaction request from others by
the joint attention and mutual gaze motions at thetiming when the agent estimates interruptibility ashigh.
ᶉᶃᶗᶑᶒᶐᶍᶉᶃᵊᴾᶋᶍᶓᶑᶃᵊ
ᶕᶇᶌᶂᶍᶕᴾᶋᶃᶑᶑᵿᶅᶃᵊ
ᶎᶐᶍᶁᶃᶑᶑᵊᴾᶃᶒᶁᵮᵡᴾᵭᶎᶃᶐᵿᶒᶇᶍᶌᴾᵰᶃᶁᶍᶐᶂ
ᵡᵿᶋᶃᶐᵿ

ᵰᶃᶏᶓᶃᶑᶒᴾᵢᶇᵿᶊᶍᶅ

ᵆᵧᶌᶒᶃᶐᶐᶓᶎᶒᶃᶐὸ

ᵢᶍᶇᶌᶅᴾᶍᶕᶌᴾᶕᶍᶐᶉᴾᶕᶆᶇᶊᶃ
ᶕᵿᶇᶒᶒᶇᶌᶅᴾᶄᶍᶐᴾᶐᶃᶑᶎᶍᶌᶑᶃᴾᶄᶐᶍᶋᴾᵠ

ᵶᵊᵷᵊᵸ
ᵤᵿᶁᶃ
ᵢᶃᶒᶃᶁᶒᶇᶍᶌ ᵮᶇᶒᶁᶆᵦᶃᵿᶂᴾᵫᶍᶒᶇᶍᶌ

ᵧᶌᶒᶃᶐᶐᶓᶎᶒᶇᶀᶇᶊᶇᶒᶗ
ᵣᶑᶒᶇᶋᵿᶒᶇᶍᶌᵤᶓᶌᶁᶒᶇᶍᶌ

ᵱᵣᵰᵴᵣᵰ

ᶇᶌᶒᶃᶐᶐᶓᶎᶒᶇᶀᶇᶊᶇᶒᶗᵰᶃᶏᶓᶃᶑᶒᵿᶎᶎᶃᵿᶊ

ᵳᵱᵣᵰ
ᵳᵱᵣᵰᴾᾐ

ᵭᶒᶆᶃᶐᴾᵳᶑᶃᶐᶑ

ᵱᶃᶁᶐᶃᶒᵿᶐᶗᴾᵟᶅᶃᶌᶒ

ᵡᵥᴾᵡᶆᵿᶐᵿᶁᶒᶃᶐ

ᵿᶌᶇᶋᵿᶒᶇᶍᶌᶁᶍᶌᶒᶐᶍᶊ

ᵰᶃᶏᶓᶃᶑᶒᴾᵟᶎᶎᶃᵿᶊᵤᶓᶌᶁᶒᶇᶍᶌὉ ᵨᶍᶇᶌᶒᴾᵿᶒᶒᶃᶌᶒᶇᶍᶌὉ ᵫᶓᶒᶓᵿᶊᴾᶅᵿᶘᶃᶂᶇᵿᶊᶍᶅᴾᶐᶃᶏᶓᶃᶑᶒ

ᵭᶒᶆᶃᶐᴾᵱᶗᶑᶒᶃᶋᶑ
ᶇᶌᶒᶃᶐᶐᶓᶎᶒᶇᶍᶌᴾᶐᶃᶏᶓᶃᶑᶒ

ᵢᶇᵿᶊᶍᶅᴾᵱᶒᵿᶐᶒᴾḷ

Figure 3: Architecture of secretary agent.

ᵟḙᶑᵟᶅᶃᶌᶒᶋᶃᶑᶑᵿᶅᶃᶒᶇᶋᶃᶐᶃᶏᶓᶃᶑᶒḙ
ᵠᴾᶑᴾᵧᶌᶒᶃᶐᶐᶓᶎᶒᶇᶀᶇᶊᶇᶒᶗᴾᵣᶑᶒᶇᶋᵿᶒᶇᶍᶌ

ᵳᵱᵣᵰᴾᾑ
ᵆᵧᶌᶒᶃᶐᶐᶓᶎᶒᶃᶂὸ

ᵠḙᶑᵟᶅᶃᶌᶒ

ᵟᶎᶎᶃᵿᶊᶇᶌᶅᴾᶀᶗᴾᶅᵿᶘᶃᴾᶁᶍᶌᶒᶐᶍᶊ
ᵿᶒᴾᶆᶇᶅᶆᴾᶇᶌᶒᶃᶐᶐᶓᶎᶒᶇᶀᶊᶃᴾᶒᶇᶋᶇᶌᶅ
ᵦᵦ

3.1ᵦᵧᶌᶒᶃᶐᶐᶓᶎᶒᶇᶍᶌ
ᵪᵿᶅᶕᶍᶐᶉᶇᶌᶅᵌᵌᵌᵬᶍᶒᶇᶁᶃᵡᶍᶌᶄᶇᶐᶋ

Figure 2: Exmaple of agent’s mediation.
This agent uses the user interruptibility estimation method during PC work. This method provides
a controlling interruption timing that is useful to startinteractions without causing problems of disturbancefrom others. However, as the estimation essentiallyhas an error, there is a risk of unintentional disturbance. Interruptions by pop-up windows or sounds
based on the error may seriously disturb the user swork. Therefore, we need to consider a presentation
method that is robust against estimation errors. In
this study, we combined the interruptibility estimation with the ambient presentation of the interruption which enables the busy user to not respond until
a breakpoint. Moreover, the ambient method can set
a brief time lag between the notice (warning of interruption) and the conﬁrm (interruption) about several
seconds. Previous studies suggested that the time lagor ”Interruption Lag” can decrease a cognitive loadby the interruption [20]. Therefore, the agent s mediation realizes the acceptable interaction initiation
without high cognitive load by interruptions.

Interruptibility estimation function

The agent estimates the user interruptibility during
PC work using the interruptibility estimation function. This function monitors user’ s PC operation
records, keystroke, mouse operation, process id, number of window and so on every 500ms. In previous
study, we experimentally collected and analyzed several thousands hours records, and selected 23 indicesthat aﬀect to the interruptibility during work. Moreover, we proposed the user interruptibility estimation
method based on PC operation records [19]. We conﬁrmed that the accuracy of our proposed method wasabout 60% in both of laboratory and oﬃce environments.
To improve the accuracy of the estimation andto expand the estimation range of the target work,
we considered using the user s head motion because
it tends to reﬂect the user s motivation toward thecurrent work. We analyzed a correlation between
the head motion during work and the interruptibility, added some indices about head motions to ourestimation method. Then, we conﬁrmed the accuracyof the estimation increased to 75%. To recognize theuser s head motion, the agent captures the user s image using the web camera, and then detects the user shead motion Kinect or faceAPI (Figure 4). Currently,
using a head motion is provided as an option because
preliminary calibration is needed for the recognition.

㻺㼛㼞㼍㼙㼘㻌㻿㼠㼍㼠㼑㻹㼍㼕㼚㻌㻹㼛㼚㼕㼠㼛㼞㻔㼒㼛㼞㻌㼃㼛㼞㼗㻕㻯㼡㼞㼞㼑㼚㼠
㼒㼛㼞㼏㼡㼟㼑㼐㼣㼕㼚㼐㼛㼣㻿㼡㼎㻌㻹㼛㼚㼕㼠㼛㼞㻔㼒㼛㼞㻌㼍㼓㼑㼚㼠㻕㻗㻌㻾㼑㼝㼡㼑㼟㼠㻰㼡㼞㼕㼚㼓㻌㼏㼛㼚㼠㼕㼚㼡㼟㻌㼣㼛㼞㼗㻚㻚㻚

Figure 4: Example of user interruptibility estimation
with head motion.

3.2㻯㼡㼞㼞㼑㼚㼠㼒㼛㼞㼏㼡㼟㼑㼐㼣㼕㼚㼐㼛㼣

Request appeal function

The agent presents a speciﬁc interaction request byusing both joint attention and mutual gaze at thetiming when the estimation function estimates highinterruptibility as following.
• Joint Attention: If the user is continually using same application (NAS), the agent occasionally observes the user s active window, whichmight be the user s current work space. Theagent appeals that there is something to informto the user by displaying the interestedness tothe user work.
• Mutual Gaze: On the user s application switching (AS), the agent gazes at the user who might
be sitting in front of the monitor. At the timeof AS, the interruptibility of the user will be
reduced temporally, and so the agent uses astronger presentation of the request, face touser, to inﬂuence the user to talk.
Moreover, when the interruptibility is high, the agent
frequently appeals the request by using appeals. Figure shows examples of both gaze controls, joint attention and mutual gaze.

4

Evaluation

4.1

Procedure

We conducted the experiment for evaluating the secretary agent. In this experiment, the agent interruptsa subject performing given tasks in four experimentalconditions. On being interrupted, the subject evaluates a subjective interruptibility at the time. Theevaluation is scaled from ”Uninterruptible (low interruptibility)” to ”Interruptible (high interruptibility).”
Two factors are crucial in this experiment. One
is the control of presentation timing; the agent either interrupts the subject at random or interrupts at㻶㼛㼕㼚㼠㻌㻭㼠㼠㼑㼚㼠㼕㼛㼚㼃㼔㼑㼚㻌㼡㼟㼑㼞㻌㼟㼣㼕㼠㼏㼔㼑㼟㻌㼍㼜㼜㼘㼕㼏㼍㼠㼕㼛㼚㻚㻚㻚㻭㻿㼁㻿㻱㻾㻹㼡㼠㼡㼍㼘㻌㻳㼍㼦㼑

Figure 5: Appeal request using joint attention andmutual gaze.

estimated highly interruptible timing. The other isthe presentation (interruption) method by the agent;interruption by dialog window which was suddenly
displaied like a most common system or ambient appealing by gaze actions. We set four experimental
conditions as follows:• A: Random timing and Dialog interruption(like a most common system)• B: Estimated timing and Dialog interruption• C: Random timing and Appeal interruption• D: Estimated timing and Appeal interruption(mediation method)
The subjects were ten university students. They
were assigned two tasks, crossword and Sudoku. Inaddition, they were informed that the agent estimatesthe interruptibility and interrupts at highly interruptible times. During this experiment, the secretary agent
was displaied on a sub monitor which set side by main
monitor like Figure 5. Moreover, the agent estimatedsubject’s interruptibility without head motions in thisexperiment.

4.2

ResultFigure 6 shows a frequency of the subjective interruptibility at interrupted timing on each conditions.
In condition B and D, the agent estimated high interruptible timing. Therefore, the frequency of highinterruptibility on both conditions can be regardedas an accuracy of estimation. In condition A (random/dialog), over 60% of interruptions were conducted
at low interruptible timing. On the other hand, incondition B (estimation/dialog), over 60% interruptions were at high interruptible timing. Therefore, theaccuracy of the estimation method was about 60%,
and this result was same as the previous study. The
appealing interruption improved the interruptibility
scores in both conditions C and D. In particular, theaccuracy of high interruptibility in condition D was85% even when the estimation algorithm is same ascondition B. Moreover, the rate of serious error which
the method evaluates low interruptibility as high one
was only 4%.

䣔䣎䢪䣵䢫䢬䢬䢬䢼䢢䣲䢾䢲䢰䢲䢷䣈䣴䣧䣳䣷䣧䣰䣥䣻䢪䣴䣣䣶䣧䢫䣕䣷䣤䣬䣧䣥䣶䣫䣸䣧䢢䣋䣰䣶䣧䣴䣴䣷䣲䣶䣫䣤䣫䣮䣫䣶䣻

Figure 7: Comparing a resumption lag on each conditions.

Figure 6: Frequency of the evaluated subjective interruptibility at interrupted timing on each conditions.
In previous studies suggested that Resumption Lag(RL) is one of indices for evaluating the cognitive loadby an interruption during multitask. We calculatedand compared RL of each interruptions in four conditions for evaluating the cognitive load. Referringto the previous study, we deﬁned RL as the periodof time required to return to the previous task (window) is until the ﬁrst observed activity subsequent toan interruption. Here the activity involved keystrokes,
mouse clicks, and wheel usage not speciﬁcally aﬀectedby the kind of task being performed.
Figure 7 shows a result of RL on each conditions.
We analyzed RLs by one-way analysis of variances,then revealed that the condition signiﬁcantly reﬂects
RL by an interruption (F (3, 170) = 4.31, p < 0.01).
After the multiple comparison by Bonferroni method,
we conﬁrmed that there are signiﬁcant diﬀerences between A and C, and A and D (p < 0.05).

4.3

Discussion

From the result, the accuracy of interruptibility estimation was almost same as previous experiments.
However, in condition D (estimation and appeal), thesubjective accuracy of estimation was over 80%, so

proposed appeal interruption method improved theaccuracy about 25%. Moreover, the rate of serious error was reduced to 4% from 21% (condition B). Therefore, this result suggested that the combination of estimation and appeal is a presentation method which isrobust against estimation errors. In condition A (likea most common system), interruptions caused about 8
seconds time lag every time. Our mediation method
signiﬁcantly decreased the time lag to 4.6 seconds.
From the view point of cognitive load, the mediation
decreased the cost of interruption comparing to previous method. Furthermore, this result suggests that
100% of accuracy is not necessarily required for theuser status esitmation if we consider people’s cognitive characteristic.

5Conclusion

In this paper, we proposed a secretary agent to mediate interactions between a user and other users orsystems in order to start interactions at acceptabletiming. We experimentally conﬁrmed that the interaction mediation is useful for smooth interaction initiation and our method reduced a cognitive load by
the interruption.

Acknowledgement
This research was supported by JSPS KAKENHI andNational Institute of Information Technology (NICT).

References[1] E. M. Altman and J. G. Trafton: Memory forgoals: An activation-based model. Cognitive Science, Vol.26, pp.39-83 (2002).

[2] D. D. Salvucci and N. A. Taatgen: Threaded
Cognition: An Integrated Theory of Concurrent
Multitasking. Psychological Review, Vol.115,
No.1, pp.101-130 (2008).

[14] C. A. Monk, D. A. Boehm-Davis and J. G.
Trafton: Recovering from interruptions: Implications for driver distraction research. Human Factors, Vol.46, pp.650-663 (2004).

[3] E. B. Cutrell, M. Czerwinski and E. Horvitz: Effects of instant messaging interruptions on computing tasks. Proc.CHI2000, pp.99-100 (2000).

[15] D. D. Salvucci: On reconstruction of task context after interruption. Proc.CHI2010, pp.89-92(2010).

[4] C. A. Monk, J. G. Trafton and D. A. BoehmDavis: The eﬀect of interruption duration anddemand on resuming suspended goals. Journalof Experimental Psychology: Applied, Vol.14,
pp.299-313 (2008).

[16] M. Czerwinski, E. Horvitz and S. Wilhite: A diary study of task switching and interruptions.
Proc.CHI 04, pp.175-182 (2004).

[5] G. Mark, V. M. Gonzalez and J. Harris: No task
left behind? Examining the nature of fragmentedwork. Proc. CHI 05, pp.321-330 (2005).
[6] G. Mark, D. Gudith and U. Klocke: The costof interrupted work: more speed and stress.
Proc.CHI2008, pp.107-110 (2008).
[7] S. Honda, et al., A Home Oﬃce Environment
Based on the Concentration Degrees of Wokers:
A Virtual Oﬃce SystemValentine, Trans. Info.
Processing Society of Japan, 39(5), pp.1472-1483(1998) (in Japanese).
[8] M. Minakuchi, T. Takeuchi, I. Kuramoto, Y.
Shibuya and T. Tsujino: An Automatic Estimation Method for Busyness at Deskwork, Trans.
Human Interface Society, Vol.6, No.1, pp.69-74(2004) (in Japanese).
[9] J. Fogarty and J. L. J. Christensen: Presence
versus availability: the design and evaluationof a context-aware communication client, Int. J.
Human-Computer Studies, Vol.61, No.3, pp.299317 (2004).
[10] D. Chen, J. Hart and R. Vertegaal: Towards a
Physiological Model of User Interruptability, INTERACT 2007, LNCS 4663, Part II, pp.439-451(2007).
[11] S. E. Hudson, J. Fogarty, C. G. Atkeson, D.
Avrahami, J. Forlizzi, S. Kiesler, J. C. Lee andJ. Yang: Predicting Human Interruptibility withSensors: A Wizard of Oz Feasibility Study, Proc.
SIGCHI conf. on Human factors in computing
systems, pp.257-264 (2003).
[12] J. P. Borst, N. A. Taatgen and H. Van Rijn: The
problem state: A cognitive bottleneck in multitasking. Journal of Experimental Psychology:
Learning, Memory, & Cognition, Vol.36, No.2,pp.363-382 (2010).
[13] S. T. Iqbal and B. P. Bailey: Investigating theeﬀectiveness of mental workload as a predictorof opportune moments for interruption. Proc.
CHI2005, pp.1489-1492 (2005).

[17] D. D. Salvucci and J. Bogunovich: Multitasking
and monotasking: The eﬀects of mental workload
on deferred task interruptions. Proc.CHI2010,pp.85-88 (2010).
[18] S. T. Iqbal and B. P. Bailey: Leveraging Characteristics of task structure to predict costs ofinterruption. Proc. CHI2006, pp.741-750 (2006).
[19] T. Tanaka and K. Fujita: Study of User Interruptibility Estimation Based on Focused Application Switching, CSCW2011 (2011).
[20] J.G. Trafton, E.M. Altmann, D.P. Brock andF.E. Mintz: Preparing to resume an interrupted
task: Eﬀects of prospective goal encoding andretrospective rehearsal, International Journal of
Human-Computer Studies, Vol.58, pp.583-603(2003).