# Welcome, Computer! How Do Participants Introduce a Collaborative Application During Face-to-Face Interaction?

Mateusz Dolata, Susanne Steigler, Fiona Nüesch, Ulrike Schock, Doris Agotai, Simon Schubiger, Mehmet Kilic, Gerhard Schwabe

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Mateusz Dolata, Susanne Steigler, Fiona Nüesch, Ulrike Schock, Doris Agotai, et al. Welcome, Computer! How Do Participants Introduce a Collaborative Application During Face-to-Face Interaction?. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.600-621, 10.1007/978-3-030-29387-1_35. hal-02553891

HAL Id: hal-02553891

https://inria.hal.science/hal-02553891

Submitted on 24 Apr 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Welcome, computer!

## How do participants introduce a collaborative application during face-to-face interaction?

Mateusz Dolata<fnref n="1" />, Susanne Steigler<fnref n="1" />, Fiona Nüesch<fnref n="2" />, Ulrike Schock<fnref n="2" />, Doris Agotai<fnref n="2" />, Simon Schubiger<fnref n="2" />, Mehmet Kilic<fnref n="1" />, and Gerhard Schwabe<fnref n="1" />

**Abstract.** During cooperative interaction, participants introduce materials, artefacts, and other individuals into the ongoing interaction. Depending on how this introduction unfolds, the participants may embrace the new element in an easy way or not. If the new element is a collaborative application of interactive software designed to support the interaction, it may or may not improve the collaboration because of how it was introduced. Therefore, understanding and designing the initial interaction is key for unleashing the positive impact of collaborative systems. The literature has identified the fact that humans employ a specific range of behaviors when introducing an element into an ongoing interaction. Those introduction rituals are determined by whether the new element is a human or a material artefact. Introduction rituals involving interactive elements are still underexplored: How do participants introduce and initiate interaction with them? This manuscript explores the introduction behaviors emerging when an augmented-reality collaborative application is being introduced into a financial advisory service. It shows that the participants employ a wider range of introduction rituals during the introduction of this application than they do when they introduce a brochure. Notably, many of the observed behaviors resemble familiar opening rituals typically used when introducing and greeting humans. This supports the *computers-are-social-actors* argument and provides evidence that introducing a collaborative application has a social rather than a material character.

**Keywords:** Advisory Service Scenario, Mixed Reality, Augmented Reality, Collaborative Applications, Rituals, Computers-are-social-actors.

## 1 Introduction

When collaborating humans engage in a range of rituals, which at first glance may appear insignificant. However, those ancillary activities set the context for the interaction and require effort from participants, but are rarely acknowledged as work. Consider a group brainstorming: would you, as a participant, pay attention to the distribution of blank post-its at the beginning? Consider a lively group conversation: would you observe the shaking hands ritual when a person joins the setting? Trivial acts like distributing post-its or shaking hands, seem out of scope – the participants’ focus is on the

<footnote n="1">¹ University of Zurich, Zurich, Switzerland</footnote>

<footnote n="2">² University of Applied Sciences and Arts Northwestern Switzerland, Windisch, Switzerland</footnote>
*actual work*. Nevertheless, ancillary activities are key: brainstorming needs post-its, or an equivalent, and greeting a person in many cultures involves a handshake.

Especially when introducing and greeting another person, humans engage in ancil-
lary activities, which seem necessary but are unrelated to the content or topic of the
conversation. Handshakes, smiles, and mutual gazing follow each other in a synchro-
nized order and the participants execute them without explicit effort or conscious focus.
However, if one does *not* engage in those acts, others might take it as an offense or
antisocial behavior. In such moments, when something does not work as expected, hu-
mans notice that those ancillary activities are a crucial part of the ‘real work.’ If one
forgets people’s names when greeting them, it is as if there were not enough post-its
for everyone during brainstorming. It seems that some performances, though apparently
superfluous, are essential for subsequent interaction between humans.

Goffman frames those routine, socially expected and formulaic performances as in-
teraction rituals [27]. He provides evidence that those rituals solidify the feelings of
regard and respect for oneself and other participants, thus setting the stage for effective
interaction [70]. By describing the extraordinary richness of interaction rituals in many
situations, like introductions or greetings, Goffman draws a picture of ancillary activi-
ties as ordered and rich performances which are integral to the social interaction rather
than simply accompanying it [27, 40]. He moves the ancillary activities to the center of
attention and, by using ritual as metaphor, adds a touch of sacred, spiritual meaning to
social interaction between individuals [40]. We follow up on Goffman’s concept of an
interaction ritual as distinct actions, which order an interaction and allow the partici-
pants to present themselves in a desired manner. In this article, we focus on the discrete
rituals involved in introduction sequences.

Introductions do not involve only humans. Humans may introduce new material,
new collaboration tools, or new software when interacting with others. However, only
a handful of researchers analyze introduction rituals related to materials or technology
in a face-to-face interaction. Even fewer app and technology developers evaluate their
designs from this point of view. This holds true in particular for collaborative systems:
studies highlight the advantages, usage patterns, and appropriation moves, but the act
of introducing the application into the ongoing interaction remains obfuscated. It seems
like collaborative applications were simply present during collaboration – anything else
remains an ancillary activity. Consequently, we have a limited understanding of what
rituals humans employ when a collaborative application is being introduced into an
ongoing face-to-face interaction. The main objective of this study is to describe intro-
duction rituals, i.e., singled out, routine and formulaic activities of individuals partici-
pating in a collaborative scenario, which feature the introduction of an IT-based collab-
orative application. Additionally, the study aims to compare the identified rituals with
introduction rituals described in the literature or observed in non-IT settings to give the
reader a better sense of how they are performed. In detail the study explores the rela-
tionship between rituals emerging around a collaborative application and those involv-
ing interaction with humans, where the performances provide a way to show respect to
someone. Making rituals explicit and accessible can help engineers to embrace and im-
plement them in the design of collaborative software.
Based on the analysis of video recordings from financial advisory services, this study identifies introduction rituals that occur when the advisor introduces a collaborative application into an ongoing advisory service. Comparison to the act of introducing a common material artefact (a brochure) reveals that introducing the collaborative application makes use of a wider range of interaction rituals. Comparison to the literature on introduction suggests that when introducing computer software applications, the participants tend to employ rituals otherwise typical for human introductions and greetings. Introduction of a brochure does not exhibit such tendencies. The study employs multiple perspectives to provide a rich description of introduction rituals and identify which features of the collaborative application afford those rituals.

The study builds upon previous discourses in research and practice. It adds to the discourse on interaction with IT in collaborative settings by supporting the *computers-are-social-actors* argument: participants embrace a software application from the very beginning, from the act of introducing it, as a social actor [55, 66] and produce performances that are normally used to show respect to other humans. This complements previous research on introducing and initiating use of a computer in an institutional setting, e.g., between a doctor and a patient [60]. It also adds to the conversation analysis efforts to understand the role of materials in coordinating communication between humans [16]. It should sensitize designers to the importance of introductions and provide them with stimuli and patterns to think about how users might proceed while using their application during collaboration. In addition, practitioners such as frontline employees at banks and other institutions may benefit from a deep analysis of micro-level behaviors and better understand how those behaviors affect the client. Overall, the study provides insights that may contribute towards software design and behavioral research.

# 2 Related Work

This paper intends to explicate introduction rituals involving a collaborative application and to compare those rituals against the backdrop of other introduction rituals just mentioned. This section provides relevant background starting with the most traditional form of introduction rituals involving an individual, and then introduction rituals involving materials such as brochures or sheets of paper (being the most widely used collaborative resources), and finally the scarce literature covering the introduction of IT. Whereas the literature acknowledges the ritual nature of introduction between humans, material and IT-focused performances are presented as material behaviors, even though they order the sequence of an interaction and let the humans keep up appearances, thus matching the characteristics of rituals.

## 2.1 Introducing a person

Conversation analysis (CA) studies established a solid basis of knowledge about how people initiate interaction with other humans [41, 63, 68]. The literature differentiates between two types of introductions: self-initiated introduction (when a person introduces oneself) and other-initiated introductions [12, 62]. Other-initiated introductions
are situations in which a third person person, such as a mediator, introduces a newcomer to a pre-present person or pre-existing group of people, i.e., the individuals who had interacted with the mediator before the newcomer arrived [62]. CA studies identify a range of ritualized verbal and non-verbal behaviors which accompany the introduction of a person [22, 49, 51, 52, 56, 64].

When introducing a person, humans use a range of non-verbal and verbal cues. Whereas cultural differences may occur, here we consider introductions among Europeans and Americans. Imagine a situation that might happen at a conference. Ann and Marc are enjoying a conversation during a coffee break, while Bob, a third colleague, is getting closer to them. Bob is approaching while signalizing an intention to join the conversation with gaze and movement [64]. Ann knows Bob and steps in the role of the mediator. According to CA studies, the interaction is likely to unfold as follows: Ann changes her position such that the three will form a circle [14, 41]; Marc adapts his position as well [62]. Ann makes a deictic reference to Bob (like “This is Bob”) including details relevant to the context (job, position, name, etc.) [61, 63]. The deictic utterances involve specific gestures: Ann gestures toward Bob when introducing him [62]. Marc follows the indication and looks at Bob. Thereafter, Marc is likely to initiate interaction with Bob using a greeting (“Hello”, “Good morning”) and an adequate bodily reaction (a handshake or a short wave) [56, 62]. He might smile [22] and adapt a welcoming posture [41] – mutual gazing and palms turned open [22]. A range of verbal behaviors comes thereafter, including introduction-specific assessments (“nice to meet you”), opening-specific utterances (“how are you?”), and repetition of the name of the introduced person (Marc saying “Hi Bob”), as well as emotional semi-language (“oh”, “ah”, “well”) [62]. The sequence takes few moments and the performances are well coordinated, even among strangers [62, 63]. Introductions have a ritual character similar to theatrical performances – all participants seem to know what to do, such that the interaction unveils smoothly [22, 29]. Furthermore, the participants behave in a respectful way and produce multiple signs of respect (positions, postures, emotional or polite utterances), which underscores the fact that rituals reinforce social order and signalize the mutual recognition between the parties [27, 40, 70]. The ritual relies not only on the mere occurrence of particular behaviors, but also on the relations between them: their sequence, mutuality, and the links between the verbal and non-verbal conduct [51, 52].

The configuration involving a mediator (Ann), a newcomer (Bob) and a pre-present participant (Marc) resemble what happens when a computer enters the stage in an advisory service. The advisor, as the one who knows the system and hosts the interaction, takes on the role of the mediator and introduces the collaborative application to the pre-present client. However, it remains unclear whether and to what extent insights about the introduction rituals related to a person can be transferred to a situation where a material, -- a collaborative application or any form of interactive IT -- gets introduced.

## 2.2 Introducing a material artefact

Whereas introduction among humans has been studied for decades, studies on how individuals introduce new material into an ongoing interaction arrived much later [48,
52]. Only a few articles consider the introduction and use of material artefacts in infor-
mal collaboration [53]. The primary focus lies on institutional settings ranging from
academic supervision [54, 74] to performance appraisal interviews [48] to doctor-pa-
tient encounters [11] to the financial advisory services at a bank [16]. Given that paper
(brochures, forms, notepads, etc.) remains the most widely spread material in institu-
tional collaboration, most insights refer to the use of paper [16, 50, 74]. The analyses
encompass the use of material in various situations as a whole and only rarely address
a particular phase of interaction like beginning or closing [9, 52]. Instead, most CA
studies relate the material rituals to conversational processes like turn taking or activity
shifts [28, 31, 52, 54, 74]. This study takes a different path and focuses on the various
verbal acts that accompany the initial use of an artefact.

Taking together the insights and observations from previous literature, one can iden-
tify some verbal and non-verbal behaviors typical for initial use of material artefacts.
Given the fact that material usage in student counselling has been extensively re-
searched [30, 73, 74], we explore that example here. There is solid evidence that similar
behaviors may occur in financial services [16] or in doctor-patient encounters [11].
Consider a situation involving Barbara, a university teacher, and a student, Mike, sitting
at a table in a university office and discussing Mike's seminar thesis. Barbara has pre-
pared by putting away documents that cluttered the table beforehand [30, 74]. While
Mike explains what he has done so far, Barbara listens, first keeping a pen in her hand,
and then leafing through a pile of documents to her left, while encouraging Mike to talk
further [74]. She takes two pieces of paper and organizes the remaining ones back into
the pile [74]; she places the two sheets of paper in the middle of the table, next to each
other [74]. Mike stops talking and Barbara says: "Okay, I get it. Well, let's take this
first; you can read it, right?" while pointing to one of the documents. "Then we will go
through that one. Maybe I remember it" – she continues while pointing to the other
document. Mike nods and responds "mhm"; they continue while looking at the docu-
ments. This short scenario comprises typical behaviors that characterize the use of ma-
terial in institutional collaboration [11, 16, 30, 73–75]. In particular, documents – even
if central to the ongoing activity – enter the stage without explicit introduction. Instead,
they are used more as an "excuse" to move to the next activity. Rituals involve ordering
the documents (in the pile and on the table), as well as keeping up the ongoing interac-
tion despite the act of introducing a material.

## 2.3 Introducing a machine

The introduction of IT into an ongoing face-to-face interaction attracted researchers' attention much later. The research splits into two streams: first, studies oriented around the introduction of “traditional” desktop computer [7, 20, 59, 60, 76] and second, those dealing with human-robot interaction [21, 25]. There is little overlap between the two streams: On the one hand, studies on the introduction of desktop computers appear in healthcare informatics [60, 76], studies of IT use in public domain [2, 45], or IT-oriented workplace studies [46]. On the other hand, studies from human-robot interaction and from robotics attend to the initial interactions between humans and robots in
the process of developing new technologies and applications [65]. Since robots are a
new phenomenon, researchers are still in the process of exploring and designing the
initial interactions – rituals still need to emerge. Therefore, we focus on introduction
rituals involving collaborative applications using state-of-the-art interfaces.

Computers entered doctors' offices and public agencies years ago. Recall a typical
interaction with a doctor (or a frontline employee, etc.). Let's call her Eve. Perhaps,
Eve has a monitor standing on her desk, the display turned toward her [2, 60]. There is
also a mouse and a keyboard on the table [58, 60]. Eve interacts with her conversation
partner, but from time to time, she turns towards the computer; she also types things
using the keyboard or uses the mouse [60, 76]. The computer remains Eve's "private"
tool during the interaction; the patient/client may see some things on the screen, but
they remain outside the scope of the conversation [59]. This is so until Eve turns the
screen and presents an interesting artefact shown there, let's say, an X-ray image. She
points to the display, says "This is your leg, you see - there is your injury from the past"
and uses her pen to pinpoint a specific area in the image [2]. Eve and her conversation
partner move closer to the screen and look at the picture, then at each other [60]. Eve
continues talking, while the other party nods and acknowledges with "uh huh" [19].
Situations like occur millions of times every day. The desktop computer does not need
an introduction: it is already there, on the table [1, 13, 57]. Nevertheless, Eve employs
deictic gestures and words when using the computer for the first time as a collaborative
resource. The rituals she engages in involve deictic utterances and rotation of the screen
analogous to the introduction of a leaflet as described before.

Given the framing of computers as a collaborative resource, this behavior seems ad-
equate. However, it contrasts with an alternative popular view on computers in social
interaction: *“computers-are-social-actors”* (CASA) [55, 66]. Research following this
paradigm claims and provides evidence that humans tend to treat technology like they
would treat other humans, thus applying the heuristics and behavior schemata used for
humans, to computers. For instance, humans may talk to their computers or assume
human intelligence behind a computer’s actions. During the introduction, one could
expect behaviors akin to those of Marc, Ann and Bob rather than those of Barbara and
Mike. But the literature on introducing computers into collaboration does not analyse
the introduction sequences in terms of the CASA paradigm. The scarce evidence in-
cludes no indication that human participants treat computers in any special way.

3 Designing for introduction rituals

3.1 Context: Status quo in IT-supported advisory services

An advisory service is a collaborative encounter between a professional advisor and an
advisee [23, 38]. The participants differ in terms of their knowledge, institutional iden-
tity and interaction rights. The client knows about his<fnref n="1" /> problems in the initial situation,

<footnote n="1">¹ To guarantee a balanced gender representation and readability of the manuscript, we refer to the advisee as a male (he, his, him) and the advisor as a female (she, her).</footnote>
which can be solved with the advisor's expertise on the potential solutions [39]. Advisors are in charge of distributing interaction rights and dominate the encounter in verbal [17, 32] and material [17, 74] terms. It is the advisor who distributes brochures and documents across the table and invites the advisee to contribute [43]. In the current study, we focus on mortgage advisory services of a Swiss local bank.

The Swiss real estate market depends on mortgage loans. Clients attend to advisory services if they fulfill (or they think they fulfill) the minimum requirements, e.g., 20% of house price available, or they identified a property to buy [10]. During an advisory service, the advisor first learns about the advisee's wishes and situation, then she explains the available products and makes a recommendation [38]. In a traditional mortgage advisory service, the participants first consider the chosen property, and then they assess the advisee's financial capability and eligibility for a mortgage. If those are met, the advisor describes the possible range of choices (fix-rate and flexible mortgage, with and without amortization) and prepares a composition that fits advisee's needs.

Rituals bind the various elements of an advisory service together, such that the participants, who never met before, establish a coherent encounter [8]. Many rituals relate to first impressions and the management of expectations [27]: a bank advisor dresses in a manner signaling her role [26, 36], positions documents in an orderly manner [16], manages mutual eye gaze and shows interest in advisee's spoken words and body language [42]. However, introducing a collaborative application is not an established ritual. Research shows that advisors and advisees try to embrace a new system ad-hoc and establish clumsy interaction protocols [42]. As a consequence, more and longer pauses occur than in a conventional encounter [17]. Overall, the literature suggests that initiating the use of a collaborative IT system during an advisory service yields unintended interruptions and destroys key rituals.

We propose the following interpretation of the problem: The systems presented in previous research neither fit the typical rituals of advisory services (such as advisor presenting oneself with a business card [16], advisor introducing the central topic with a drawing or leaflet [16], or advisor showing knowledge about the property to be supported with the mortgage and its location [16]), nor do they explicitly afford an alternative ritual that could accompany the introduction of the collaborative application. As a consequence, the idea of an advisory encounter embodied in routines, roles and expectations falls apart and leaves the participants in an improvisation mode.

To address this problem, we launched a project to develop a system aligned with the conventional advisors' practices. Since those practices rely on paper, pen and brochures, the designed system handles physical interaction. On the other hand, to enhance the interaction, the system allows, e.g., for dynamic adaptation of graphics, which is impossible with "normal" paper. An article, which describes the system in more detail, is under review. Herein we focus on the initial interaction with the system and thus, review only the elements used to initiate the application as a collaborative resource.
## 3.2 Technology: Mixed reality for physical interaction

Mixed reality (MR) research has long studied integration of embodied and physical interaction with digital processing. It addresses the topic of digital and material reality in a way that allows for full use of space and body during interaction with digital content [18], thus making it possible to support specific rituals. MR embraces a set of research directions. Augmented reality (AR) studies how to overlay physical spaces with computer-generated content [3, 5]; spatial AR uses projection to put the content directly into the environment of the user [6, 15], however it requires powerful and high-fidelity overhead projectors and affordable 4K beamers, which arrived on the market only three years ago. Organic user interfaces focus on projecting content onto non-planar spaces that users can interact with through bending, folding, and manipulating the form [33, 34]. Tangible user interfaces focus on enabling interaction, manipulation, and collaboration [69] with digital content through physical objects and space [35, 79]. Pen-and-paper user interfaces try to bridge the gap between digital interaction and paper, which remains an essential tool in many human activities [47, 71, 72]. All those areas of research share an interest in enhancing the physical world with digital content and functionalities to enable natural interaction with objects, avatars and other people [4]. The subsequent section explains how MR-inspired features were leveraged to enable introduction of the collaborative application during a mortgage advisory service.

## 3.3 Design: Mixed reality for introduction rituals in advisory services

Observations and preliminary interviews with the advisors revealed that a critical point during the beginning of an advisory service is the first mention of the property. When switching from opening small talk to the core of the service, advisors often address the property in various ways [16]: an advisor demonstrates her knowledge about the specific neighborhood to give the impression of an interested and informed professional; another advisor asks questions about the private motivations behind the purchase to better understand the client's emotions; and other advisors simply engage in a courteous discussion concerning the property to establish rapport with the client. So far, the advisors supported this part of the conversation with no material or, at most, with printouts of basic data about the house, such as an advertisement. This contributed little to the content of the conversation. Those practices inspired the design team: on the one hand, there was potential to make the conversation more informative by introducing additional information resources; on the other hand, it offered an opportunity to introduce the IT as a collaborative application before the participants move on to hard math tasks like calculating the mortgage rate and interest. Taking into account the possibilities of mixed reality, the designers proposed a *house token & map* design.

The *house token & map* builds on and extends mortgage advisory service. It envisions the following interaction: When the advisor wants to shift the conversation topic to the property, she uses a 3D model of a house made out of concrete and puts it in the middle of the table. She can zoom in or out and switch between map and satellite view. When moving the house token around the table, she moves the whole projection with it, such that the house token always marks the location of the property (based on its
address). The map allows viewing of the facilities in the neighborhood, like schools, grocery shops, or train stations to fuel the conversation about the property. The *house token & map* design sources from existing practice, but extends it and moves the focus of conversation beyond numerical and technical values (size, number of rooms, year of construction, floor covering, etc.) or purely emotional topics (motivation to buy) to the location. This design augments the conversation rituals with additional data and inspires a different outlook on the property. We developed a system that supplied the *house token & map* functionality at the beginning of service, just after small talk.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Map projection with the concrete house token in the middle.</figcaption></figure>

# 4 Methodology

## 4.1 Design and implementation

LivePaper is a collaborative application for financial advisory services. It uses paper practices and augments them with projected content. The system was developed by a Swiss local bank and two universities in a joint project intended to support client-centric advisory services. It provides the means for easy generation of documentation, as well as streamlining the overall process. The whole project followed the research through design paradigm [24, 78]. Starting with practical problems, solutions were developed in a creative and iterative manner based on fieldwork, technological potentials, and models of advisory services and human interaction (e.g., theories about human introductions and greetings). Scenarios as well as clickable and functional prototypes were used to describe problems and solutions [67] and to conduct intermediate testing with key users, according to constructive design [44]. Scenario-based design and constructive design research have been widely used in Human-Computer Interaction (HCI) [77].

The *house token & map* design was integrated into LivePaper as one of its components. LivePaper uses a Kinect sensor to track objects on the table (sheets of paper, tokens) as well as participants' gestures. A 4K projector is used to project the content onto the surface of a table. Based on recognition of objects and hands, it can adjust the
position of the projection, such that the content moves around together with the ele-
ments on the table. The same principle was applied to implement the *house token &
map*: thanks to its form and a reflective sticker, the token could be recognized and its
location tracked; the map appeared as soon as the token was placed in the projection
area on the table between the advisor and the advisee; the map functionalities (zoom in,
zoom out, switch view) were linked to buttons projected next to the map.

## 4.2 Evaluation

The developed prototype was tested in a realistic scenario to observe what kind of in-
troduction rituals it affords. In particular, 24 non-IT and 24 LivePaper-supported ser-
vice encounters were carried out within two weeks. They involved 8 professional bank
advisors from the partner bank and 24 test advisees. The advisee participants were ac-
quired through an announcement on a university platform (quota sampling). They were
between 17 and 71 years old (avg. 32). There were 12 males and 12 females. Partici-
pants came from diverse professions, e.g., lawyer, therapist or architect. They received
a short introduction to the setting and a realistic scenario to act upon. It included rough
data about their financial status and data about two properties to be discussed with the
advisor (one property per treatment). Each participant attended a LivePaper-supported
and a conventional consultation. After both treatments, each advisee participated in an
interview to report on their experiences and compare the two treatments. The experi-
ment took place in two branches of the partner bank.

Overall, even though the observed advisory services were not real (the advisees were
not real clients), by setting the context, scenarios and working with actual bank em-
ployees, we established a realistic situation for both clients and advisors. The advisors
were neither aware of the property that the advisees received before the consultation,
nor did they see the advisee beforehand. This preserves high ecological validity. In the
conventional treatment, they used a calculator, notepad, and pen. In the LivePaper treat-
ment, they employed the system designed as described above. Consequently, the task
that participants needed to complete during the experiment was representative of and
similar to what advisors and advisees do in actual advisory services at a bank, thus
supporting the face validity. However, we acknowledge that external validity and trans-
ferability of the results beyond the scope of mortgage advisory services requires further
research, as these experiments explicitly were designed to capture the complexity of
mortgage advisory services.

The advisors used the *home token & map* prototype to switch between opening small talk and the core part of the encounter. Thereby they also introduced the system into an ongoing interaction: *home token & map* was the first feature of the system used during an advisory service. The advisors were trained in two iterations on how to handle the LivePaper: a half-day workshop one week before the test and a short training at the day of the session. The prototype training focused on use of LivePaper, but the advisor was free to decide when and how to introduce it to the interaction.

The primary data used for the analysis of system introduction sequences was col-
lected through video recording. We recorded each advisory service with three cameras:
from above, from the side and facing the advisee. Consequently, we captured the con-
duct on the table, bodily motions, and verbal contributions of both participants, as well
as the advisee's mimics. This allows for blow-by-blow study of the verbal and non-
verbal conduct of how the participants introduce and accommodate the collaborative
application. Observation of the conventional, non-IT treatment allows for comparison
of rituals and performances between the two settings.

4.3 Data analysis

The current study focuses on the initial interaction between human participants and
LivePaper in mortgage advisory services. A set of qualitative methods is used: in-situ
observation, coding of the video-recorded consultations, transcription of the interac-
tion, and the interview analysis. For LivePaper treatment, the video-coding and the
transcription focused on the 15 seconds surrounding the introduction of the system, i.e.,
putting the token in the center of the table. For conventional settings, the analysis fo-
cused on the introduction of a brochure as the first artefact of work. In both treatments,
introduction occurs after the initial presentation of the concern by the advisee, thus
yielding comparable episodes. The limit of 15 seconds was set arbitrarily, but consid-
ering time frames used in earlier studies on human introductions [41, 62].

These introduction episodes were first observed for all 48 consultations to create an
ethogram of all performances. It considered body posture, face gaze direction, hand
movements and gestures, semi-language (oh”, “ah”), utterances and mimics. These ep-
isodes were then encoded in the video sequences. Twenty four representative excerpts
were later transcribed and annotated with multimodal data according to the Jeffersonian
notation [37]. The researcher conducting the coding and multimodal transcription dis-
cussed the analysis methods and results regularly with two experienced researchers.

Apart from video analysis, interviews were used as additional source of insight. A
researcher coded the 24 advisee interviews focusing on statements concerning the open-
ing of an encounter, the first use of *house token & map*, as well as the first use of a
leafletlet. The coded segments were ordered according to how positive or negative they
were – the results include quotes of varying sentiments. Overall, the chosen methods
capture the reactions of the participants to the introduction of the collaborative applica-
tion and compare it to the introduction of traditional, material artefacts.

5 Results

5.1 Introduction sequences

The first excerpt (Table 1) shows how a female advisor introduces the system and how
a male advisee reacts. We join the participants in the seventh minute of the encounter
after the advisee has explained to the advisor what property he is interested in. The
advisor has finished collecting the key data in her notebook (including property's ad-
dress) and puts the writing utensils away. Thereby she clearly signals an activity shift
[48]. This shift is perceptible by the client, who stops looking at the property data sheet
in front of him and now observes the advisor. Advisor continues by verbally referring to "this property" and engages in several performances to introduce and use the Live-Paper for the first time in this encounter. Figure in Table 1a depicts this situation.

<table><caption>**Table 1.** Introduction of the collaborative application into an ongoing encounter according to the *home token & map* design (15 seconds).</caption><thead><tr><th></th><th>a. Indicating the end of the note-taking activity</th><th>b. First approach to introduce the system</th></tr></thead><tbody><tr><td>A: so yeah: this property [0.9]<br>A moves open palms away from the table and gazes at the table.<br>C remains in sitting position, his hands rest on the property data sheet he brought with him. A and C keep eye contact</td><td>A stretches her right arm far to the right, to the house token. Gaze follows her moves.<br>C: yeah: [1.0]<br>C sits upright and moves his sheets of paper away, to the left. Gaze is directed towards his sheets of paper.</td></tr><tr><td>A: he:re [.]<br>A stops her movement and moves back to the upright sitting position. While gazing at the notepad and smiling she casts the note-pad to her left, and creates a free area.<br>C: "mhm" [1.0]<br>C gazes at the pile to his left.</td><td>c. Interrupt and return to preparation<br>d. Introductory deictic turn<br>[...]<br>A: let's look at it he:re/ [0.8]<br>A continues to smile and places the house token with map projection on the table.<br>C gazes at the projection in the middle and rolls up his sleeves.</td></tr><tr><td>A smiles and gazes at C.<br>C: "wow" [1.0]<br>C switches gaze from the middle of the table to the paper to his left and continues to pile again.<br>A: [laughter] [0.8]</td><td>e. Making way and organizing space<br>f. Showing signs of interest<br>A gazes at the house token, moves the note-pad farther away to the left.<br>C: ah [1.2]<br>C smiles. Casts sheets of paper from left to right side and relaxes the arms, gazes at the projection.</td></tr></tbody></table>

First, the advisor searches for the house token and then stretches her arm to catch it. In parallel, the advisee starts preparing the space for something to come – he collects his documents, which had been distributed across two piles (Table 1b). However, the
advisor stops her action suddenly and returns to her earlier position for a moment. She moves her notepad away as well. At that moment, the advisor and the advisee simultaneously engage in de-cluttering the interaction space (Table 1c). We observe the de-cluttering sequences before the house token arrives in the middle of the table. In all LivePaper excerpts the sequences were mostly initiated by the advisor, who starts de-cluttering her side of the table, and the advisee joins in. In Table 1, the advisee starts moving away pieces of paper directly after the advisor turns to the right and stretches her arm towards the house token. In any case, the advisor engages in behavior which signals something new to come and the advisees react by removing paper and other objects from the space between them.

As soon as the middle of the table is clear, the advisor moves the house token from the edge of the table to the middle and positions it between herself and the client (Table 1d). She uses a deictic expression and points to the house token or the map. The observed deictic expressions include “here”, “this”, “there” and “where.” Often the advisors refer to the token as “this property,” “your house.” So does the advisor in the excerpt. The advisee reacts by engaging in several behaviors in parallel: he smiles, says “wow,” gazes at the projection, moves towards the map and, as observed in this particular, case rolls up his sleeves (Table 1f). In other excerpts advisees engage in similar actions. Some advisors reorganize the interaction space even more before they proceed further with the advisee or with the system (Table 1e). The advisor alternates her gaze between the projection and the advisee. She reacts with laughter, smiling, and assuming an open posture. Having introduced the system, the advisor begins discussing the property’s location. No hesitation or long explanation phases occur – the participants continue the interaction naturally. The conversation about a property’s location may take from 30 seconds up to three minutes depending on the advisee’s interest. In many cases, advisees refer to the map not only verbally, but also by pointing or even touching the surface when discussing particular elements (e.g., the way to the nearest train station; Figure 3). After the introduction, the advisory service continues: participants discuss the options of a mortgage credit and other important features. Finally, the advisor creates one or several offerings for the client. The overall service takes up to 45 minutes.

<figure><img src="image_4.png" /><figcaption>**Table 2.** Introduction of a leaflet into an ongoing service encounter (15 seconds).</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>c. Interaction with the leaflet</figcaption></figure>

A: do we have it here [1.2]
A flattens the folded leaflet with both hands. Gaze is directed towards the leaflet.
C leans forward and gazes short at the leaflet. His left hand stays calmly on his paper.
A: different options [1.0]
C: 'yeah'
A moves right hand back. C and A keep eye contact. C's hands support his posture.

The second excerpt (Table 2) illustrates how a leaflet gets introduced in a conventional setting. One observes several introduction routines here as well. The advisor uses deictic language and gestures when putting the leaflet on the table, using a single noun to describe it (“roadmap”; Table 2b). Also, she sends out signals of an upcoming activity shift (Table 2a). However, those actions are not linked to reactions from the advisee: he does not declutter the space or spend much time gazing at the introduced artifact or say “wow” or “ah.” He remains still and focused on the advisor. Even though the advisor employs some means to make the leaflet easy to read for the advisee (she flattens it, puts it in a way such that both can read it and easily point to it), the advisee’s focus remains on the advisor. He keeps his hands on the table. Overall, neither the advisor nor the advisee prepares the table or make way for the common artefact to come.

## 5.2 Introduction experiences

When asked about their experiences and comparison between the two settings, the advisees often explicitly refer to the introduction of the system as a special moment. An advisee remembers the opening: “*and then he places the little house and a map is shown and (...) sees everything in total – you can zoom in and out. This is an incredibly great opening. Something like this, is incredibly amazing, I was perplexed*” (K18). While many clients confirm this effect, some doubt that it has impact on the pragmatic side of the advisory service: “*First you see the house where it is located, just this. I mean, it is really nice that I can see it already. Just «wow»! But it is not required for the business*” (K05) or “*It was pretty chic at the beginning with the house to see where it is. But the practical benefit seems limited: I know where the property is I am going to buy*” (K14). Still, when explicitly asked for their feelings, many advisees see the introduction as a relevant phase to generate positive emotions: “*The emotions were stronger with Live-Paper because we have looked at the object and where it is and how is the environment. We have zoomed out and in. During the first conventional advice, the advisor has not even asked for the street. It was really just about the purchase price and that's it. (...) That you yourself see the object again. That you are also really sure that you really want to buy the object. This is what makes you have a positive feeling*” (K15).
The advisees liked the choice to use a 3D model of a house as a token made of concrete. “You can relate to that (...) that may be a play-house, but it attracts my attention or places me in the atmosphere or environment of this house” (K25). However, this decision also generated problems. As an advisee explains: “So it was great to see where is the property is - bird's eye view. What bothered me: it would be better if the property was not covered by the concrete house token. That was right in the middle of it. This means that one does not have a chance to look at the actual property on the map” (K05). Even though the clients notice limitations of the design, they acknowledge that it positively impacted their emotional engagement in the initial interaction.

The advisees like the fact that the introduction of the system happens with an interesting topic. An advisee puts it as follows: “The advisor was showing interest in my property (...) actually more with the LivePaper. This was so because he was really asking where it is and where it has been displayed on the map, (...) where it is exactly and what location it has, actually more with the Life Paper. The conventional service is more so serious: «I have the house and I want it and it costs so and so much»; and then «okay, let's start to calculate»” (K06). The observations of the frequency of advisees' positive reactions during introduction episodes confirm the overall emotional attitude of advisees towards the interaction around the house token – see Table 3.

<table><caption>**Table 3.** The frequency of specific behaviors in the introduction sequences (15 seconds).</caption><thead><tr><td>Behavior</td><td>LivePaper</td><td>No IT</td></tr></thead><tbody><tr><td>C smiles</td><td>11</td><td>0</td></tr><tr><td>C bends over the table</td><td>20</td><td>10</td></tr><tr><td>C declutters the table</td><td>17</td><td>0</td></tr><tr><td>C moves hand to the middle</td><td>4</td><td>0</td></tr><tr><td>C says “oh!” or “ah!”</td><td>3</td><td>0</td></tr></tbody></table>

# 6 Discussion

This paper's objective was to describe the rituals involved in the introduction of a collaborative application and to compare them to other typical introduction rituals. The above analysis points to key differences concerning the introduction of collaborative material artefacts and the introduction of LivePaper, which uses the *home token & map* design. When introducing a leaflet, the advisor selects the correct leaflet from a folder, carefully positions it on the table, assuring that she and the client can read it, and verbally encourages the client to look at it and points at it with her hand. This sequence of actions resembles what earlier research described for a range of institutional situations [11, 16, 30, 73–75]. Also, the reactions of the client are not much different from what was already observed: he bends over slightly, looks at the leaflet and/or at the advisor, he shortly confirms his interest with a short “yeah.” We can confirm what literature has already stated: material supports activity shift and the advisor can use it to move the focus of the client to a specific topic, as well as to push the conversation along further
[28, 31, 52, 54, 74]. Perhaps because the introduction of the leaflet as the collaborative artefact is so ordinary, no participant remarks on it explicitly in the interview.

It is quite the opposite with the *house token & map*. Many interviewees refer to the house token, map or their introduction even if not asked directly. In fact, advisors and advisee behave extraordinarily when they initiate the interaction with LivePaper using the *house token & map* design. In those introduction sequences, the advisor not only produces deictic utterances and positions the token carefully on the table, but also prepares the space necessary for it, smiles, and rearranges the space again when the projection is visible on the table. She refers to the house token as a “property” and repeats some deictic reference when presenting it (“here,” “it”). Even without comparison to the reference literature, one observes that the advisor puts forth additional effort during the introduction sequence. She behaves as if she was preparing the stage for an important element to enter the ongoing interaction. This goes beyond the usual rituals of introducing a material artefact [11, 16, 30, 73–75] or an artefact on a desktop computer’s monitor [13, 19, 45, 46, 57–60, 76]. In particular, it is not a straightforward turn of a monitor [2] or a positioning of material [74]. The interaction with the house token is a theatrical performance and involves emotional gestures from the advisor, such as showing both palms at the beginning of the performance or repeated pointing to the token. The client’s reactions confirm that the situation is unusual: he declutters the space, smiles, bends forward, and sometimes even dares to make physical contact with the application. Also, we observe some emotional semi-language (“oh!”, “ah!”) in addition to the more usual “yes,” “mhm”, and “okay.” Those reactions do not resemble observations reported for the introduction of material artefacts or artefacts displayed on a desktop monitor [13, 19, 60]. Something essential is different.

We argue there is a shallow and a deeper explanation for the reported results. The shallow interpretation relies on comparison between the typical human introduction as analyzed in ethnomethodological studies [14, 41, 52, 56, 61–64]. They share a lot: participants adapt new, open positions; they are directly referring to the newcomer by explaining meaning and background; there is an intensive gazing towards the newcomer; and, finally, there are welcoming gestures, touches, and emotional semi-language expressions. It seems that the *home token & map* design affords rituals otherwise employed when introducing and greeting humans. But what makes the simple design elements (a concrete house model and a map) more *social* than the leaflet?

The *house token & map* design, in fact, implements some features that make the system and its elements easier to embrace by the participants than the leaflet (or any generic element like a piece of paper or a desktop monitor):

* The house token is *meaningful based on its form and has a clear relation to the content of the ongoing interaction*. It simply stands for the property, which the advisee is going to buy. The advisee can easily decipher the meaning of the object when the advisor moves it to the center of the table. It is similar to what happens when a human newcomer enters an ongoing interaction: the pre-present participants perceive the newcomer’s intention to join based on the context and her or his apparent role. Before the newcomer enters, the pre-present participants plot a story in their minds about what the newcomer’s intentions and expected behaviors are. Think of a
situation when a beggar approaches someone on the bus stop with their hand stretched out – one can easily predict what to expect. This is not possible when a monitor gets turned or when a brochure enters the stage during an advisory session: those elements do not provide enough data for the advisee to craft a meaningful plot.

* The house token and the map *afford deictic, physical references*. It seems natural to the participants to point at the house as a physical object and to denote locations with “here” and “there.” It resembles what happens when a newcomer joins an ongoing interaction, and someone introduces them: pointing toward someone (or something) in our physical space belongs to the standard repertoire of communication. While pointing at objects displayed on the screen or in a brochure is possible as well, it requires more precise pointing, because these things are smaller and often more numerous (screens and brochures contain multiple images). Using a pen to point to a small element seems more natural then, while pointing with a pen to a person or larger object is unusual. Due to hand-sized elements *House token & map* promotes physical reference with gestures involving hands, fingers or a pen.

* Finally, the design *affords movement in space to signalize activity shift*. And this movement is very specific: the advisor moves the house token into the center of the table from the edge. This horizontal trajectory resembles the situation when a newcomer approaches a pre-existing group of people. When someone turns a monitor around or pulls a piece of paper from a pile or a folder, the object moves along a different path and this movement is shorter. In fact, starting the house token nearer the center was not possible because it would directly turn on the map and disturb the participants’ small talk. Admittedly, this technical limitation contributed to the theoretical character of the introduction sequence, but it does not explain such behaviors as decluttering the space or showing open palms. Overall, the advisor’s movements marked activity shifts and signalized a new element coming to the stage.

According to this shallow interpretation of the results, the above design decisions make participants attach a social character to the introduction of the *house token & map*. This would align with the “computers-are-social-actors” paradigm: the participants conduct actions otherwise typical for ritual introduction of a human actor. They prepare a dedicated space, produce emotional semi-language or attempt physical contact in a careful and respectful manner. It seems as if the participants were following typical schemata for introducing a (small) human individual. One could claim that the results even extend the previous paradigm. Originally, the CASA paradigm claimed that users transfer human attributes to technology because modern technologies use language as input and output, are interactive, or replace typically human roles [66]. The current results suggest that due to a hunch or intuition that an interactive technology might come to the table, users apply introduction schemata reserved for humans. This is surprising, given the fact that previous research on introducing computers has not reported similar observations [1, 13, 57]. There are two possible explanations for this difference: (1) previous studies have not topicalized the social character of the observed behavior or (2) humans treat computers like social actors only in specific circumstances. Identifying and describing those circumstances would require further research. This study suggests that the physical form and emotional value of the computer may play a role in this regard.
However, consideration of what a social ritual is and why it emerges provides a complementary, yet deeper explanation. Ritualized performances in interaction with other humans emerge not only through mindless repetition. Humans replicate rituals because, through those behaviors, they can show respect to the other party (or at least pretend to do so) [27, 40, 70]. This understanding helps interpret the results. For instance, an advisee's decluttering is a ritual response to advisor's efforts: when the advisee notices the advisor reaching for something or decluttering her side of the table, he is driven to acknowledge this by making space for what is coming. The advisee's smiles and emotional utterances show not only interest and pleasure, but also signalize acknowledgement of the advisor's efforts. Finally, advisee's bending forward or pointing towards the map do not follow only from curiosity in the content, but acknowledge the advisor's effort. Following this line of argumentation, the advisor's actions can be seen as a consequence of the physical affordances of the *house token & map* design and the reactions from the advisee as ritualized acknowledgment of the advisor's work. The collected data does not provide a conclusive answer as to which explanation is the right one. Of course, these two mechanisms (*computers-are-social-actors* and *ritualized respect signs*) are not incompatible; the observed behaviors can be driven by both of them.

# 7 Conclusion and limitations

The presented insights do not come without limitations. First, we rely on data collected in a realistic, but experimental context. While this helps us to compare the encounters and identify patterns in predefined conditions, it also generates questions about external validity. Replicating this study with clients in real advisory services outside the mortgage context could produce more insight, especially because of potentially stronger emotional engagement. Second, the results rely on field observations and video material. We report on observed regularities rather than making quantitative claims about them. Measuring the occurrences of particular performances might be the next step to further support the findings. Third, the transferability of the identified design rationales still needs to be confirmed. Analyzing whether doctor-patient or teacher-student encounters exhibit similar patterns requires design and observational research.

All in all, this paper provides a genuine, focused analysis of how participants introduce and “welcome” a collaborative application. The paper makes several contributions: It identifies atypical behaviors in introduction sequences for a collaborative application thus extending previous literature on initial interaction with computers [1, 13, 57]. It relates the observations to the implemented design decisions, thereby adding to the research on supporting advisory services [17, 42], which did not explicitly consider initial interaction or MR technology so far. Finally, it offers interpretation of the rituals in light of two social interaction theories and provides supporting evidence from realistic, controlled setting rather than from abstract experiments or anecdotes [55, 66]. It also compares those behaviors to the introduction sequences derived and consolidated from ethnomethodological research. The study highlights a crucial but neglected aspect of technology use – the “welcoming” phase, provides new perspectives on it, and offers both theoretical and practical interpretations of the observed behaviors.
References

1. Als AB (1997) The desk-top computer as a magic box: patterns of behaviour connected with the desk-top computer; GPs' and patients' perceptions. Fam Pract 14:17-23

2. Arvola M (2004) Shades of use: The dynamics of interaction design for sociable use. Linköping University: Department of Computer and Information Science, Linköping

3. Azuma RT (1997) A survey of augmented reality. Presence Teleoperators Virtual Environ 6:355-385

4. Bailenson JN, Blascovich J, Beall AC, Loomis JM (2003) Interpersonal Distance in Immersive Virtual Environments. Pers Soc Psychol Bull 29:819-833.

5. Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Found Trends® Human-Computer Interact 8:73-272

6. Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. CRC press

7. Brassac C, Fixmer P, Mondada L, Vinck D (2008) Interweaving Objects, Gestures, and Talk in Context. Mind Cult Act 15:208-233.

8. Broderick AJ (1998) Role theory, role management and service performance. J Serv Mark 12:348-361.

9. Broth M, Mondada L (2013) Walking away: The embodied achievement of activity closings in mobile interaction. J Pragmat 47:41-58.

10. Brown M, Guin B (2013) How Risky are Residential Mortgages in Switzerland? Schweizerisches Institut für Banken und Finanzen

11. Buse C, Martin D, Nettleton S (2018) Conceptualising ‘materialities of care’: making visible mundane material culture in health and social care contexts. Sociol Health Illn 40:243-255.

12. Chen FJ (1993) Social Status and the Sequencing Rules of Other-Introductions. Work Pap Educ Linguist 9:13-27

13. Christopher P, Pushpa K, Simon de L (2010) Getting Seamless Care Right from the Beginning &ndash; Integrating Computers into the Human Interaction. Stud Health Technol Inform 196-202.

14. Ciolek MT, Kendon A (1980) Environment and the Spatial Arrangement of Conversational Encounters - Ciolek - 1980 - Sociological Inquiry - Wiley Online Library. Sociol Inq 50:237-271

15. Cuendet S, Bonnard Q, Do-Lenh S, Dillenbourg P (2013) Designing augmented reality for the classroom. Comput Educ 68:557-569.

16. Dolata M, Schwabe G (2017) Paper Practices in Institutional Talk: How Financial Advisors Impress their Clients. Comput Support Coop Work CSCW 769-805.

17. Dolata M, Schwabe G (2017) Tuning in to More Interactivity – Learning from IT Support for Advisory Service Encounters. -Com J Interact Media 16:23-33.

18. Dourish P (2001) Where the action is: the foundations of embodied interaction. MIT Press, Cambridge, USA

19. Dowell A, Stubbe M, Scott-Dowell K, Macdonald L, Dew K (2013) Talking with the alien: interaction with computers in the GP consultation. Aust J Prim Health 19:275-282

20. Duke P, Frankel RM, Reis S (2013) How to Integrate the Electronic Health Record and Patient-Centered Communication Into the Medical Visit: A Skills-Based Approach. Teach Learn Med 25:358-365.

21. Edwards A, Edwards C, Westerman D, Spence PR (2019) Initial expectations, interactions, and beyond with social robots. Comput Hum Behav 90:308-314.

22. Firth R (1972) Verbal and bodily rituals of greeting and parting. In: La Fontaine J (ed) The Interpretation of Ritual: Essays in Honour of A.I. Richards. Tavistock Publications, pp 1-38
23. Fischer JE, Costanza E, Ramchurn SD, Colley J, Rodden T (2014) Energy advisors at work: charity work practices to support people in fuel poverty. ACM Press, pp 447–458

24. Gaver W (2012) What should we expect from research through design? In: Proc. Conf. Human Factors in Computing Systems. ACM, pp 937–946

25. Gehle R, Pitsch K, Dankert T, Wrede S (2017) How to Open an Interaction Between Robot and Museum Visitor?: Strategies to Establish a Focused Encounter in HRI. In: Proc. Intl. Conf. Human-Robot Interaction. ACM Press, Vienna, Austria, pp 187–195

26. Goffman E (1959) The presentation of self in everyday life, 1st Anchor Books ed., [rev. and expanded]. Anchor Books, New York

27. Goffman E (1967) Interaction ritual: essays on face-to-face behavior, 1st Pantheon Books ed. Pantheon Books, New York

28. Goodwin C (2000) Action and embodiment within situated human interaction. J Pragmat 32:1489–1522.

29. Goody E (1972) 'Greeting,' 'begging,' and the presentation of respect. Interpret Ritual Essays Honour Al Richards Lond 39–72

30. Hazel S, Mortensen K (2014) Embodying the institution—Object manipulation in developing interaction in study counselling meetings. J Pragmat 65:10–29.

31. Heidtman D (2010) Zur Herstellung von Situationseröffnungen in Arbeitsgruppen. In: Situationseröffnungen: zur multimodalen Herstellung fokussierter Interaktion. Institut für Deutsche Sprache, Bibliothek

32. Heritage J, Clayman S (2010) Dimensions of Institutional Talk. In: Talk in Action. Wiley-Blackwell, Chichester, West Sussex; Malden, Mass., pp 34–50

33. Holman D, Vertegaal R (2008) Organic User Interfaces: Designing Computers in Any Way, Shape, or Form. Commun ACM 51:48–55.

34. Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. In: Proc. Conf. Human Factors in Computing Systems. ACM, pp 591–599

35. Ishii H (2008) The Tangible User Interface and Its Evolution. Commun ACM 51:32–36.

36. Jacobsen M, Kristiansen S (2015) Goffman’s Sociology of Everyday Life Interaction. In: The Social Thought of Erving Goffman. SAGE Publ. Inc., Thousand Oaks, pp 67–84

37. Jefferson G (2004) Glossary of transcript symbols with an introduction. Pragmat New Ser 125:13–34

38. Jungermann H (1999) Advice giving and taking. In: Proc. Hawaii Intl. Conf. System Sciences. pp 11.

39. Jungermann H, Fischer K (2005) Using expertise and experience for giving and taking advice. Routines Decis Mak 157–173

40. Kemper TD (2011) Ritual: Goffman’s Big Idea. In: Status, power and ritual interaction: a relational reading of Durkheim, Goffman, and Collins. Ashgate Pub, Farnham, Surrey, England; Burlington, VT

41. Kendon A, Ferber A (1973) A description of some human greetings. In: Michael R, Crook J (eds) Comparative ecology and behaviour of primates. Academic Press

42. Kilic M, Dolata M, Schwabe G (2016) How IT-Artifacts Disturb Advice Giving – Insights from Analyzing Implicit Communication. In: Proc. Hawaii Intl. Conf. System Sciences. pp 878–887

43. Kilic M, Dolata M, Schwabe G (2017) Why do you ask all those questions? Supporting client profiling in financial service encounters. In: Proc. Hawaii Intl. Conf. System Sciences. Wai-koloa Beach, HI, USA
44. Koskinen I, Zimmerman J, Binder T, Redstrom J, Wensveen S (2013) Design Research Through Practice: From the Lab, Field, and Showroom. IEEE Trans Prof Commun 56:262–263.

45. Landsbergen D (2004) Screen level bureaucracy: Databases as public records. Gov Inf Q 21:24–50.

46. Luff P, Heath C, Greatbatch D (1992) Tasks-in-interaction: Paper and Screen Based Documentation in Collaborative Activity. In: Proc. Conf. Computer Supported Cooperative Work. ACM, New York, NY, USA, pp 163–170

47. Luff P, Pitsch K, Heath C, Herdman P, Wood J (2009) Swiping paper: the second hand, mundane artifacts, gesture and collaboration. Pers Ubiquitous Comput 14:287–299.

48. Mikkola P, Lehtinen E (2014) Initiating activity shifts through use of appraisal forms as material objects during performance appraisal interviews. Interact Objects Lang Mater Soc Act 57–78

49. Mondada L (2009) Emergent focused interactions in public places: A systematic analysis of the multimodal achievement of a common interactional space. J Pragmat 41:1977–1997.

50. Mondada L (2012) Video analysis and the temporality of inscriptions within social interaction: the case of architects at work. Qual Res 12:304–333.

51. Mondada L (2018) Greetings as a device to find out and establish the language of service encounters in multilingual settings. J Pragmat 126:10–28.

52. Mondada L, Schmitt R (2010) Zur Multimodalität von Situationseröffnungen. In: Situationseröffnungen: zur multimodalen Herstellung fokussierter Interaktion. Institut für Deutsche Sprache, Bibliothek

53. Mondada L, Svinhufvud K (2016) Writing-in-interaction. Lang Dialogue 6:1–53.

54. Mortensen K, Hazel S (2014) Moving into interaction—Social practices for initiating encounters at a help desk. J Pragmat 62:46–67.

55. Nass C, Fogg BJ, Moon Y (1996) Can computers be teammates? Int J Hum-Comput Stud 45:669–678.

56. Oloff F (2010) Ankommen und Hinzukommen. Zur Struktur der Ankunft von Gästen. Schmitt Hg Koord Anal Zur Multimodalen Interakt Tüb Narr 171–228

57. Patel MR, Vichich J, Lang I, Lin J, Zheng K (2016) Developing an evidence base of best practices for integrating computerized systems into the exam room: a systematic review. J Am Med Inform Assoc ocw121.

58. Pearce C (2017) Chapter 1 - Computers, Patients, and Doctors—Theoretical and Practical Perspectives. In: Shachak A, Borycki EM, Reis SP (eds) Health Professionals’ Education in the Age of Clinical Information Systems, Mobile Computing and Social Networks. Academic Press, pp 5–22

59. Pearce C, Arnold M, Phillips C, Trumble S, Dwan K (2011) The patient and the computer in the primary care consultation. J Am Med Inform Assoc JAMIA 18:138–142.

60. Pearce C, Trumble S, Arnold M, Dwan K, Phillips C (2008) Computers in the new consultation: within the first minute. Fam Pract 25:202–208.

61. Pillet-Shore D (2010) Making Way and Making Sense: Including Newcomers in Interaction. Soc Psychol Q 73:152–175

62. Pillet-Shore D (2011) Doing Introductions: The Work Involved in Meeting Someone New. Commun Monogr 78:73–95.

63. Pillet-Shore D (2018) How to Begin. Res Lang Soc Interact 31

64. Pillet-Shore DM (2008) Coming together: Creating and maintaining social relationships through the openings of face-to-face interactions. Ph.D., Univ. of California, Los Angeles

65. Pitsch K, Kuzuoka H, Suzuki Y, Sussenbach L, Luff P, Heath C (2009) The first five seconds;: Contingent stepwise entry into an interaction as a means to secure sustained
engagement in HRI. In: Proc. IEEE Intl. Symposium Robot and Human Interactive Communication. IEEE, Toyama, Japan, pp 985–991

66. Reeves B, Nass C (2003) The media equation: how people treat computers, television, and new media like real people and places, 1. paperback ed., [reprint.]. CSLI Publ, Stanford, Calif

67. Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human-computer interaction. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA

68. Schegloff EA (1968) Sequencing in Conversational Openings. Am Anthropol 70:1075–1095.

69. Schneider B, Sharma K, Cuendet S, Zufferey G, Dillenbourg P, Pea RD (2015) 3D tangibles facilitate joint visual attention in dyads. In: Proc. Intl. Conf. Computer Supported Collaborative Learning. pp 156–165

70. Smith GWH (2006) Erving Goffman. Routledge, Abingdon, Oxon ; New York, NY

71. Steimle J (2009) Designing pen-and-paper user interfaces for interaction with documents. In: Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. ACM, pp 197–204

72. Steimle J (2012) Pen-and-Paper User Interfaces. Springer Berlin Heidelberg, Berlin, Heidelberg

73. Svinhufvud K (2016) Nodding and note-taking. Lang Dialogue 6:81–109.

74. Svinhufvud K, Vehviläinen S (2013) Papers, documents, and the opening of an academic supervision encounter. Text Talk 33:139–166

75. Ten Have P (1991) Talk and institution: a reconsideration of the ‘asymmetry’ of doctorpatient interaction.’ Talk Soc Struct Stud Ethnomethodology Conversat Anal 138–163

76. Toerien M (2013) Using electronic patient records in practice: A focused review of the evidence of risks to the clinical interaction. Seizure 22:601–603.

77. Zimmerman J, Forlizzi J (2014) Research Through Design in HCI. In: Olson JS, Kellogg WA (eds) Ways of Knowing in HCI. Springer New York, New York, NY, pp 167–189

78. Zimmerman J, Forlizzi J, Evenson S (2007) Research through design as a method for interaction design research in HCI. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pp 493–502

79. Zuckerman O, Gal-Oz A (2013) To TUI or not to TUI: Evaluating performance and preference in tangible vs. graphical user interfaces. Int J Hum-Comput Stud 71:803–820.