Next Steps in Human-Computer Integration
Florian ‘Floyd’ Mueller 1 ∗ , Pedro Lopes 2 ∗ , Paul Strohmeier 3 , Wendy Ju 4 , Caitlyn Seim 5 ,Martin Weigel 6 , Suranga Nanayakkara 7 , Marianna Obrist 8 , Zhuying Li 1 , Joseph Delfa 1 ,Jun Nishida 2 , Elizabeth M. Gerber 9 , Dag Svanaes 10 , Jonathan Grudin 11 , Stefan Greuter 12 ,Kai Kunze 13 , Thomas Erickson 14 , Steven Greenspan 15 , Masahiko Inami 16 , Joe Marshall 17 ,Harald Reiterer 18 , Katrin Wolf 19 , Jochen Meyer 20 , Thecla Schiphorst 21 , Dakuo Wang 22 ,Pattie Maes 23

Figure 1. Exemplars of Human-Computer Integration: extending the body with additional robotic arms; [70] embedding computation into the bodyusing electric muscle stimulation to manipulate handwriting [48]; and, a tail extension controlled by body movements [86].
1

Exertion Games Lab, Monash University, Melbourne, Australia.
University of Chicago, Chicago, United States.
University of Copenhagen, Copenhagen, Denmark and Saarland University, Saarbrücken, Germany.
4
Cornell Tech, New York, Unites States.
5
Stanford University, Stanford, United States
6
Honda Research Institute Europe, Offenbach, Germany
7
Augmented Human Lab, University of Auckland, Auckland, New Zealand.
8
SCHI Lab, University of Sussex, Brighton, UK.
9
Northwestern University, Evanston, Illinois, United States.
10
Department of Computer Science, NTNU, Trondheim, Norway and IT University of
Copenhagen, Denmark.
11
Microsoft, Redmond, Washington, United States.
12
Deakin University, Melbourne, Victoria, Australia.
13
KMD, Keio University, Tokyo, Japan.
14
Independent researcher, Minneapolis, Minnesota, United States.
15
Strategic Research, CA Technologies, Pittsburgh, Pennsylvania, United States.
16
University of Tokyo, Tokyo, Japan.
17
Mixed Reality Lab, University of Nottingham, Nottingham, UK.
18
University of Konstanz, Konstanz, Germany.
19
Beuth University of Applied Sciences Berlin, Berlin, Germany.
20
OFFIS-Institute for Information Technology, Oldenburg, Germany.
21
School of Interactive Arts, Simon Fraser University, Vancouver, Canada.
22
IBM Research, Cambridge, United States.
23
MIT Media Lab, Cambridge, Massachusetts, United States.
∗
Authors contributed equally.
2
3

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.

CHI’20, April 25–30, 2020, Honolulu, HI, USA© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ISBN 978-1-4503-6708-0/20/04. . . 15.00
DOI: 10.1145/3313831.3376242

ABSTRACT

Human-Computer Integration (HInt) is an emerging paradigm
in which computational and human systems are closely interwoven. Integrating computers with the human body is notnew. However, we believe that with rapid technological advancements, increasing real-world deployments, and growing
ethical and societal implications, it is critical to identify anagenda for future research. We present a set of challenges
for HInt research, formulated over the course of a five-day
workshop consisting of 29 experts who have designed, deployed, and studied HInt systems. This agenda aims to guideresearchers in a structured way towards a more coordinatedand conscientious future of human-computer integration.
Author Keywords

Integration; augmentation; cyborg; implants; bodily
extension; fusion; symbiosis
CCS Concepts•Human-centered computing → Interaction paradigms;
INTRODUCTION

In designing the future of computing, it is no longer sufficientto think only in terms of the interaction between users anddevices. We must also tackle the challenges and opportunities
of integration between users and devices. This perspective isessential to fully understand and co-shape technology whereuser and technology together form a closely coupled system
within a wider physical, digital, and social context.

Looking at past eras of computing, these typically have aunique ratio of users to computers as one of their identifying
feature: The mainframe era of one-machine-to-many-users
shifted to the one-machine-to-one-user era of the personal
computer, followed by the one-user-to-many-machines eraof mobiles, to finally the many-machines-to-many-users eraof today’s ubiquitous computing era [96] (Table 1). Looking
towards the future, however, it appears that the next era will not
be described by such a ratio, but rather that its distinguishingfeature will be the blurring of the boundary between humanand computer [46].
Era / Paradigm

Users : Machines

Mainframe

many : 1
1:1

PC
Mobile1 : many

Ubiquitous

many : many

Integration

blurred boundary

Table 1. Eras of Human-Computer Interaction.

We call this Human-Computer Integration (HInt) and consider it a new paradigm with the key property that computers
become closely integrated with the user. Such integration
occurs primarily at an individual level through sensory fusion, with computers providing information directly to humansenses rather than through symbolic representations and understanding the user’s implicit, precognitive needs throughbio-sensing. However, we also note that this integration happens at a societal level, where human and interface agents
display coordinated effort towards achieving a common goal.
As such, HInt research shifts the focus of HCI away fromthe question “How do we interact with computers?” towards“How are humans and computers integrated?”. In this paper,
we present key challenges identified in a 5-day workshop withthe aim of moving the field forward. We believe this provides
HCI researchers with a set of challenges to guide their future
work in a coordinated and conscientious manner. We believethat such a structured approach is preferable over, for example,
following industry trends or financial short-term gains.
Although prior work has discussed challenges in related [19]and overlapping areas (e.g., cybernetics [6, 43], intellect augmentation [18, 65, 50], cyborgs [13], wearables [80, 78, 45]),
we have not yet seen any articulation of the key challenges
facing HInt. Unlike most of the prior theoretical work above,
our work does not stem from an individual mind, but rather
from a collective group of people who have developed an understanding of HInt through their own practice, coming froma wide range of backgrounds. Therefore, our work presents
a collective set of future steps coming from the “trenches”,
which aims to extend prior road maps by articulating roadblocks we found along the way.
Our key contribution is the synthesis of the challenges that thehuman-computer integration field faces across four thematic
areas: (1) Human-Compatible Technology; (2) Effects of Integration on Identity and Behavior; (3) Human Integration andSociety; and (4) Designing Integrated Interaction. This will
help researchers and practitioners interested in HInt to: (a)identify current knowledge, capabilities and areas of opportunity where they can contribute; (b) situate their work withina larger HInt research agenda; (c) and also allow policy makers to better understand the HInt community, state-of-the-art
technology and research, as well as potential applications.
PROCESS

To formulate the challenges for Human-Computer Integration,
we organized a 5-day workshop with 29 participants.
Participants

Our participants included 19 senior academics, 5 senior industry participants and 6 junior academics. Participants represented broad areas of expertise across computer science,design, art, psychology in areas including human computerinteraction, mobile, wearable, printed electronics, haptics, multisensory experiences, gaming, user experience design, cognitive psychology, social psychology, and multimedia art. Nine
participants identified as women and 20 as men. Participants
were distributed throughout the globe (Europe 9; North America 12; Asia, 2; Australia, 6) and across a range of ages (21-30,
4; 31-40, 8; 41-50, 9; 51-60, 4; 61-70, 4).
Participants were invited because they have designed, taught,
deployed or studied HInt systems. Examples of such systems
are depicted in Figure 1. For instance: MetaArms, a systemthat augments the user’s body with two additional robotic arms
controlled by the user’s feet to allow handling complex tasks
that two hands might have difficulty with [70]; Muscle Plotter,a system that uses electric muscle stimulation (EMS) to turnthe user’s wrist into a computerized pen plotter, allowing todraw, for example, simulated wind effects on the shape of acar’s sketch [48]; or a motion-controlled tail extension to thewearer’s body, which can be used for enriched self-expression
either in daily life or as part of artistic performances [86].
Discussion process

Prior to the workshop, participants shared with the group 12 seminal readings related to Human-Computer Integration.
The 5-day workshop began with a Pecha Kucha where each researcher presented their research related to the topic of HumanComputer Integration and the seminal reading(s) they selected.
Subsequently, we broke out into subgroups (4-5 participantseach) to explore: Key Challenges, Definition of HInt, Motivations for HInt and Dark Patterns and Ethics; following
each of the break-out sessions, the groups would re-convene
to share insights and findings for discussion with the larger
group. Participants would rotate between subgroups to share
their expertise. In addition, there was a generative design
session which also took place in sub-groups, with a subsequent guided reflection on the theoretical topics that emerged.
These insights led to the creation of a collaborative documentin which authors refined the key challenges and insights thatwould ultimately be shared with the HCI community.
RELATED WORKThe idea of integration between a system and their user canbe traced back in the history of computing, art, philosophy,
neuroscience, and even science fiction. In this paper, we fo-

Examples of devices integrating with the user’s body can betraced almost to the start of the field of interactive computing, which was empowered by derivative ideas from Norbert
Wiener’s cybernetics movement, such as closed-loop machine
systems [36]. One canonical example is Licklider’s “(Hu)manComputer Symbiosis”, which, building on the cybernetics
ideas, postulated that “the cooperation between users and machines was an expected development”, and, moreover, that thiswould require a “very close coupling between the human andthe electronic member of the partnership”, alluding to notionsof body-integration [43]. Another seminal example is Engelbart’s vision of HCI as “augmentation of human intellect” [18].
This was well depicted in Engelbart’s GUI system, designed in
1960s, attempting to not only simplify input (using the mouseand a chorded keyboard) but, also, to amplify a user’s cognitive abilities. These ideas were echoed in other prototypes of
the time. For instance, Sutherland’s “ultimate display” wasnot only a critical advancement in display techniques but anattempt to fuse the human spatial senses (not only vision, butalso proprioception) with that of the device [85]. From hereon, the list of examples runs long and we will refer directly tothese as they assist our argumentation.
TYPES OF HUMAN-COMPUTER INTEGRATION

While the concept of integration between humans and computers echoes the initial efforts of pioneers, such as Licklider [43],Engelbart [18], and even Clark [13], it is Farooq and Grudin’smore recent articulation of human-computer integration [19]that we use as point of departure. We expand upon Farooq and
Grudins work to include other aspects of integration, which
we consider to be particularly relevant.
Integration between humans and technology can occur and has
already occurred in many ways. When we speak of HInt systems, we refer to a subset of these. Figure 2 shows an overview
of ways in which technology and humans can integrate. The
x-axis depicts agency; ranging from devices in which humans
are in full control (left), to shared control (middle), and tosystems in which all control remains at the device (right). The

6

7

1

2

Individual

Scale

Symbiosis

43

FusionOrgan

Tracing back the precise origins of the concept of “integration”
is outside the scope of our work as we intend to focus iton the direct challenges this concept poses for the field of
HCI. Yet, we briefly illustrate how this concept originated invarious shapes and in a wide variety of knowledge fields. The
concept itself can be seen in science fiction, in concepts, suchas “man-machine mixture” in Edgar Allan Poe’s writing in
1843 or the humanoid-“robot” in Karel Capek’s 1920s play; inneuroscience where Manfred Clynes and Nathan Kline coined
the term cyborg in the 1960s; philosophy, as echoed in D.
S. Halacy’s 1965 essay on the Cyborg; art, for example in
Stelarc’s 1990s work, and, of course, in early works in humancomputer interaction, which we detail in the following.

Societalcus predominantly on the perspective taken by researchers
in HCI. Therefore, we review prior work that inspired andgrounded our research mostly from this lens. Note that we
kept this section intentionally short as throughout our following argumentation we bring up the relevant references in our
discussions.

5

8
Human

Mixed

Technology

AgencySymbiosis
1 Integration as discussed by Farooq and Grudin [19]2 Humanistic Intelligence [50]FusionOther

3 MetaArms [69]

6 Telecommunication Systems

4 Muscle-Plotter [48]7 AI Dictator

5 Ping Body [82]8 Pacemaker

Figure 2. Map of integration between humans and devices.

y-axis represents the scale at which the integration occurs:
ranging from whole cultures integrated with technology (top),
to organs and organelles at the micro level of integration (bottom). Together, these two dimensions map out a subset ofways in which humans and technology can relate. Our primaryinterest lies in two types of integration that we will describe
next, these are: symbiosis and fusion (indicated in horizontally
stripped purple and vertically striped blue, respectively).
Symbiosis

We call systems in which humans and digital technology work
together, either towards a shared goal or towards complementary goals, symbiotic (Figure 2, purple). In this type of integration, agency is shared between humans and digital systems,
and integration can occur on the individual level or between
groups of people and technological systems. Examples of
symbiosis, depicted in Figure 2, include: 1 the scenarios
presented by Farooq and Grudin [19], which describe digitalsystems, which continuously work on the humans behalf, evenwhen the human is not attending them. Their examples include
integration beyond the individual level, as activities such as
autonomous driving or intelligent rescheduling of meetings,
require technologies to mediate between multiple people and
2 at an individual level, Mann’s vision of Humanistic Intelligence [50] where there is a continuous feedback loop between
a human and a digital system, each augmenting the other.
The key characteristic of symbiosis is not that computers enable software agents or that the agents are smart. The key
is that the agency is truly shared between technology andhumans acting in concert, for example by collaborating increative tasks [23, 10] or working together towards engag-

ing experiences [5]. This excludes various ways in which
technology is currently integrated in our society, for example
6 telecommunication technologies have become an integral
part of our culture, yet they do not have any agency of theirown. Similarly, 7 a future where the government is replaced
by AI agents which dictate human laws, would also not be
considered symbiosis as the agency of the human is lost.
FusionWe define fusion as an integration in which devices extend theexperienced human body or in which the human body extends
devices (Figure 2, blue). Fusion occurs on an individual leveland often only affects a sub-part of the user, such as a limb
or a sense. Unlike symbiosis, which requires shared agency,
fusion can occur throughout the spectrum of agency. A key
characteristic of fused systems is information that is not represented symbolically. Instead, humans perceive through fusion
systems by embodied mediation [92]. Similarly, fused systems
do not require explicit input from humans, but simply act as
extensions of human bodies.
Examples of fusion systems, depicted in Figure 2, include:
3 MetaArms [69] (depicted also in Figure 1), a system inwhich the agency is almost entirely with the user, but thetechnology feels like a natural extension of one’s body; 4Muscle-Plotter [48], a system that controls the user’s hand
via electrical muscle stimulation to empower the user withcomputer based simulations (also depicted in Figure 1); and,
5 Ping Body [82], an art piece by Stelarc in which the performer’s body is controlled by via muscle stimulation in a waythat only minimal agency remains with the human. Fusion
might also occur with implanted devices [28, 84], ingested
devices [41, 81], or epidermal electronics [80], as well as devices which extend or manipulate the body (e.g. [86, 74]), or
stimulate the senses [47, 73, 83, 98].
Not all systems where technology and the human body physically connect fall within our definition of fusion. For example,
devices which augment individual organs below the perceptual
threshold, such as 8 pacemakers are also outside of the scopeof HInt as we present it. While literal integration occurs due tothe implantation, these devices do neither provide an interface
to the user nor provide the user with any agency.
HInt as Analytical Lens

When we speak of Human-Computer Integration, we explicitly
refer to both fusion and symbiosis 1 . Fusion and symbiosis
should not be understood as supported by a specific technology,rather they describe ways in which humans and technology
relate. HInt then becomes an analytical lens for analysing
and designing such relations, whereas Human-Computer Interaction (HCI) is a broader lens for analysing and designinginterfaces of various types.
The HInt and the HCI lenses can be used for analysing thesame scenario, e.g., a user with an exoskeleton [69], but eachsheds light on a different aspect. The key difference is that
1 In our subsequent analysis, we mostly highlight fusion aspects oversymbiosis, as we found these to be less prevalent in prior discussionsof integration [19, 21, 20].

the HInt lens considers as its starting point that we are observing one human-technology assemblage; instead of considering that we observe an interaction between the user and theexoskeleton–this would be the result of looking through themore general HCI lens. Therefore, the HInt lens encourages
analysis of this human-technology assemblage, for example,
by asking how the agency is distributed or by describing thetype of integration by measuring the amount of physical or
cognitive coupling between user and interface.
CHALLENGES

Shneiderman et al. [75] suggested that HCI as a field needs“grand challenges” to steer the direction of future research,
design, and commercial development. As such, challenges
have been articulated across respective fields, see, for example,
the work on next steps and challenges within shape-changing
interfaces [3], information retrieval work [7], social robotics
research [89], and crowdwork investigations [37]. Just as they
have advanced their respective fields, we hope that our work
will move the HInt field forward. In this paper, we describe
four sets of challenges that we expect to be at the core of
future HInt research: (1) Human-Compatible Technology; (2)Effects of Integration on Identity and Behavior; (3) Integrationand Society; and (4) Designing Integrated Interaction”. We
describe these next.
CHALLENGE #1: HUMAN-COMPATIBLE TECHNOLOGY

We believe that, especially due to the fusion between computerand human body, HInt systems will benefit from a deeper understanding of the user’s physiological and mental state. This
understanding requires collecting and interpreting data fromthe human body. For example, the combination of biochemical and electrophysiological signals allows devices to reason
about the user’s health and fitness [31] (see Figure 3 for more
examples). However, current systems, such as wearables, areoften designed around a rigid form factor that restricts theirplacement in the user’s body. Hence, they can only access alimited amount of information and are not well integrated into
the body.
In recent years, advances, such as epidermal electronics andinteractive textiles have emerged that make use of flexible andstretchable electronics, which enable stronger fusions withthe human body. Beyond the material aspects, the integrationof electronics with the body raises the need for customisingfor different body sizes and shapes as well as personalisingthat will allow people to express themselves and their aesthetic preferences. Moreover, the integration with the bodyrequires rethinking how devices are deployed, maintained, andconnected to their surroundings. We therefore structure thissection of human-compatible technology into key types ofhuman-compatible technology, materials for integration, tailored technologies for the body, and connecting the body andthe world.
Key types of human-compatible technology

We identified five key types of human-compatible technology,
noting that future devices could act across multiple layers at
once (see Figure 3 and Table 2).

a

c

e

g

i

b

d

f

h

ij

Epidermal Technologies

SubdermalTechnologies

Transdermal Technologies

Implanted Technologies

Pass-through Technologies

Figure 3. Examples of human-compatible technologies. (a) Tacttoo [97], (b) iSkin [94], (c) Hobbyist use of insertable devices [27], (d) RFID implants [24],(e) Wear It Loud [64], (f) The tongue and ear interface [68], (g) Cerebral shunts [76], (h) pacemaker [87], (i) Stomach Sculpture [81], (j) ChewIt [22].
Type

Body Contact

Application

Permanence

Maintenance

Lifetime

Epidermal

Epidermis

User-controlled(sticker, spray-on)

Removable by theuser

Removable andreplaceable

User-controlled, allows
for short term usage

Transdermal

Epidermis and dermis

Piercing or surgery

User-controlled
or surgery

Through external port

Medium to long-term
usage

Subdermal

DermisSyringe or small
surgery

Permanence
through surgery

Through surgery or
wireless update

Medium to long-term
usage

Deep Implanted

Internal organs

Surgery

Permanence
through surgery

Through surgery or
wireless updateLong-term usage

Pass–ThroughDigestive system

User-controlledNo

Not intended: wait till
it passes throughUsually 24-26 hours

Users : Machines

Table 2. Key types of human-compatible technology and their properties.

Epidermal technologies are worn on the skin. In contrast to
wearable devices, their thin and stretchable form factor enables
them to better integrate into human skin. Examples of suchtechnologies in interactive devices include: iSkin [94], Skintillates [44] DuoSkin [33], and SkinMarks [95]. These devices
offer users a variety of applications, such as on-body input [94,44, 33, 95, 58], on-body NFC [33], visual displays [95, 33]
and haptic output [97, 98]. Their main advantage is their easy
application and removability.
Subdermal technologies integrate devices into a deeper layerof the skin: the dermis. In contrast to epidermal technologies,
these technologies can access more body information, e.g.,
by analyzing the interstitial fluids. For example, Holz et al.
investigated the feasibility of interacting with subdermallyimplanted touch sensors, LEDs, vibration motors, and microphones [28], and Heffernan et al. surveyed hobbyists’ use ofmany subdermal devices [27].
Transdermal technologies contain an epidermal and subdermal part, similar to a piercing. They combine several advan-

tages of the previous technologies, i.e., they enable a deeper
integration with the body while supporting easy access fromoutside the body. Body parts that allow for piercing (e.g.,
the ear and the nose) can host electronics that are still easily
removable. Transdermal objects, common in the body modification scene, open a path from the outside to the inside of the
body but require constant care to avoid infections.
Deep implanted technologies are permanently inside the human body (e.g., a pacemaker or an insulin pump). These
devices can have the deepest integration with the human body,but they are hard to replace, and application requires surgery.
Homewood and Heyer explored how users might want to interact with such devices [29].
Pass-through technologies are technologies that enter the
body only for a specific duration. The advantage is that thesedevices automatically exit the body or dissolve after a certain
time. Examples include digested pills that are able to track
body temperature [41, 42], chewing-gum-like interfaces that

provide hands-free interactions [22], and some early artistic
performances by Stelarc [81].
Having discussed key technologies for human-compatible technology, we now turn to the materials required to make thisintegration with the human body successful.
Materials for Integration

We find that a close integration with the human body benefits
from devices that feel and behave like parts of the body. For
such devices it is beneficial to be biocompatible, miniaturized
and, deformable, three aspects we discuss next.
Biocompatibility: Due to their close proximity to the human
body, integrated devices require a higher level of biocompatibility than traditional interactive devices, such as mobile
phones, smart watches, or head-mounted displays. For example, an integrated implant should not expose possible allergens,such as nickel. We note that the biocompatibility is dependent
upon how a device is integrated. For example, an on-skindevice only needs to be skin-compatible [94], whereas devices
that are implanted [28] or ingested [41] must meet higher
requirements to avoid immune system responses. When nonbiocompatible materials cannot be avoided, sealing the device
is an option; these seals, however, must be robust enough under high mechanical and chemical stress to avoid leakages orcompromising the user’s body.
Miniaturization: Despite the impressive miniaturization ofelectronics in recent years, most wearable devices are still toolarge to be integrated into our bodies. Recent work with onskin electronics shows that thin touch sensors (4–46um) [95]permit the use of small body landmarks. These enable the useof the body’s geometry for input [95] or to help in recalling
virtual elements [8]. Similarly, devices that are implanted
require also a very small form factor to be worn comfortably
under the skin. In addition to input and output surfaces, suchintegration requires much smaller processing units, batteries,
energy harvesters, and antennas.
Deformability: We believe that integrated devices should be
deformable, for example, flexible, compliant, and stretchable.
First, compliance allow for robust devices that are better inabsorbing shocks and damage. Second, flexibility and compliance are important to ensure a good fit to the curved andflexible human body.
Tailored Technologies for the Body

We find that integrated devices should support the wide rangeof shapes and sizes of the human body [17, 99]. For example,
body sensors need to be automatically calibrated for eachindividual to allow for continuous usage as examplified in
the work by Knibbe et al. [38]. Hence, a “made to measure”
approach for technologies is required. Beyond size and shapedifferences, body decorations have a long tradition in many
cultures [16]. Therefore, visual customization and aesthetic
electronics could increase the acceptability of such devices.
Early examples are interactive beauty products [90] and visual
aesthetic on-skin devices [94, 33, 44].

We note that personalized devices are a stark contrast to today’s
mass-fabrication of technologies, which excels in producing
identical devices in high quantities at a low cost, but whichcannot be easily customized. A potential solution is end-user
customization (e.g., cuttable electronics [60]), which enables
the mass-fabrication of a single form-factor that allows for
subsequent adaptations by the end-user. An alternative could
be the use of single-unit fabrication methods (e.g., printed
electronics [79] and 3D printing [49]). These technologies
create highly personalized devices, but are currently slower
and more expensive than traditional mass-fabrication methods.
Beyond the fabrication step, it is important to consider thewhole design process, which requires easy ways to gather thegeometric information from the body and novel CAD software(as shown, for example, for multi-touch surfaces [58]) for theend-user that translates geometric shapes and visual designs
into functional devices.
Connecting the Integrated Body with Additional Devices

Integrated technologies will often require communication between the interactive device and devices on the internet or
around the user’s body (e.g., to store and backup data, etc.).
Different methods have already been proposed to form suchbody area networks [12], but miniaturization, data transfer,and energy consumption are still an open challenge. With
the event of connectivity, integrated devices also require highstandards of security to prevent malicious digital attacks on
the user’s body.
In particular, we find that energy management is an interestingdomain for integrated devices. They can be either charged
through an epidermal port, wirelessly [28, 84], or harvest energy from the body [71]. Energy harvesting could be the mostuseful form for integrated devices since they do not require
manual charging, but rather take the required energy from thehuman body. However, most current methods generate too
small amounts of power for today’s electronic systems.
CHALLENGE #2: EFFECTS ON IDENTITY AND
BEHAVIOR

We now describe the challenge concerning identity and behavior around HInt systems. We previously described thetechnologies that one might use to create a new integrated
self (e.g., user that is integrated with a particular interface);
however, a new integrated self comes with a possible shift
in the perception of self. Furthermore, this integrated self
will most likely be also perceived differently by the surrounding people, whether they are other integrated selves or selves
without these types of interface augmentation. This creates aperceptual feedback loop between the integrated self and theinteractions with others as human beings are influenced by
how they are perceived by others. Addressing these challenges
will produce empirical and theoretical insights about who weare and who we want to become [56] within a future where
integrated and non-integrated selves interact with each other.
Perception of the Integrated Self

The relational self is the part of an individual’s self-concept,
which consists of the feelings and beliefs that one has regard-ing oneself and develops based on interactions with others [4].
Self-perception is thus said to be created through informationfrom different sources and modalities. While the majority ofinterfaces mostly addressed the visual and auditory senses, as
we described, HInt systems tend to operate at a physical levelthat involves other bodily senses, such as proprioception, etc.
Therefore, we believe that the usage of a larger multisensory
stimulation in HInt systems might have a significant impact
on the relational self [59, 77, 91].

might be more critically perceived than others, especially whena technology fails to respect personal or privacy rights (e.g.,cameras or microphones that eavesdrop on conversations). For
example, we may wish to be informed if a device in our surroundings is switched on or off, or what data is being recorded
for what purpose and by whom. Furthermore, the HInt user
also faces challenges as bystanders may develop mistrust andact differently or distanced. Hence, it is beneficial that the
HInt technology provides transparency in its interface [40].

Furthermore, one’s self-image can be modified through technology by either changing the perception of ourselves or byphysically changing ourselves. As an example, Riva et al. [66]and Nishida et al. [57] demonstrated that one’s body schema
can be changed by simply seeing a different body to their own
through a head mounted display. Furthermore, technology
has the potential to enhance our sensory system, for example, by extending the abilities of the visual sensation throughwearing thermal imaging glasses [1] or by a brain-computer interface that allows accessing other people’s indicated cognitive
load [26].

Moreover, we believe that designers of HInt systems shouldtake into account its context of application and the ownershipof its benefits. For example, bystanders are often more open
to accept integrated selves if the technology enables the user
to have skills that others commonly have, e.g., the acceptanceof cameras that empower visually impaired people is higher
compared to the acceptance of cameras of users who have no
visual impairment [39].

Lastly, the perception of self implies a social loop in which
we react to others’ reactions on us. Thus, we now describe
a possible lens of analysis from the opposite perspective, theperception of other integrated selves.
Perception of other Integrated Selves

Social groups are built around individuals to which they belong
to various extents. This is often based on a relation between
individual attributes and a social expectation set of accepted ordenied status symbols and a set of behavior rules that the groupcommonly agrees on [88]. If an individual has more attributeswith a high social acceptance, it is more likely that their role
within the group is a leadership position. Interestingly, an
attribute, such as a perceived strength of an integrated self thatis positively perceived in one individual can cause a negative
perception in another as the relationship to individuals biases
our emotions toward individuals positively or negatively [15].
What does this mean for HInt? Let us envision interacting withan individual using HInt technology that might be able to, forexample, see through their augmented vision when we become
nervous or they might – while conversing with us – be able
to access past conversations (e.g., Mann’s AR interface [50]).
Such ability extends their previous non-integrated self, andother selves will, according to Cuddy [15], appreciate theincrease of the integrated self’s abilities if they are a friend andbelong to their social group. We might see this HInt-individual
as a positive addition to their group as their increased abilities
ultimately improve the group. However, if the individual isseen as a competitor or might not not belong to our social
circle, we may not want to let that person know when we, forexample, are insecure or nervous as that information could
bring our competitor into a better position. Consequently, theability of the integrated self will be perceived as an increaseof competition and may even be disliked, arouse envy or evencreate mistrust.
Besides the effect of emotions towards HInt users caused by
the relationship between user and others, some technologies

Evaluating Potential Issues of the Self

When we integrate technology with ourselves, how do we evaluate the effects in regards to the self? Questionnaires could
help in evaluating effects of the perception of oneself or ofothers. For example, Schwind et al. [72] investigated the acceptance of VR technology in different social setups throughmodifying the questionnaire from Profita et al. [63]. Moreover,qualitative approaches could also be beneficial. For example,
psycho-phenomenology can provide a relevant lens into human subjective experiences, which could be useful here. One
specific method is the explicitation interview technique [52],which is a form of guided introspection that seeks first-person
accounts by using distinctions in language, internal sensory
representations, and imagery. The value of this interview
technique lies in the way of asking questions that supports
participants in expressing their experiences linked to a specificmoment. For example, the interviewer asks questions like
“Please describe what you feel, see, hear, or perceive” andfollows up with questions that help to place the participant
in an evocation state so they talk about that specific lived experience (including action, sensory perception, thoughts, andemotions) in all its details rather then focusing on conceptual,
imaginary, and symbolic verbalizations, such as theories, rules,
or knowledge.
Similarly, Koelle et al. [40] demonstrated a participatory design approach to develop devices that take into account such
issues, which we believe would be useful to consider whenaiming to evaluate issues of the self. The design challenges
that the authors explicitly focused on were the user experienceof smart cams, which bystanders do not feel comfortable with.
Their approach starts with (1) development while highlighting social acceptance challenges; (2) involve experts frommultiple relevant disciplines to create a set of prototypes; (3)analyze the prototypes to aggregate design strategies; (4) andevaluate the design strategies with UX experts to (5) define
solutions that incorporate product requirements, such as social
acceptability, interface transparency, and interfaces that not
only please the user but also respect the bystander.

CHALLENGE #3: INTEGRATION AND SOCIETY

Current and future HInt devices will affect society in a variety of ways. Designers, researchers, industry, and regulatory
bodies will need to attend to these. While we believe theprocess of developing new products, services, and regulations
should be democratic, informed, inclusive, and involve public dialogues, in this paper, we do not take a strong stanceon a specific ethical principle, but rather present a list of key
societal challenges.
Digital Divide

Previous interactive technologies, such as interactive devices
with internet access have led to concerns about inequality inaccess, often referred to as the “digital divide”. We see thisbeing potentially amplified as devices are becoming integrated.
When technologies are augmenting senses or giving people
new capabilities, new divides will be created which may havenew and unexpected consequences. For example, if areas of
public space are designed for people with new sensory capabilities, does the sensory-divide created by this design exclude
people who cannot afford the augmentation? We can alreadysee how new technologies affect public space with, for example, the decline in publicly visible clocks as mobile phones
with time information became widespread [53]. If, for example, new visual capabilities remove the need for navigation andinformation systems, public maps and signs might be removed;
the result would be a world that is profoundly disabling forhumans without access to these systems.
Body Bias

ecosystem of devices, which may exhibit a level of agency,
we need to understand what the different motivations of thedifferent systems are (e.g., “Is the system providing a service
in return for advertising to the user?”) and develop ways
for attributing responsibility for the actions of the combined
system (“Who is at fault if my exoskeleton makes me harm
someone; me or the software developer who programmed theexoskeleton wrongly?”). We also need to consider ongoing
maintenance and support. Many modern hardware systems
are highly dependent upon the continued running of cloud services, which blurs the nature of the ownership of the devices,even if any physical device belongs to a user. As devices are
increasingly integrated and users become reliant on them, wemust consider what the effect on users is at the point whendevices become unsupported. In the case of in-body devices,
regulation may even be required to enable ongoing supportand maintenance by third parties in the event of the failure of
the company.
CHALLENGE #4: DESIGNING INTEGRATEDINTERACTION

This section describes the challenges HInt poses for interaction
design. HInt has two key qualities which are relevant to design:(1) the system can exhibit a form of autonomy that needs tobe coordinated with the user and (2) the system’s real-time
feedback fuses with the user’s sensations. HInt systems are
therefore uniquely challenging in their integration of autonomy
and real-time feedback. In response, we have identified three
key design challenges: applying novel technologies, designing
implicit interactions, and designing for variable agency.

We believe that direct fusions interfacing with the user’s bodymight also lead to an increase in "body bias". Integrated systems are inherently more dependent on the nature of the bodythey are fusing with, for example, some biosensors function
differently on skins of different ethnicities [93]. More basically, variability in body shape and size may increasingly
become a factor as we fuse technology with the body. Differences relating to gender and age may also affect how people
are able to interact with their integrated bodies. We note that
this challenge is also a software one: as we design softwaresystems to integrate with a person’s body, we need to consider the potential for such designs to embed assumptions ofcultural, gender, or physical differences.

One of the key challenges for HInt is to develop common
understandings and tools for designing, developing, refining,
testing, and evaluating HInt systems, especially as many HInt
systems contain novel technologies that afford new types ofinteraction. Thus, novel technologies are like new materials
which require careful characterisation and profiling. Many ofthese interface advances draw on disciplines – e.g. physiology,
chemistry, neurophysiology – that are relatively new to HCI.
It might therefore be desirable to develop toolkits for thesetechnologies to make them easier for designers to apply to new
applications.

Mental and Physical Health

Designing Implicit Interaction

As with all technologies that form a long-term part of people’s
lives, such as the car or the smartphone, there is also a potential for negative impact on mental and physical health, forexample see the impact of the aforementioned technologies on
increased poor posture, greater stress levels, and heightenedrisk levels due to the distractions they afford.

Unlike the application and integration issues that stem fromthe migration of HCI technologies to the bodily domain, theinteraction issues that stem from operating just beneath or justabove the user’s awareness as well as just ahead or just behindthe user’s intent, have little direct precedent in the space of
medical devices or even prosthetics [32, 73]. Therefore, we
believe that integrated systems would benefit from knowledgeof how tightly coupled performers operate, like dance partners.

Ownership and Accountability

Business models of companies such as social networking software providers aim to create a range of dependencies in usersthat ensure the continued use of the systems and ultimately
drive profit. We find that, as systems become more integrated,
this increasingly creates a range of challenges for users, developers, and regulators. When our body is part of a combined

Integrating Novel Technologies

Designing for Variable Agency

We find that a HInt system is often experienced in various ways
based on how its control is distributed to its users. For example, it might be designed in such a way that the technology acts
as an extension of the body while displaying minimal or noautonomous behavior at which one can interact in a reflexive
manner [51]. Many systems offload cognitive or motor effort
away from users and instead automate technology. As systems
shift from explicit control to autonomous (e.g., from driving
the car to having the car drive itself), the interaction gradually shifts from a singular entity exploring the world throughtechnology (e.g., driving the car) to an agent-like system that
explores the world by itself (e.g., a self-driving car). As such,from a phenomenological perspective, HInt systems might
therefore feel like an extra limb, either natural to use, such as
the extra hand holding your bag when you need to rummage
through it or with its own agency, like an EMS-controlledarm [45] that acts of its own intent.

must understand that perception is an activity that the body
performs. By acting in the world, the body changes the sensory information it is exposed to, which in turn triggers new
actions [2]. Injecting information into such interactions enables the presentation of artificial sensory experiences, forexample providing the experience of texture where there isnone [67] or sensations such as resistance and weight [83].
While examples of systems that support embodied mediationof information exist, they are typically limited to a particularsensory modality or to specific information. Creating systems
without such constraints requires a generalized approach ofachieving sensory transparency for all our senses and is thus
open challenge.

Therefore, the key challenge for the designer is to determine
what parts of the system should be provided with agency andto what extent. Furthermore, increasing a person’s agency
in this context also presents a low level challenge for thedesigner. For example, how does one design a technology that
truly provides the experience of “I did that” rather than “Thetool did that for me”? Intentional binding provides us witha tool from neuroscience for evaluating these phenomenas
quantitatively [14, 9]. However, we find that the design space
is not yet well understood as more and more interfaces find new
configurations for shared agency [34]. A design framework
might therefore be useful to guide designers regarding both,where and how to endow a system with agency.

Transparency through understanding other minds

Perceptual Transparency

When perceiving information, we have different strategies
available for interpreting it. For example, information might
be mediated symbolically and require an interpretive stepfrom the user. If, for instance, checking the weather in anapp, we are provided with a number that represents the outside temperature. We then cross-reference this with our lived
experiences and infer what it might feel like. Due to this interpretive step, such mediation of information is referred toas “hermeneutic” [30, 92]. Hermeneutic mediation is often
juxtaposed with embodied mediation [92]. The canonical example of embodied mediation is perceiving the world througha cane. A more technologically sophisticated example might
be a haptic teleoperation system, which directly provides thetarget sensation [62]. Rather than demanding the interpretive
step described before, embodied mediation allows reflecting
on one’s current state of being to understand the information.
It is this latter, a more direct way of understanding, which
we wish to be achieved with HInt systems. We see this directtransfer of sensations between the user and the device as a
type of "perceptual transparency" that can be achieved in two
possible ways:
Transparency through sensory accessIn the context of haptic teleoperation, a system is said to betransparent “if the human operator feels as if they are interacting directly with the environment” [61]. Virtual reality isable to visually transport users to remote or imaginary places
by providing users with the necessary visual cues. However,for other senses, this is less trivial, leading designers to often
use proxy symbols (e.g., green clouds for bad smell, vibration for object collisions). To avoid such proxy symbols, we

People already have an embodied understanding of others
through our shared experience of having a body. People assume their one movements to be equivalent to the movementsof others [11]. These experiences need not be in perfect sensorial agreement for a set of individuals to interact, e.g., a sheep
dog, sheep, and a shepherd can act in unison, even though
they perceive the situation radically different by means of
their senses [35]. We find that the tool set we have at our
disposal for understanding others is currently poorly suitedfor inferring motivations of others, whether they are humans
or interfaces for interactive devices. Conversely, such agentsbehind interactive devices often only have very sparse information of their user and the world around them, paired withsimplistic resources for interpreting these. However, evensimple information, such as body temperature or step count,change their meaning based on the user’s context. We findthat providing interfaces with access to other’s, for example,
mental states, goals, and motivations [54], ranging from raw
data to actionable information [55], is a non-trivial process.
As such, the challenge becomes how to design interfaces sothat we can intuitively asses their motivations and goals andhow to provide these with the tools they need to understand us
in a way that supports a partnership.
CONCLUSION

In conclusion, we find that the field of human-computer integration (Hint) is rapidly expanding, embracing new technologies, and incorporating new disciplines. At the same time, thefield is beginning to converge on key questions, surrounding
technology, self-perception, societal, and design implications,which we identified in this paper.
We discussed what we consider to be the key set of challenges
the field currently faces. These challenges emerged from aworkshop with 29 experts. The challenges will evolve astechnology advances, society changes, design knowledge improves, and our self-understanding increases. Many HInt systems already exist and offer engaging experiences. However,
we believe the challenges we identified need to be addressedin order to reap the full benefits of a HInt future.
Our hope is that designers venturing into human-computer
integration will use this work to become aware of the challenges they will face. Moreover, those deep in the field (whomight already know many of these next steps and challenges)

will also benefit as the work will provide them with an initial
vocabulary to describe their experiences. Similarly, we expectthat theorists will also benefit from our work as they can use
it to verify their theories regarding real-world practice or tohelp answer some of the immediate challenges. Developers
can also use our work to identify opportunities for innovation
by looking at specific implementation challenges. Finally,educators can also benefit from this work by looking at thechallenges to identify future research topics.
It is important that one also recognizes the limitations of theHInt field. In research on proxemics, researchers have identified “Dark Patterns” [25], which are application scenarios
where users are deliberately deceived through a particular interaction technology. We can envision that similar scenarios
might occur within the HInt field, and therefore we encourage
future work to conduct such investigations, while hoping that
our work can be useful to structure such investigations.
We acknowledge that our approach of conducting a workshop
with 29 experts has not only advantages, but also disadvantages, for example our experts are eager to drive this fieldforward, as such might be optimistic about broader social acceptability. However, we believe our workshop format is alsoa unique approach when it comes to compiling key challenges
for HInt that goes beyond one individual’s own work. As such,
we see the identified challenges not necessarily as problems
that need fixing, but rather aspects that need more examination
and research. Therefore, we believe that our challenges are
only a starting point that needs to be developed and critiquedfurther by others, including theorists and designers.
In summary, we are excited about the potential of the HInt
field and how it will affect how users engage with technology.
With the articulation of our set of challenges, we hope wewill motivate further research efforts towards this exciting
future. Ultimately, with our work, we want to support you incontributing to the future of human-computer integration.
ACKNOWLEDGEMENTS

The Exertion Games Lab acknowledges the support of the
School of Design at RMIT University. This work was supported by the European Research Council, grants No.: 648785
and 714797. The Human Computer Integration Lab at the University of Chicago thanks the support of the Grant-in-Aid for
JSPS Research Fellow (JP16J03777). The work of the SCHI
Lab has been supported by the European Research Council(ERC) under the European Union’s Horizon 2020 researchand innovation program under Grant No.: 638605. The Augmented Human Lab at the University of Auckland thanks the
supported of Assistive Augmentation research grant under the
Entrepreneurial Universities (EU) initiative of New Zealand.
Finally, we thank the Leibnitz Zenrum für Informatik for hosting us at Schloss Dagstuhl.
REFERENCES[1] Yomna Abdelrahman, Eduardo Velloso, Tilman Dingler,
Albrecht Schmidt, and Frank Vetere. 2017. Cognitive
heat: exploring the usage of thermal imaging to

unobtrusively estimate cognitive load. Proceedings of
the ACM on Interactive, Mobile, Wearable andUbiquitous Technologies 1, 3 (2017), 33.
[2] Ehud Ahissar and Eldad Assa. 2016. Perception as aclosed-loop convergence process. eLife 5, MAY2016(2016). DOI:http://dx.doi.org/10.7554/eLife.12830[3] Jason Alexander, Anne Roudaut, Jürgen Steimle, Kasper
Hornbæk, Miguel Bruns Alonso, Sean Follmer, andTimothy Merritt. 2018. Grand Challenges in
Shape-Changing Interface Research. In Proceedings ofthe 2018 CHI Conference on Human Factors in
Computing Systems (CHI ’18). ACM, New York, NY,
USA, Article 299, 14 pages. DOI:
http://dx.doi.org/10.1145/3173574.3173873[4] Susan M Andersen and Serena Chen. 2002. The
relational self: an interpersonal social-cognitive theory.
Psychological review 109, 4 (2002), 619.
[5] Josh Andres, Julian de Hoog, and Florian’Floyd’
Mueller. 2018. I had super-powers when eBike riding
Towards Understanding the Design of Integrated
Exertion. In Proceedings of the 2018 Annual Symposiumon Computer-Human Interaction in Play. ACM, 19–31.
[6] W Ross Ashby. 1961. An introduction to cybernetics.
Chapman & Hall Ltd.
[7] Nicholas J Belkin. 2008. Some (what) grand challenges
for information retrieval. In ACM SIGIR Forum, Vol. 42.
ACM, 47–54.
[8] Joanna Bergstrom-Lehtovirta, Sebastian Boring, andKasper Hornbæk. 2017. Placing and Recalling Virtual
Items on the Skin. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems(CHI ’17). ACM, New York, NY, USA, 1497–1507.
DOI:http://dx.doi.org/10.1145/3025453.3026030[9] Joanna Bergstrom-Lehtovirta, David Coyle, Jarrod
Knibbe, and Kasper Hornbæk. 2018. I Really Did That:
Sense of Agency with Touchpad, Keyboard, andOn-skin Interaction. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, Article 378, 8
pages. DOI:http://dx.doi.org/10.1145/3173574.3173952[10] Mason Bretan and Gil Weinberg. 2017. Integrating the
Cognitive with the Physical: Musical Path Planning for
an Improvising Robot.. In AAAI. 4371–4377.
[11] Taylor Carman. 1999. The Body in Husserl andMerleau-Ponty. Philosophical topics 27, 2 (1999),
205–226.
[12] Min Chen, Sergio Gonzalez, Athanasios Vasilakos,
Huasong Cao, and Victor C. Leung. 2011. Body Area
Networks: A Survey. Mob. Netw. Appl. 16, 2 (April2011), 171–193. DOI:
http://dx.doi.org/10.1007/s11036-010-0260-8[13] Andy Clark. 2001. Natural-born cyborgs? In Cognitivetechnology: Instruments of mind. Springer, 17–24.

[14] David Coyle, James Moore, Per Ola Kristensson, Paul
Fletcher, and Alan Blackwell. 2012. I Did That!
Measuring Users’ Experience of Agency in Their Own
Actions. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI ’12). ACM,
New York, NY, USA, 2025–2034. DOI:
http://dx.doi.org/10.1145/2207676.2208350[15] Amy JC Cuddy, Susan T Fiske, and Peter Glick. 2008.
Warmth and competence as universal dimensions ofsocial perception: The stereotype content model and the
BIAS map. Advances in experimental social psychology
40 (2008), 61–149.
[16] Margo DeMello. 2007. Encyclopedia of bodyadornment. ABC-CLIO.
[17] Henry Dreyfuss, Henry Dreyfuss Associates, andAlvin R Tilley. 1993. The measure of man and woman:human factors in design. Whitney Library of Design.
[18] Douglas C Engelbart. 2001. Augmenting human
intellect: a conceptual framework (1962). PACKER,
Randall and JORDAN, Ken. Multimedia. From Wagner
to Virtual Reality. New York: WW Norton & Company(2001), 64–90.
[19] Umer Farooq and Jonathan Grudin. 2016.
Human-computer integration. interactions 23, 6 (2016),
26–32.
[20] Umer Farooq, Jonathan Grudin, Ben Shneiderman,
Pattie Maes, and Xiangshi Ren. 2017. Human Computer
Integration versus Powerful Tools. In Proceedings of the2017 CHI Conference Extended Abstracts on Human
Factors in Computing Systems (CHI EA ’17).
Association for Computing Machinery, New York, NY,
USA, 1277–1282. DOI:
http://dx.doi.org/10.1145/3027063.3051137[21] Umer Farooq and Jonathan T. Grudin. 2017. Paradigm
Shift from Human Computer Interaction to Integration.
In Proceedings of the 2017 CHI Conference Extended
Abstracts on Human Factors in Computing Systems(CHI EA ’17). Association for Computing Machinery,
New York, NY, USA, 1360–1363. DOI:
http://dx.doi.org/10.1145/3027063.3049285[22] Pablo Gallego Cascón, Denys J.C. Matthies, Sachith
Muthukumarana, and Suranga Nanayakkara. 2019.
ChewIt. An Intraoral Interface for Discreet Interactions.
In Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems (CHI ’19). ACM, New
York, NY, USA, Article 326, 13 pages. DOI:
http://dx.doi.org/10.1145/3290605.3300556[23] Ashok K Goel and Spencer Rugaber. 2015. Interactive
meta-reasoning: Towards a CAD-like environment for
designing game-playing agents. In Computational
creativity research: Towards creative machines.
Springer, 347–370.
[24] Amal Graafstra, Katina Michael, and MG Michael.
2010. Social-technical issues facing the humancentric
RFID implantee sub-culture through the eyes of Amal
Graafstra. In 2010 IEEE International Symposium on
Technology and Society. IEEE, 498–516.

[25] Saul Greenberg, Sebastian Boring, Jo Vermeulen, andJakub Dostal. 2014. Dark patterns in proxemic
interactions: a critical perspective. In Proceedings of the2014 conference on Designing interactive systems.
ACM, 523–532.
[26] Mariam Hassib, Stefan Schneegass, Philipp Eiglsperger,
Niels Henze, Albrecht Schmidt, and Florian Alt. 2017.
EngageMeter: A system for implicit audienceengagement sensing using electroencephalography. In
Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems. ACM, 5114–5119.
[27] Kayla J. Heffernan, Frank Vetere, and Shanton Chang.
2016. You Put What, Where?: Hobbyist Use of
Insertable Devices. In Proceedings of the 2016 CHI
Conference on Human Factors in Computing Systems(CHI ’16). ACM, New York, NY, USA, 1798–1809.
DOI:http://dx.doi.org/10.1145/2858036.2858392[28] Christian Holz, Tovi Grossman, George Fitzmaurice,and Anne Agur. 2012. Implanted User Interfaces. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI ’12). ACM, New
York, NY, USA, 503–512. DOI:
http://dx.doi.org/10.1145/2207676.2207745[29] Sarah Homewood and Clint Heyer. 2017. Turned On /Turned Off: Speculating on the Microchip-based
Contraceptive Implant. In Proceedings of the 2017
Conference on Designing Interactive Systems (DIS ’17).
ACM, New York, NY, USA, 339–343. DOI:
http://dx.doi.org/10.1145/3064663.3064726[30] Don Ihde. 1990. Technology and the lifeworld : fromgarden to earth. Indiana University Press. 226 pages.
[31] Somayeh Imani, Amay J Bandodkar, AM Vinu Mohan,
Rajan Kumar, Shengfei Yu, Joseph Wang, and Patrick P
Mercier. 2016. A wearable
chemical–electrophysiological hybrid biosensing system
for real-time health and fitness monitoring. Naturecommunications 7 (2016), 11650.
[32] Wendy Ju. 2015. The design of implicit interactions.
Synthesis Lectures on Human-Centered Informatics 8, 2(2015), 1–93.
[33] Hsin-Liu (Cindy) Kao, Christian Holz, Asta Roseway,
Andres Calvo, and Chris Schmandt. 2016. DuoSkin:
Rapidly Prototyping On-skin User Interfaces Using
Skin-friendly Materials. In Proceedings of the 2016
ACM International Symposium on Wearable Computers(ISWC ’16). ACM, New York, NY, USA, 16–23. DOI:
http://dx.doi.org/10.1145/2971763.2971777[34] Shunichi Kasahara, Jun Nishida, and Pedro Lopes. 2019.
Preemptive Action: Accelerating Human Reaction
Using Electrical Muscle Stimulation Without
Compromising Agency. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems(CHI ’19). ACM, New York, NY, USA, Article 643, 15
pages. DOI:http://dx.doi.org/10.1145/3290605.3300873

[35] Paul G Keil. 2015. Human-Sheepdog Distributed
Cognitive Systems: An Analysis of Interspecies
Cognitive Scaffolding in a Sheepdog Trial. Journal of
Cognition and Culture 15, 5 (2015), 508–529.

[45] Pedro Lopes and Patrick Baudisch. 2017. Interactive
Systems Based on Electrical Muscle Stimulation.
Computer 50, 10 (2017), 28–35. DOI:

[36] Kevin Kelly. 2009. Out of control: The new biology ofmachines, social systems, and the economic world.
Hachette UK.

[46] Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel
Hoffmann, Patrik Jonell, and Patrick Baudisch. 2015.
Proprioceptive Interaction. In Proceedings of the 33rd
Annual ACM Conference on Human Factors in
Computing Systems (CHI âĂŹ15). Association for
Computing Machinery, New York, NY, USA,939âĂŞ948. DOI:[37] Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein,
Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt
Lease, and John Horton. 2013. The future of crowd work.
In Proceedings of the 2013 conference on Computer
supported cooperative work. ACM, 1301–1318.
[38] Jarrod Knibbe, Paul Strohmeier, Sebastian Boring, andKasper Hornbæk. 2017. Automatic calibration of highdensity electric muscle stimulation. Proceedings of the
ACM on Interactive, Mobile, Wearable and UbiquitousTechnologies 1, 3 (2017), 68.
[39] Marion Koelle, Matthias Kranz, and Andreas Möller.
2015. Don’T Look at Me That Way!: Understanding
User Attitudes Towards Data Glasses Usage. In
Proceedings of the 17th International Conference on
Human-Computer Interaction with Mobile Devices andServices (MobileHCI ’15). ACM, New York, NY, USA,362–372. DOI:
http://dx.doi.org/10.1145/2785830.2785842[40] Marion Koelle, Katrin Wolf, and Susanne Boll. 2018.
Beyond LED Status Lights - Design Requirements ofPrivacy Notices for Body-worn Cameras. In
Proceedings of the Twelfth International Conference on
Tangible, Embedded, and Embodied Interaction (TEI’18). ACM, New York, NY, USA, 177–187. DOI:
http://dx.doi.org/10.1145/3173225.3173234[41] Zhuying Li, Felix Brandmueller, Stefan Greuter, andFlorian Mueller. 2018a. The Guts Game: Designing
Playful Experiences for Ingestible Devices. In Extended
Abstracts of the 2018 CHI Conference on Human
Factors in Computing Systems (CHI EA ’18). ACM,
New York, NY, USA, Article VS12, 1 pages. DOI:
http://dx.doi.org/10.1145/3170427.3186604[42] Zhuying Li, Rakesh Patibanda, Felix Brandmueller, Wei
Wang, Kyle Berean, Stefan Greuter, and Florian Mueller.
2018b. The Guts Game: Towards Designing Ingestible
Games. In Proceedings of the 2018 Annual Symposiumon Computer-Human Interaction in Play (CHI PLAY
’18). ACM, New York, NY, USA, 271–283. DOI:
http://dx.doi.org/10.1145/3242671.3242681[43] Joseph CR Licklider. 1960. Man-computer symbiosis.
IRE transactions on human factors in electronics 1(1960), 4–11.
[44] Joanne Lo, Doris Jung Lin Lee, Nathan Wong, David
Bui, and Eric Paulos. 2016. Skintillates: Designing andCreating Epidermal Interactions. In Proceedings of the2016 ACM Conference on Designing Interactive Systems(DIS ’16). ACM, New York, NY, USA, 853–864. DOI:
http://dx.doi.org/10.1145/2901790.2901885

http://dx.doi.org/10.1109/MC.2017.3641627

http://dx.doi.org/10.1145/2702123.2702461[47] Pedro Lopes, Ryan Lavoie, Rishi Faldu, Nick Aquino,
Jason Barron, Mohamed Kante, Basel Magfory, andWaleed Meleis. 2012. Eye-Controed Robotic Feeding
Arm Technology. (2012).
[48] Pedro Lopes, Doga Yüksel, François Guimbretière, andPatrick Baudisch. 2016. Muscle-plotter: An interactive
system based on electrical muscle stimulation that
produces spatial output. In Proceedings of the 29th
Annual Symposium on User Interface Software andTechnology. ACM, 207–217.
[49] Eric MacDonald and Ryan Wicker. 2016. Multiprocess
3D printing for increasing component functionality.
Science 353, 6307 (2016). DOI:
http://dx.doi.org/10.1126/science.aaf2093[50] Steve Mann. 2001. Wearable computing: Toward
humanistic intelligence. IEEE Intelligent Systems 16, 3(2001), 10–15.
[51] Denys JC Matthies, Bodo Urban, Katrin Wolf, andAlbrecht Schmidt. 2019. Reflexive Interaction Extending the concept of Peripheral Interaction. In
Proceedings of the 31st Australian Conference on
Human-Computer-Interaction (OZCHI’19). ACM. DOI:
http://dx.doi.org/10.1145/3369457.3369478[52] Maryse Maurel. 2009. The explicitation interview:
examples and applications. Journal of Consciousness
Studies 16, 10-11 (2009), 58–89.
[53] Alexis McCrossen. 2013. Marking Modern Times: A
History of Clocks, Watches, and Other Timekeepers in
American Life. University of Chicago Press.
[54] Aske Mottelson and Kasper Hornbæk. 2016. An Affect
Detection Technique Using Mobile Commodity Sensors
in the Wild. In Proceedings of the 2016 ACM
International Joint Conference on Pervasive andUbiquitous Computing (UbiComp ’16). ACM, New
York, NY, USA, 781–792. DOI:
http://dx.doi.org/10.1145/2971648.2971654[55] Aske Mottelson, Jarrod Knibbe, and Kasper Hornbæk.
2018. Veritaps: Truth Estimation from Mobile
Interaction. In Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (CHI ’18).
ACM, New York, NY, USA, Article 561, 12 pages. DOI:
http://dx.doi.org/10.1145/3173574.3174135

[56] Florian ’Floyd’ Mueller and Sarah Jane Pell. 2016.
Technology Meets Adventure: Learnings from an
Earthquake-interrupted Mt. Everest Expedition. In
Proceedings of the 2016 ACM International Joint
Conference on Pervasive and Ubiquitous Computing(UbiComp ’16). ACM, New York, NY, USA, 817–828.
DOI:http://dx.doi.org/10.1145/2971648.2971683[57] Jun Nishida, Soichiro Matsuda, Mika Oki, Hikaru
Takatori, Kosuke Sato, and Kenji Suzuki. 2019.
Egocentric Smaller-Person Experience through a
Change in Visual Perspective. In Proceedings of the2019 CHI Conference on Human Factors in Computing
Systems (CHI ’19). Association for Computing
Machinery, New York, NY, USA, Article Paper 696, 12
pages. DOI:http://dx.doi.org/10.1145/3290605.3300926[58] Aditya Shekhar Nittala, Anusha Withana, Narjes
Pourjafarian, and Jürgen Steimle. 2018. Multi-Touch
Skin: A Thin and Flexible Multi-Touch Sensor for
On-Skin Input. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, Article 33, 12
pages. DOI:http://dx.doi.org/10.1145/3173574.3173607[59] Marianna Obrist, Carlos Velasco, Chi Vi, Nimesha
Ranasinghe, Ali Israr, Adrian Cheok, Charles Spence,
and Ponnampalam Gopalakrishnakone. 2016. Sensing
the future of HCI: touch, taste, and smell user interfaces.
interactions 23, 5 (2016), 40–49.
[60] Simon Olberding, Nan-Wei Gong, John Tiab, Joseph A.
Paradiso, and Jürgen Steimle. 2013. A Cuttable
Multi-touch Sensor. In Proceedings of the 26th Annual
ACM Symposium on User Interface Software andTechnology (UIST ’13). ACM, New York, NY, USA,245–254. DOI:
http://dx.doi.org/10.1145/2501988.2502048[61] Abdenbi Mohand Ousaid, Dogan Sinan Haliyo,
Stephane Regnier, and Vincent Hayward. 2015. A Stable
and Transparent Microscale Force Feedback
Teleoperation System. IEEE/ASME Transactions on
Mechatronics 20, 5 (2015), 2593–2603. DOI:
http://dx.doi.org/10.1109/TMECH.2015.2423092[62] Abdenbi Mohand Ousaid, Guillaume Millet, Sinan
Haliyo, Stéphane Régnier, and Vincent Hayward. 2014.
Feeling what an insect feels. PLoS ONE 9, 10 (2014),
e108895. DOI:
http://dx.doi.org/10.1371/journal.pone.0108895[63] Halley Profita, Reem Albaghli, Leah Findlater, Paul
Jaeger, and Shaun K. Kane. 2016. The AT Effect: How
Disability Affects the Perceived Social Acceptability of
Head-Mounted Display Use. In Proceedings of the 2016
CHI Conference on Human Factors in Computing
Systems (CHI ’16). ACM, New York, NY, USA,4884–4895. DOI:
http://dx.doi.org/10.1145/2858036.2858130[64] Halley P. Profita, Abigale Stangl, Laura Matuszewska,
Sigrunn Sky, Raja Kushalnagar, and Shaun K. Kane.
2018. "Wear It Loud": How and Why Hearing Aid andCochlear Implant Users Customize Their Devices. ACM
Trans. Access. Comput. 11, 3, Article 13 (Sept. 2018),
32 pages. DOI:http://dx.doi.org/10.1145/3214382[65] Howard Rheingold. 2013. Mind Amplifier: Can Our
Digital Tools Make Us Smarter?. Ted Conferences.
[66] Giuseppe Riva, Monica Bacchetta, Margherita Baruffi,
and Enrico Molinari. 2001. Virtual reality–based
multidimensional therapy for the treatment of body
image disturbances in obesity: a controlled study.
Cyberpsychology & behavior 4, 4 (2001), 511–526.
[67] Joseph M. Romano and Katherine J. Kuchenbecker.
2012. Creating realistic virtual textures from contact
acceleration data. IEEE Transactions on Haptics 5, 2(2012), 109–119. DOI:
http://dx.doi.org/10.1109/TOH.2011.38[68] Himanshu Sahni, Abdelkareem Bedri, Gabriel Reyes,
Pavleen Thukral, Zehua Guo, Thad Starner, andMaysam Ghovanloo. 2014. The Tongue and Ear
Interface: A Wearable System for Silent Speech
Recognition. In Proceedings of the 2014 ACM
International Symposium on Wearable Computers(ISWC ’14). ACM, New York, NY, USA, 47–54. DOI:
http://dx.doi.org/10.1145/2634317.2634322[69] MHD Yamen Saraiji, Tomoya Sasaki, Kai Kunze, Kouta
Minamizawa, and Masahiko Inami. 2018. MetaArms:
Body Remapping Using Feet-Controlled Artificial Arms.
In Proceedings of the 31st Annual ACM Symposium on
User Interface Software and Technology (UIST ’18).
Association for Computing Machinery, New York, NY,
USA, 65–74. DOI:
http://dx.doi.org/10.1145/3242587.3242665[70] Tomoya Sasaki, MHD Yamen Saraiji, Charith Lasantha
Fernando, Kouta Minamizawa, and Masahiko Inami.
2017. MetaLimbs: Multiple Arms Interaction
Metamorphism. In ACM SIGGRAPH 2017 Emerging
Technologies (SIGGRAPH ’17). ACM, New York, NY,
USA, Article 16, 2 pages. DOI:
http://dx.doi.org/10.1145/3084822.3084837[71] Edward Sazonov and Michael R Neuman. 2014.
Wearable Sensors: Fundamentals, implementation andapplications. Elsevier.
[72] Valentin Schwind, Jens Reinhardt, Rufat Rzayev, Niels
Henze, and Katrin Wolf. 2018. Virtual Reality on the
Go?: A Study on Social Acceptance of VR Glasses. In
Proceedings of the 20th International Conference on
Human-Computer Interaction with Mobile Devices andServices Adjunct (MobileHCI ’18). ACM, New York,NY, USA, 111–118. DOI:
http://dx.doi.org/10.1145/3236112.3236127[73] Caitlyn Seim, John Chandler, Kayla DesPortes,
Siddharth Dhingra, Miru Park, and Thad Starner. 2014.
Passive haptic learning of Braille typing. In Proceedingsof the 2014 ACM International Symposium on Wearable
Computers. ACM, 111–118.

[74] Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie
Maes, and Suranga Chandima Nanayakkara. 2015.
FingerReader: A Wearable Device to Explore Printed
Text on the Go. In Proceedings of the 33rd Annual ACM
Conference on Human Factors in Computing Systems(CHI ’15). ACM, New York, NY, USA, 2363–2372.
DOI:http://dx.doi.org/10.1145/2702123.2702421[75] Ben Shneiderman, Catherine Plaisant, Maxine Cohen,
Steven Jacobs, Niklas Elmqvist, and Nicholoas
Diakopoulos. 2016. Grand challenges for HCI
researchers. interactions 23, 5 (2016), 24–25.
[76] Garrett J Soler, Mengdi Bao, Devina Jaiswal, Hitten P
Zaveri, Michael L DiLuna, Ryan A Grant, and Kazunori
Hoshino. 2018. Focus: Medical Technology: A Review
of Cerebral Shunts, Current Technologies, and Future
Endeavors. The Yale Journal of Biology and Medicine
91, 3 (2018), 313.
[77] Charles Spence, Marianna Obrist, Carlos Velasco, andNimesha Ranasinghe. 2017. Digitizing the chemical
senses: possibilities & pitfalls. International Journal of
Human-Computer Studies 107 (2017), 62–74.
[78] Thad Starner. 2001. The challenges of wearable
computing: Part 2. Ieee Micro 21, 4 (2001), 54–67.
[79] Jürgen Steimle. 2015. Printed Electronics for
Human-computer Interaction. interactions 22, 3 (April2015), 72–75. DOI:http://dx.doi.org/10.1145/2754304[80] Jürgen Steimle. 2016. Skin–The Next User Interface.
Computer 49, 4 (2016), 83–87.
[81] Stelarc. 1993. Stomach Sculpture. (1993).
https://stelarc.org/?catID=20349 Accessed:
2018-09-19.
[82] Stelarc. 96. Ping Body. (96).
http://v2.nl/events/ping-body[83] Paul Strohmeier, Sebastian Boring, and Kasper
Hornbæk. 2018. From Pulse Trains to "Coloring withVibrations": Motion Mappings for Mid-Air Haptic
Textures. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems (CHI ’18).
Association for Computing Machinery, New York, NY,
USA, Article Paper 65, 13 pages. DOI:
http://dx.doi.org/10.1145/3173574.3173639[84] Paul Strohmeier, Cedric Honnet, and Samppa Von
Cyborg. 2016. Developing an Ecosystem for Interactive
Electronic Implants. Proc. Living Machines 2016 (2016).
DOI:http://dx.doi.org/10.1007/978-3-319-42417-0[85] Ivan E Sutherland. 1965. The ultimate display.
Multimedia: From Wagner to virtual reality (1965),
506–508.
[86] Dag Svanaes and Martin Solheim. 2016. Wag Your Tailand Flap Your Ears: The Kinesthetic User Experience ofExtending Your Body. In Proceedings of the 2016 CHI
Conference Extended Abstracts on Human Factors in
Computing Systems. ACM, 3778–3779.
[87] Milos Taborsky, Tomas Skala, Martin Kocher, Marian
Fedorco, and others. 2018. Extraction of a dislocatedleadless pacemaker in a patient with infective
endocarditis and repeated endocardial and epicardial
pacing system infections. (2018).
[88] Henri Tajfel. 1974. Social identity and intergroup
behaviour. Information (International Social Science
Council) 13, 2 (1974), 65–93.
[89] Adriana Tapus, Maja J Mataric, and Brian Scassellati.
2007. Socially assistive robotics [grand challenges ofrobotics]. IEEE Robotics & Automation Magazine 14, 1(2007), 35–42.
[90] Katia Vega and Hugo Fuks. 2013. Beauty Technology
As an Interactive Computing Platform. In Proceedingsof the 2013 ACM International Conference on
Interactive Tabletops and Surfaces (ITS ’13). ACM,
New York, NY, USA, 357–360. DOI:
http://dx.doi.org/10.1145/2512349.2512399[91] Carlos Velasco, Marianna Obrist, Olivia Petit, andCharles Spence. 2018. Multisensory technology for
flavor augmentation: a mini review. Frontiers in
psychology 9 (2018), 26.
[92] Peter-Paul Verbeek. 2005. What Things Do:
Philosophical Reflections on Technology, Agency and
Design. Pennsylvania State University Press,
Pennsylvania.
[93] Aaron I Vinik, A Gordon Smith, J Robinson Singleton,
Brian Callaghan, Barry I Freedman, Jaakko Tuomilehto,
Lyse Bordier, Bernard Bauduceau, and Frederic Roche.
2016. Normative values for electrochemical skinconductances and impact of ethnicity on quantitative
assessment of sudomotor function. Diabetes technology& therapeutics 18, 6 (2016), 391–398.
[94] Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta,
Carmel Majidi, and Jürgen Steimle. 2015. iSkin:
Flexible, Stretchable and Visually Customizable
On-Body Touch Sensors for Mobile Computing. In
Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems (CHI ’15). ACM,
New York, NY, USA, 2991–3000. DOI:
http://dx.doi.org/10.1145/2702123.2702391[95] Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, andJürgen Steimle. 2017. SkinMarks: Enabling Interactions
on Body Landmarks Using Conformal Skin Electronics.
In Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems (CHI ’17). ACM, New
York, NY, USA, 3095–3105. DOI:
http://dx.doi.org/10.1145/3025453.3025704[96] Mark Weiser. 1991. The Computer for the 21 st Century.
Scientific american 265, 3 (1991), 94–105.
[97] Anusha Withana, Daniel Groeger, and Jürgen Steimle.
2018. Tacttoo: A Thin and Feel-Through Tattoo for
On-Skin Tactile Output. In Proceedings of the 31st
Annual ACM Symposium on User Interface Software
and Technology (UIST ’18). ACM, New York, NY, USA,
365–378. DOI:
http://dx.doi.org/10.1145/3242587.3242645

[98] Katrin Wolf and Timm Bäder. 2015. Illusion of surfacechanges induced by tactile and visual touch feedback. In
Proceedings of the 33rd Annual ACM Conference
Extended Abstracts on Human Factors in Computing
Systems. ACM, 1355–1360.

[99] Xiuxiu Yuan. 2018. User specific assistive technology:
hand mounted switch control platform design. Ph.D.
Dissertation. Georgia Institute of Technology.