# Defining a Role-Centered Terminology for Physical Representations and Controls

Guillaume Rivière

<figure><img src="image_1.png" /></figure>

## To cite this version:

Guillaume Rivière. Defining a Role-Centered Terminology for Physical Representations and Controls. 2025. hal-05454944

HAL Id: hal-05454944

https://hal.science/hal-05454944v1

Preprint submitted on 12 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Defining a Role-Centered Terminology for Physical Representations and Controls

Guillaume Rivière

g.riviere@estia.fr

Univ. Bordeaux, ESTIA Institute of Technology

Bidart, France

## Abstract

Previous classifications advanced research through a better understanding of the field and the variety of tangible user interfaces and related physical user interfaces, especially by discretizing a degree of tangibility based on the specimens produced by the community over the years, since the conceptualization of Tangible User Interface initiated a research effort to deepen the exploration of the concept. However, no taxonomy enables the classification of tangible user interfaces at the application level. This article proposes to refine the description of tangible user interfaces' interactional components through a terminological approach. The resulting terms are blended words, built from known words, that self-contain *what* digital role is represented or controlled and *how* it becomes physical. This holistic terminology then enables the definition of applications' hallmarks and four classes of tangibility for applications, which surpass the description of physical user interface specimens' morphology by abstracting and discriminating specimens at the applicative level. The descriptiveness and holisticness of the new terminology, as well as the clustering and discriminative power of the limited number of four classes, are showed on a corpus of applicative tangible user interfaces' specimens from the literature. Promising future work will benefit from the holistic terminology, the applications' hallmarks, and the tangibility classes, to describe applicative tangible user interfaces and related physical user interfaces to better understand the dozens of specimens that were produced by the field over three decades. Indeed, describing and classifying this whole set would deepen our understanding to provide tools for future developers and designers.

## CCS Concepts

* Human-centered computing → HCI theory, concepts and models.

## Keywords

terminology, linguistics, blended words, physical computing, physical user interfaces, graspable user interfaces, tangible user interfaces, embodied interfaces, taxonomy, classification

## ACM Reference Format:

Guillaume Rivière. 2025. Defining a Role-Centered Terminology for Physical Representations and Controls. *arXiv.org, Cornell University, Ithaca, NY, USA*. 17 pages.

<figure><img src="image_3.png" /></figure>

## 1 Introduction

Conceptualization often precedes a community's focus on the same research path. Whereas precursor specimens—of what was later conceptualized as Tangible User Interfaces (TUIs)—arose in the mid-1970s and early 1980s [70: Ch. 2, 81], physical computing came to the forefront of Human-Computer Interaction (HCI) researchers' attention only fifteen years later [22, 35, 86], after the concept of ubiquitous computing was conceptualized in the late 1980s [84, 85]. Several works initiated physical computing's exploration, from "passive real-world props" [35], to "graspable user interfaces" [22], to "manipulative user interfaces" [33], to "embodied user interfaces" [20], which were the growing roots of the same tree [19]. Likewise, the deeper exploration of the design and technology of tangible user interfaces began after they were conceptualized in 1997 [75]. Since then, while the number of specimens has grown yearly, taxonomies, paradigms, and frameworks have emerged [37, 59, 70: Ch. 5], defining terms to describe the roles of physical artifacts (e.g., phicon [75], token [36, 71, 79], and tangible representation [77]). However, no global terminology is shared in the literature, even if refining and conceptualizing tangible user interfaces' interactive components better could help refine the design space, deepen specimens' exploration, and broaden applications' possibilities—indeed, sharing a common vocabulary and the same concepts also helps aggregating research efforts on a common path.

Previous work provided isolated terms, common terms, and general terms, but none can distinguish all the entities involved in tangible user interfaces in a single namespace. On the one hand, isolated terms are limited in the scopes and paradigms that define them (e.g., Token And Constraints [71]), and, therefore, remain separated in diverse namespaces. On the other hand, some terms are found across several research works, but with inconsistent meanings (e.g., "tokens" [36, 71, 79]). As a consequence, many researches are built upon general terms (e.g., "tangibles," "physical object," and "tangible object" [70]), thus escaping terminological ambiguity and describing all physical artifacts as a whole set, without distinction. However, such general terms remain broad and undescriptive. For instance, the term "tangibles" is plural and cannot name merely a single artifact. The terms "physical object" and "tangible object" lend a new meaning to the term "object", which is already overused in computer science [71]. Finally, "objects" are already "physical" and "tangible" in the usual sense.

Furthermore, scopes other than pure tangible user interfaces resort to physical entities as representations of controls, but not especially for data representation (e.g., in Augmented Reality [13] and Virtual Reality [58] scopes). The lack of precise terms can lead to consider any user interface resorting to physical objects as a "tangible user interface," even if these objects are only a subpart
of a user interface that does not represent data with physical
bodiement, thus leading to confusion and misunderstanding (e.g.,
before general frameworks, recognizing tangible user interfaces
was mainly relying on “I know one when I see one” approaches [19]).
Fishkin’s taxonomy overcame these considerations by “treating
tangibility as a spectrum rather than a binary quantity” (i.e.,
“tangible” and “not tangible” user interfaces) and organizing
“proof of concept” specimens over a twenty-item discretized space
[19]. However, this taxonomy, which has contributed to better
delineating and understanding the field, does not address tangible
user interfaces at the applicative level (i.e., user interfaces that
gather and assemble components to meet the needs of final
users). Indeed, this taxonomy is mainly focused on analyzing data
representations (i.e., distance between data output and the inputs
enabling data manipulation and edition, and the user actions’ degree
of analogy to perceive and manipulate data) and describes user
interfaces as whole things (i.e., without refining the components
that compose applicative user interfaces). Except that such a global
description requires determining what is “the” datum of the user
interface, but this often becomes tricky in applicative user interfaces,
which often involve multiple tasks, outputs, or users. Although
describing the morphology and meaning of tangible user interfaces
globally enabled refining the concept of “tangibility” in physical
user interfaces, developing applications of such user interfaces
may require describing components separately and beyond data
representations.

For instance, the Urp specimen (Urban Planning Workbench) [5, 41, 82] melts, on the surface of a table, models of buildings with editing and navigation tools and with measurement objects, and all of these are augmented with video-projection to display information from the computational simulation. Describing this “workbench” application globally, as a whole thing, ignores all the subtleties of its functioning and, for example, the role played by intangible representations. As well, determining the distance between input and output fails when an applicative tangible user interface only conveys data (i.e., no input).

In addition, terms describing intangible representations are
underrepresented: all the previous terms—isolated, common, and
general ones—focused nearly exclusively on detailing tangible
representations (whereas many terms were defined for GUIs, such
as icons, windows, or sliders). Consequently, could we discriminate,
in a single namespace, all the tangible, graspable, and intangible
entities found in tangible user interfaces (which represent data
physically) and in any other related physical user interfaces? Since
previous work provided a profusion of specimens, could we use the
refined terminology to better discriminate tangible user interfaces
according to the description of their components? Consequently,
could such a refined terminology of physical entities lead to a
classification of applicative tangible user interfaces?

This article proposes a role-centered terminology that names
physical entities by defining twelve terms that self-contain what
is represented or controlled and how. This terminology serves
two objectives: the first is to provide terms that better align
with foundational definitions, thereby improving conceptual
understanding; the second is to refine the description of the
representations and controls involved in tangible and related
physical user interfaces. To show the descriptive power of this

terminology, this article successfully discriminates all the physical
representations and controls found in a collection of applicative
tangible user interfaces. Furthermore, this holistic discrimination
also enables the classification of applicative tangible user interfaces
into four tangibility classes. We envision and expect that such
terminological and classification advances will serve as a foundation
for future work to investigate the composition of previous
applicative tangible user interfaces and related physical user
interfaces in greater depth and to provide tools and toolkits for
future designers.

After reviewing related work, this article introduces a new
terminology based on a taxonomy with two axes, and then derives
four tangibility classes of applicative tangible user interfaces
from this terminology. Subsequently, the descriptiveness of the
terminology is validated on a corpus of applicative tangible
user interfaces, which are then clustered into the four classes.
Then, previous terms from the literature are mapped to the
new terminology, and the four classes are linked to previous
classifications. Finally, before concluding, some limitations are
highlighted, and directions are provided for future work.

2 Related Work

This section reviews the related work on terms for physical
representations and controls in tangible user interfaces, the creation
of new words in the usual language through the word blending,
and taxonomies and classifications of tangible user interfaces.

## 2.1 Terms for Physical Representations

The term “tangible” appeared early to describe a “tangible model”
for 3D input in Computer-Aided Architectural Design (CAAD)
[2] and “tangible manipulation” in the Digital Desk [86] (but the
latter was later renamed “tactile interaction” [87]). Later, the terms
“graspable” and “tangible” were introduced to define Graspable
User Interfaces [22] and Tangible User Interfaces [75], respectively.
Finally, the term “intangible” appeared to name audio and video
output modalities in the MCRit model<fnref n="1" /> [74, 79]. Whereas the
term “tangible” is used close to dictionary definitions, the term
“intangible” differs from the usual sense: it simply means “not
tangible.” Therefore, intangible representations are those generated
by the computer; they may disappear when the computer is
turned off (e.g., sound, video projection, and any audio and visual
modalities or displays).

This section summarizes the usual terms used in the literature
to name the interactional components of tangible user interfaces,
categorized into general, isolated, and shared terms, as well as terms
employed in specific toolkits and design tools.

2.1.1 General Terms. Some terms are used in the literature to generically name the physical part of tangible user interfaces that embody digital information, including “physical artifacts” [77], “physical representations” [77], “tangible representations” [74, 79], “tangibles” [74], and “tangible objects” [70]. Those terms are variations to name graspable and tangible interactional components broadly. However, the common use of the term “tangible” was initially defined for bodied data representations (i.e., not for

<footnote n="1">¹MCRit: “Model-Control-Representation (intangible and tangible)” [74, 79], first known
as MCRpd: “Model-Control-Representation (physical and digital)” [77].</footnote>
graspable anchors on tabletops). Finally, only one general term refers to neither graspable nor tangible components: “intangible representations” [74, 79].

### 2.1.2 Isolated Terms.
The term *passive real-world prop* [35] is used to name bodied representations of tools and data that map sensed user actions directly to control digital data displayed on a distant screen. Props can be *symbolic* [77] (e.g., a sphere [35]) or *iconic* [77] (e.g., a doll's head [35]) data representations, combined with tools (e.g., a slicing plate [35]). The term “passive” refers to the system’s ability to track and reproduce movements made on the props. We can understand the choice of this term with the first loop of tangible interaction [40]: props provide a first passive feedback loop before computation’s feedback arises—the relative position of the props informs about cutting plane location without watching the screen (what is reinforced when using the doll’s head [34]).

The term *physical handle* [22] (or *phandle* [75]) refers to physical artifacts of general form (e.g., bricks and pucks) that can be attached to intangible shapes displayed on a surface. Once the physical handle acquires an intangible shape, motions are transmitted to control the digital object (e.g., position, orientation, and size).

The term *phicon* [75] stands for “physical icons” of a specialized form that instantiates graphical user interface (GUI) components physically. Once put on a horizontal surface, phicons are linked to digital data (e.g., a building [75]), and intangible data is updated to match phicons’ physical states (e.g., a displayed map is updated [75]).

### 2.1.3 Shared Terms.
Some terms do not name the artifacts by what they are but by their role and appear across several research works. For example, the terms “token,” “container,” and “tool” [36] define three roles in giving access to digital information. Tokens are specialized artifacts whose physical properties match the represented digital information (e.g., phicons [75]). Containers are generic artifacts that can represent any type of information. Whereas tokens are continuously assigned to the same digital information, containers are associated with any digital information over time. Finally, tools are physical artifacts that have computational functions, enabling them to act on the displayed data (e.g., physical handles [22]). Beyond naming physical artifacts, access points to digital information are also named: digital information is rendered by information faucets (e.g., display devices, speakers, and tactile devices). However, the “Token+Constraint” approach [79] suggests that all those artifacts are “tokens”: “containers” are “symbolic tokens with permanent binding,” “tokens” are “iconic tokens with dynamic binding,” and “tools” are “tokens bound to operations” [79].

The “Token+Constraint” approach [79] reuses the term “token” in its usual sense, which is also its standard usage in computer science. The term “token” is then used in combination with the term “constraint,” where *tokens* are physical objects representing digital information, and *constraints* are physical regions bound to digital operations. Physical constraints structure the placement and combination of tokens, which systems can sense and interpret to trigger operations.

The term *token* is also used in the usual sense in the “TAC Paradigm” [71], based on the aforementioned “Token+Constraint” approach [74, 79]. However, the TAC Paradigm first names physical

objects composing a tangible user interface under the term *pyfo* to move away from the various uses of the term “object.” Pyfo is a constructive and recursive term (not atomic): pyfos are made of pyfos (e.g., the six sides of a box are also pyfos [71]). As well, pyfos are not limited to whole graspable physical objects: pyfos are also intangible physical objects (e.g., intangible constraints drawn on a surface, such as Senseboard’s displayed grid, are pyfos [51]). Thus, *tokens* are graspable pyfos representing digital information or computational functions; *constraints* are graspable or intangible pyfos that structure the behavior of tokens. Beyond physical representations, TAC Paradigm’s terminology also includes terms for digital elements, such as the term *variable*, which names digital information or computational functions. Whereas some variables can be bound to tokens, semantic variables cannot. Finally, systems behave according to a series of rules: association triplets between a token, a variable, and some constraints. These rules are referred to as TAC (i.e., Token And Constraints).

### 2.1.4 Terms in Toolkits and Design Tools.
Toolkits and design tools also require terms to assign names to physical representations and controls, as well as to differentiate their roles.

The term *phidgets* [30] stands for “physical widgets”: packaged, low-level devices and software architecture inspired by GUI toolkits to ease the building of physical user interfaces. The Phidgets toolkit provides physical primitives for controllers, sensors, and mechanical parts that are commercially available<fnref n="2" />.

ASUR++ [17, 18] is a notation for describing the physical and digital entities of mobile mixed systems. This notation extends the ASUR notation for mixed reality systems and is usable for tangible user interfaces [17]. The notation defines two terms for physical objects, by distinction of two digital roles: *tools* that support a task ($R_{\text{tool}}$) and *targeted objects* that are modified by a task ($R_{\text{object}}$).

The ROSS API [89] defines four nested levels to describe physical representations and controls. A strength of this low-level framework is that it goes beyond the description of physical objects ($R_{\text{Object}}$) by denoting active spaces ($R_{\text{Space}}$) and surfaces ($R_{\text{Surface}}$) that can sense interaction. However, even if input controls—such as buttons, switches, and sliders—are described ($R_{\text{Control}}$), physical objects’ roles—such as tools and data representations—are not describable by the API.

### 2.1.5 Existent Terms' Limitations.
The previous terms are built from two main strategies: drawing a parallel with GUI elements (e.g., “icon,” “widget,” “handle,” and “container”) or naming their physical roles (e.g., “prop,” “token,” and “constraint”). At certain points, these approaches enable the description of low embodiment (e.g., “phandles”) and higher embodiment (e.g., “phicon”). However, these approaches fail to establish a global terminology that can describe all the components residing within tangible user interfaces. For example, “tokens” can equally name artifacts representing data or functions. Moreover, tangible user interfaces often comprise representations that are not “purely tangible” (i.e., not representing data and not with high embodiment), but no terms distinguish a “pure” tangible user interface from other related physical user interfaces. Thereby, the lack of descriptive and discriminative

<footnote n="2">²Phidgets' website: https://www.phidgets.com/, last accessed 2022-06-09.</footnote>
power leads to the wide use of general terms to name any artifact
indistinctly.

Some of the terms listed above come from everyday objects (e.g.,
"tokens," "props," "handles," "faucets," "bricks," or "pucks"); some
are blended words created specifically (e.g., "phandle," "phicon," or
"phidget"); and some others associate two words (e.g., "physical
artifact," "tangible representation," or "tangible object"). Most terms
describe the artifacts but not how they relate to the digital world,
which is suggested by a metaphor of their original use transposed
to the digital world (e.g., "tokens," "props," and "faucets"). Only
a few support the idea of digital entities coming to the physical
world (such as "phidgets" and "phicons")—but they are isolated
terms—and only a few express the coupling with a digital entity
(e.g., "tangible representation" and "physical representation").

As a general term, this article considers that “physical representations” comprise tangible, graspable, and intangible representations (as pyfos that also consider intangible representations as being physical)—the difference between “tangible” and “graspable” representations residing in the degree of embodiment: high and low, respectively. High embodiment occurs when some of the digital entities’ representational characteristics are embedded in their representations (similarly to the *specialized form* artifact definition [21]) and have iconic meaning [77]. Low embodiment occurs when no digital entities’ representational characteristics are embodied by their representations (similarly to the *generic form* artifact’s definition [21]) and that have symbolic meaning [77]. This article also considers that the term “artifacts” refers to bodied objects, regardless of whether they are graspable or tangible. Finally, this article introduces a new holistic terminology that refines the descriptive level of the aforementioned physical representations by employing word-blending formation processes.

## 2.2 Blended Words

Linguistics studies the evolution of everyday language over
time, including word-formation processes (e.g., derivations,
abbreviations, and acronyms). Some new words are created by
combining *splinters* [3] excerpted from *source words*. For example,
*blended words* concatenate a basis from a first word and a suffix
from a second word, with phonemic overlap [4, 12, 31]. Linguists
observed several possibilities and strategies of splinters exemption
and combination that enter the definition of “blended words” that
keep source words recognisable [31, 32]. For instance, “heliport”
(heli(copter) + (air)port), “motel” (mot(or) + (h)otel), and “brunch”
(br(eakfast) + (l)unch) are blend words [12, 31]. However, blends
can follow some other word-formation processes. For example,
*clipping compounds* concatenate the bases of two words [4, 31, 32]
(e.g., “fintech” (fin(ancial) tech(nology)), “sysadmin” (sys(tem) +
admin(istrator)), and “hydrail” (hyd(rogen) + rail(way))). Word
blending enables creating new words of only 1, 2, or 3+ syllables
[12], thus creating some new vocabulary that designates concepts
through single words, which also facilitates communication.

Blended words already exist in Computer Science and Human-
Computer Interaction: for example, “pixel” (pic(ture)s + el(ement))
and “widget” (wi(ndow) + (ga)dget) are kinds of blended words that
have become common in the usual language and are referenced in
standard dictionaries. In the field of tangible user interfaces, several

of the aforementioned terms are blended words, such as “phicon”
(ph(ysical) + icon) [75], “phandle” (ph(ysical) + (h)andle) [75], and
“phidget” (ph(ysical) + (w)idget) [30].

This article employs word blend strategies to create new three-
syllable terms that name physical entities found in tangible and
related physical user interfaces.

## 2.3 Taxonomies and Classifications

The literature has resorted to taxonomies and classifications to
characterize the conceptual space of physical user interfaces.
Whereas the seminal work on graspable user interfaces proposed
a conceptual space with thirteen axes [22], Ishii later classified
tangible user interfaces through eight genres [40]. About two
dozen frameworks organized the conceptual space along different
numbers of axes [59, 70: Ch. 5, 81: Ch. 3]. For example, Fishkin’s
taxonomy of physicality (or tangibility) [19] organized physical
user interfaces as a twenty-item conceptual space along two
axes: embodiment (i.e., the physical relationship between the
input and the output of interaction loops) and metaphor (i.e.,
the digital meaning given to interaction loops with physical
representations). All those frameworks were organized by types:
abstracting, designing, and building; and by facets: experiences,
domains, physicality, interactions, and technologies [59].

This article proposes a new taxonomy on two axes for the
physical entities found in applicative tangible user interfaces and
related physical user interfaces, based on the *role* (i.e., *what*) and
*tangibility* (i.e., *how*) of the digital entities that are represented or
controlled. A terminology is then built by taking the values of these
two axes as source words for a blended-word formation process.

# 3 Proposing a Terminology and a Classification

This section introduces a new terminology that shifts the existing
viewpoint from “physical artifacts that are coupled to the digital
world” to a role-centered viewpoint of “data coming to the physical
world.” Before defining the terminology, the definition of physical
entities is refined, and then the concept of a tangible user interface
is recalled. Finally, an abstraction of applicative tangible user
interfaces and related physical user interfaces is proposed, along
with their classification into four tangibility classes.

## 3.1 Physical Interactional Entities

Let us first define the set of physical interactional entities involved
in tangible and related physical user interfaces, which the new
terminology aims to name. Indeed, the definitions in this article
build upon the definition of “Physical Interactional Entities,” which
are considered the highest level of abstraction of anything that is
generated, animated, or sensed by computing resources and that is
perceivable, editable, or manipulable by users. Such “Interactional
Entities” are composable, couplable, and hybridable. “Physical”
comprises matter (e.g., objects, sides, and substances), sound
(e.g., audio, percussion, and noise), and any graphical item (e.g.,
shapes, texts, and images). For example, the sides of a cube can
be considered as entities, as well as the whole cube, depending on
what is sensed, mapped, or represented. Some other entities are
added to this initial set that are not generated, nor animated, nor
sensed by computing resources, but that support, hold, or guide
entities from the initial set during interaction (e.g., surfaces, slots, and trails).

## 3.2 Back to Tangible User Interfaces' Concept

According to Ullmer and Ishii: “Tangible interfaces give physical form to digital information, employing physical artifacts both as representations and controls for computational media” [77]. This definition’s phrasing reveals Ullmer and Ishii’s datum-centered viewpoint on tangible user interfaces [70: p. 19]. The adjective “tangible” should then be used to qualify data that becomes tangible, not to qualify artifacts or bodied objects (the latter is already “tangible” in the usual sense). This article claims we should then rather talk about “tangible data,” and no more about “tangible artifacts” or “tangible objects.” Thus, tangible data are data that become tangible through artifacts (that give physical form to data): only those “pure” artifacts must be qualified as tangible representations to match Ullmer and Ishii’s definition strictly. By extending this line of reasoning, this article proposes naming all artifacts according to their digital role—as instantiations of digital information—by providing meaning about how they represent digital entities. This way, the following section defines a what—how terminology to name physical representations and controls from a role-centered viewpoint.

## 3.3 A Role-Centered Terminology

The new terminology uses blended words [4, 12] formed from a taxonomy on “what” digital entities come into the digital world and “how”. The first part concerns what digital entities are represented or controlled by physical representations:

(1) *Data* refers to datum models and data states stored in computer memory and computed by the processor.

(2) *Tools* are instruments that can edit the data represented in the physical world (regardless of whether data representation is tangible, graspable, or intangible).

(3) *Operations* involve triggering an event or action in the computing system (e.g., core operations [80] include loading or saving data, loading an app, and validation).

(4) Finally, *constraints* structure the behavior of data representations, thus structuring mapped digital information. They can give meaning to artifacts by triggering actions or operations (e.g., detecting a block’s presence to trigger an action [76, 78]). Broadly, “constraints” affect physical objects’ positioning (e.g., table surface) before being sensed by the computer. “Constraints” are not always computer-generated: for example, some circles where to place artifacts can be video-projected onto a surface or drawn with paint.

The second part concerns *how* digital entities come to the physical world:

(1) Digital entities made “tangible” are represented by iconic artifacts of specialized form—with a high embodiment (of either or both representation and control).

(2) Those that are made “graspable” are represented by symbolic artifacts of a generic form, with a low level of embodiment. Saying tangible or graspable is not saying better or worse, but simply different gradations of embodiment [19] that

correspond to different design and technology answers to different needs.

(3) As well, digital entities made “intangible” are represented by volatile computer-generated physical phenomena that cannot be grasped and disappear once the computer is turned off (e.g., display, video projection, or audio).

From the seven source words mentioned above, the seven following splinters [3] are excerpted: four bases from *what* words: “dat-,” “tol-,” “op-,” and “const-”; and three suffixes from *how* words: “-ible,” “-able,” and “-nible.” Several attempts were required to select those splinters by a compromise between pronunciation, syllable count, and recognition of original words in the final terms (at least for experts aware of the terminology’s construction). Blending those bases and suffixes thus provides twelve terms embedding meaning about *what* and *how* digital entities come to the physical world (see Table 1).

The first term, “datable”—“datum is tangible”—better matches the meaning of the definition of “purely tangible” user interfaces, in contrast to previous terminologies. Moreover, none of the twelve terms confuses words used for everyday objects. Finally, they provide a subtle and global descriptive power by naming four roles with three degrees in a unique namespace. The following section suggests relying on the holisticness of the terminology to characterize applicative tangible user interfaces and related physical user interfaces through vertices.

## 3.4 An Abstraction Through Hallmarks

We propose that applications could be abstracted by counting occurrences of physical representations and controls according to the twelve terms discriminated by the holistic terminology. Such a characterization would then discriminate physical user interfaces at the application level, independently of their morphological and technological descriptions (e.g., “input is distant from output” [19], “this interface’s genre is tabletop” [40], or “input is sensed in 3D space by 2×6DOF props”). Therefore, this characterization could discriminate, for example, between two tabletop applications. Furthermore, this characterization could enable the comparison of applications within the same space, regardless of whether they are tabletop or ambient interfaces, or whether they have input that is distant from or nearby the output.

<table><caption>Table 1: The twelve terms refining physical representations.</caption><thead><tr><th rowspan="2">What</th><th colspan="3">How</th></tr><tr><th>Tangible</th><th>Graspable</th><th>Intangible</th></tr></thead><tbody><tr><td>Datum</td><td>DATIBLE<br/>['dertibol]<br/>'Datum is tangible'</td><td>DATABLE<br/>['dertobol]<br/>'Datum is graspable'</td><td>DATNIBLE<br/>['dert.nibol]<br/>'Datum is intangible'</td></tr><tr><td>Tool</td><td>TOLIBLE<br/>[tulibol]<br/>'Tool is tangible'</td><td>TOLABLE<br/>[tulobol]<br/>'Tool is graspable'</td><td>TOLNIBLE<br/>[tul.nibol]<br/>'Tool is intangible'</td></tr><tr><td>Operation</td><td>OPIBLE<br/>[opibol]<br/>'Operation is tangible'</td><td>OPABLE<br/>[opobol]<br/>'Operation is graspable'</td><td>OPNIBLE<br/>[op.nibol]<br/>'Operation is intangible'</td></tr><tr><td>Constraint</td><td>CONSTIBLE<br/>['konsibol]<br/>'Constraint is tangible'</td><td>CONSTABLE<br/>['konsobol]<br/>'Constraint is graspable'</td><td>CONSTNIBLE<br/>['kons.nibol]<br/>'Constraint is intangible'</td></tr></tbody></table>
To this end, we define the two following vertices:

* Let the *hallmark* of an application be the vertex of entities' count for each term, where terms are ordered first by “what” (i.e., Datum, Tangible, Operation, then Constraint), and second by “how” (i.e., Tangible, Graspable, then Intangible). The resulting hallmark is the following vertex: `( #datable, #datable, #datnible, #tolible, #tolable, #tolnible, #opible, #opable, #opnible, #constible, #constable, #constnible )`.

* Let the *binary hallmark* of an application be the vertex built from the hallmark where components only indicate entities' presence for each term (i.e., components' values greater than 1 are cut to 1).

Some distances could be calculated between these vertices. However, this article primarily proposes using hallmarks to cluster applicative tangible user interfaces into distinct classes.

## 3.5 Four Role-Centered Tangibility Classes

We propose to cluster applicative tangible user interfaces according to how roles are represented, with a particular focus on the role of data, whose representation is determinant to the definition of tangible user interfaces. The four resulting tangibility classes are the following:

* In *Class I* specimens, data are represented only by tangible or graspable entities (i.e., presence of either datibles or datables, but absence of datnibles).

* In *Class II* specimens, data are represented by intangible entities, as well as tangible or graspable entities (i.e., presence of datnibles with either datibles or datables).

* In *Class III* specimens, data are represented only by intangible entities, which are coupled with the use of tangible or graspable tools (i.e., presence of datnibles combined with tolivles or totables, but absence of datibles and datables).

* In *Class IV* specimens, only operations are provided, through tangible or graspable entities (i.e., presence of opibles or opables, and data and tools are unknown or absent). This class provides specimens that are not restricted to a specific application and can thus control operations shared across applications (i.e., core operations [80]).

The application hallmarks' patterns that correspond to these tangibility classes are reported in Table 2. The following section illustrates the descriptive power of the twelve terms through examples from the literature.

## 4 Illustrating on a Corpus of Applications

This section illustrates the usage of the new terminology by describing and classifying a corpus of applications from the literature.

### 4.1 Specimens' Selection

Because researchers have produced hundreds of tangible user interface specimens over the past few decades [24], the present collection is not meant to be exhaustive, but a rather representative sample. The specimen selection process is neither as rigorous as that in systematic literature reviews nor intended to provide reliable statistics. Instead, the objective is to illustrate the

<table><thead><tr><th rowspan="3">Class</th><th colspan="8">Hallmark pattern</th></tr><tr><th colspan="2">What:</th><th colspan="2">T</th><th colspan="2">O</th><th colspan="2">C</th></tr><tr><th>How:</th><th>T</th><th>G</th><th>I</th><th>T</th><th>G</th><th>I</th><th>T</th><th>G</th><th>I</th></tr></thead><tbody><tr><td>I</td><td colspan="2">(+, 0, 0, *, *, *, *, *, *, *, *)</td><td colspan="2">(0, +, 0, *, *, *, *, *, *, *)</td><td colspan="2">(+, 0, +, *, *, *, *, *, *, *)</td><td colspan="2">(0, +, +, *, *, *, *, *, *)</td><td colspan="2">(0, 0, +, 0, +, *, *, *, *, *)</td></tr><tr><td>II</td><td colspan="2">(+, 0, +, *, *, *, *, *, *, *, *)</td><td colspan="2">(0, +, +, *, *, *, *, *, *)</td><td colspan="2">(+, 0, +, *, *, *, *, *, *)</td><td colspan="2">(0, 0, +, +, 0, *, *, *, *, *)</td><td colspan="2">(0, 0, +, +, 0, +, *, *, *)</td></tr><tr><td>III</td><td colspan="2">(0, 0, +, 0, +, *, *, *, *, *, *)</td><td colspan="2">(0, 0, +, +, 0, *, *, *, *, *)</td><td colspan="2">(0, 0, +, +, 0, 0, 0, +, *, *, *)</td><td colspan="2">(0, 0, +, +, 0, 0, 0, 0, +, *, *)</td><td colspan="2">(0, 0, +, +, 0, 0, 0, 0, 0, +, *)</td></tr><tr><td>IV</td><td colspan="2">(0, 0, 0, 0, 0, 0, 0, 0, 0, +, *, *)</td><td colspan="2">(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +, *)</td><td colspan="2">(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +, *)</td><td colspan="2">(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +, *)</td><td colspan="2">(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +, *)</td></tr></tbody></table>

Note. What: 'D' = Data, 'T' = Tool, 'O' = Operation, 'C' = Constraint.

How: 'T' = Tangible, 'G' = Graspable, 'I' = Intangible.

'a.' = some representations (one or more). 'b.' = none or some representations.

<caption>Table 2: Representations in the four tangibility classes.</caption>

terminology coverage and usefulness by describing a variety of emblematic tangible user interface specimens and related physical user interfaces. Because research interests vary over the years, restricting the analysis to a specific date range could have led to the exclusion of some tangible user interface kinds; therefore, the entire research period, from the premise to the more recent work, was taken into account. The selection thus gathered various genres<fnref n="3" /> of tangible user interfaces that were studied during this entire period. Finally, the selection was filtered to retain work that showed real applications. Work merely demonstrating a new technology or a new kind of tangible user interface was not retained, as its naming could vary across applications. Indeed, knowing the mapping to the digital world is necessary because being physical is not enough to be a tangible representation: the same representation can be tangible in some applications but not in others.

The selection process is summarized in Figure 1. Several sources were browsed. First, a previous state of the art [70] and some specimens already known to the author (e.g., some specimens that were experienced during demonstration sessions at conferences over time). Second, the projects page<fnref n="4" /> of the MIT Tangible Media Group, which is certainly the most prolific laboratory in this field (accounts for 45.5% of the final collection's provenance). Third, to obtain even more recent work, sessions with 'tangible' in their title from the last ten CHI conference proceedings (from 2013 to 2022 venues). Finally, the last ten TEI conference proceedings from 2013 to 2022 were also browsed. Specimens regarding remote computer-supported conditions were rejected.

Once many examples were added to the collection, the terms' coverage was computed to check whether the whole terminology is relevant. Next, more examples were selected and analyzed until the full coverage of the whole terminology's terms could be reached. Table 3 shows the number of physical representations gathered across the twelve terms.

<footnote n="3">³Seven genres of tangible user interface applications were described in 2008 [40]: (1) tangible telepresence, (2) tangibles with kinetic memory, (3) constructive assembly, (4) tokens and constraints, (5) interactive surfaces—tabletop TUI, (6) continuous plastic TUI, (7) augmented everyday objects, and (8) ambient media, to which actuated TUI and shape-changing TUI must now be added.</footnote>

<footnote n="4">⁴MIT Tangible Media Group's projects page: https://tangible.media.mit.edu/projects/, last accessed 2022-08-03.</footnote>
<table><thead><tr><th colspan="4">Application</th><th colspan="10">Physical Representation</th></tr><tr><th rowspan="2">#</th><th rowspan="2">Name</th><th rowspan="2">Year</th><th rowspan="2">References</th><th rowspan="2">Name or Description</th><th colspan="4">What</th><th colspan="3">How</th><th rowspan="2">WHT Name</th></tr><tr><th>D</th><th>T</th><th>O</th><th>C</th><th>T</th><th>G</th><th>I</th></tr></thead><tbody><tr><td>1.</td><td>Slot Machine</td><td>1976</td><td>[63] and see in [81: Ch. 1]</td><td>Rows (procedure)</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Buttons (on each row)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Plastic cards (commands)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Display triangle (turtle)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Circular robot (turtle)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td>2.</td><td>CAAD 3D Modelling System</td><td>1979</td><td>[1, 2]</td><td>Building Blocks</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Design geometry (perspective view)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Evaluation measures (Isoplots)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td>3.</td><td>Self-Builder Model (Segal Model)</td><td>1980</td><td>[26, 27] cited from [73]</td><td>Panels</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Board</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Wireframe rendering</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Feedback tool (house area, cost)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td>4.</td><td>Marble Answering Machine</td><td>1992</td><td>See in [25, 48] and [81: Ch. 1]</td><td>Machine</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Machine's indentation (play slot)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Marble (message)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Message</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td>5.</td><td>Head Prop</td><td>1994</td><td>[35]</td><td>Doll's head (brain)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Plate (slicing)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Plate thumb button (clutch)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Foot pedal (clutch)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Brain 2D slice</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td>6.</td><td>GraspDraw</td><td>1995</td><td>[22]</td><td>Bricks</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>2D shapes</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Inkwell</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Functions' tray (select, delete, shapes)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>ActiveDesk surface</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>7.</td><td>Tangible Geospace</td><td>1997</td><td>[48]</td><td>Phicons</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Campus Map</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Overlay view (passive lens)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>3D view (active lens)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Scaling and Rotating Device</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Desk's surface</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>8.</td><td>Build-IT</td><td>1997</td><td>[23, 65, 66]</td><td>Bricks</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Plan view</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Objects (robots, tables...)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Virtual cameras</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>EyeCatchers</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Table surface</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>3D view</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td>9.</td><td>Pinwheels</td><td>1998</td><td>[88]</td><td>Pinwheels</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td>10.</td><td>Voodoo Dolls</td><td>1998</td><td>[50]</td><td>Spongy object</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Graphics model</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td>11.</td><td>mediaBlocks</td><td>1998</td><td>[76, 78]</td><td>mediaBlock</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Slots</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>12.</td><td>musicBottles</td><td>1999</td><td>[42, 45]</td><td>Music</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Bottle (music)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Cork</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Triangular table</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Central "stage" area</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTNIBLE</td></tr><tr><td>13.</td><td>Urp (Urban Planning Workbench)</td><td>1999</td><td>[5, 41, 82]</td><td>Architectural Model</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Road-object (strips)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Material-transformation-object (wand)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Video-camera-object</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Clock-object</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Wind-generating tool</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Anemometer-object (arrow)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Wind magnitude (number)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Distance-measuring-object</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Distance (number)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Shadows/Relections</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Airflow grid</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Workbench</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>14.</td><td>Senseboard</td><td>2001</td><td>[51]</td><td>Grid</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Rectangular pucks</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>View detail puck</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Arrow puck</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Values</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATNIBLE</td></tr><tr><td></td><td></td><td></td><td></td><td>Board</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr></tbody></table>

Note. "-" = empty. What: 'D' = Data, 'T' = Tool, 'O' = Operation, 'C' = Constraint. How: 'T' = Tangible, 'G' = Graspable, 'I' = Intangible. 'WHT Name' = What-How Terminology name.

<caption>Table 4: Physical representations named for 14 applications from the literature, ordered by ascending years, from 1976 to 2001.</caption>
<table><thead><tr><th colspan="3">Application</th><th colspan="10">Physical Representation</th></tr><tr><th rowspan="2">#</th><th rowspan="2">Name</th><th rowspan="2">Year</th><th rowspan="2">References</th><th rowspan="2">Name or Description</th><th colspan="4">What</th><th colspan="3">How</th><th rowspan="2">WHT Name</th></tr><tr><th>D</th><th>T</th><th>O</th><th>C</th><th>T</th><th>G</th><th>I</th></tr></thead><tbody><tr><td rowspan="5">15.</td><td rowspan="5">Illuminating Clay</td><td rowspan="5">2002</td><td rowspan="5">[46, 64]</td><td>Clay Model</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>DATIBLE</td></tr><tr><td>Rotative platform</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>Crosshairs</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Cross Sections</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Analysis Function Thumbnails</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td rowspan="5">16.</td><td rowspan="5">AudioPad</td><td rowspan="5">2002</td><td rowspan="5">[61, 62]</td><td>3-D perspective view</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Sounds</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Pucks (audio tracks)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td>Star-shape puck (sound selector)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Hierarchical menu</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPNIBLE</td></tr><tr><td rowspan="7">17.</td><td rowspan="7">ReacTable</td><td rowspan="7">2003</td><td rowspan="7">[52]</td><td>SenseTable surface</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>CONSTABLE</td></tr><tr><td>Music</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Point (audio output)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Square puck (audio source)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td>Rounded square puck (filter)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td>Round puck (controller)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td>Decagon puck (control filter)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>TOLABLE</td></tr><tr><td rowspan="5">18.</td><td rowspan="5">IP Network Design Workbench</td><td rowspan="5">2003</td><td rowspan="5">[53]</td><td>Pentagon puck (audio mixer)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Lines (audio flow)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Table surface</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>CONSTABLE</td></tr><tr><td>Pucks</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Nodes</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td rowspan="10">19.</td><td rowspan="10">Query Shapes</td><td rowspan="10">2004</td><td rowspan="10">[38]</td><td>Links</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Nodes menu</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPNIBLE</td></tr><tr><td>Parameter puck (with button)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Link bandwidth</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Router service priority</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Number of client users</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Server performance</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Table surface</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>CONSTABLE</td></tr><tr><td>Measurement graphs</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPNIBLE</td></tr><tr><td>Cubes (ActiveCubes)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATABLE</td></tr><tr><td>Voxel data representation</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>3D shape models</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>20.</td><td>TUISTER</td><td>2004</td><td>[9]</td><td>TUISTER</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>OPIBLE</td></tr><tr><td rowspan="3">21.</td><td rowspan="3">I/O Brush</td><td rowspan="3">2004</td><td rowspan="3">[68, 69]</td><td>Brush</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>TOLIBLE</td></tr><tr><td>World (palette)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>-</td><td>OPIBLE</td></tr><tr><td>Canvas</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td rowspan="10">22.</td><td rowspan="10">PICO</td><td rowspan="10">2005</td><td rowspan="10">[60]</td><td>Pucks (antenas)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td>Map (city)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Flexible curve</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTIBLE</td></tr><tr><td>Rubber band</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTIBLE</td></tr><tr><td>Collar</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTIBLE</td></tr><tr><td>Teflon / Sandpaper</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>Table surface</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>CONSTABLE</td></tr><tr><td>Props</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATABLE</td></tr><tr><td>3D fragments</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Foot pedals (clutch)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPABLE</td></tr><tr><td rowspan="10">23.</td><td rowspan="10">ArcheoTUI</td><td rowspan="10">2007</td><td rowspan="10">[67]</td><td>Map (cube top view)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Cutting line</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Two-puck prop</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Ruler prop</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Button box</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPABLE</td></tr><tr><td>Tabletop</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>CONSTABLE</td></tr><tr><td>Eyedropper</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPIBLE</td></tr><tr><td>24.</td><td>GeoTUI</td><td>2008</td><td>[14]</td><td>2.5D shape display (terrain)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>DATIBLE</td></tr><tr><td>Map (cube top view)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Cutting line</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Two-puck prop</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Ruler prop</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Button box</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPABLE</td></tr><tr><td>Tabletop</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>CONSTABLE</td></tr><tr><td>25.</td><td>Slurp</td><td>2008</td><td>[90]</td><td>Eyedropper</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>OPIBLE</td></tr><tr><td rowspan="2">26.</td><td rowspan="2">Relief</td><td rowspan="2">2010</td><td rowspan="2">[56, 57]</td><td>Topographical map</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATIBLE</td></tr><tr><td>2.5D shape display (terrain)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>DATNIBLE</td></tr><tr><td rowspan="10">27.</td><td rowspan="10">Teegi</td><td rowspan="10">2014</td><td rowspan="10">[28]</td><td>Teegi character (user's activity)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATIBLE</td></tr><tr><td>Brain model (user's activity)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATIBLE</td></tr><tr><td>Filter area</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Mini-Teegis (filters)</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLIBLE</td></tr><tr><td>EEG Raw Signals</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr><tr><td>Color map amplitude</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLNIBLE</td></tr><tr><td>Color map cursor</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>TOLABLE</td></tr><tr><td>Table surface</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>X</td><td>CONSTABLE</td></tr><tr><td>Trigger pins</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>OPABLE</td></tr><tr><td>Soundwave pins</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATIBLE</td></tr><tr><td>Sound</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>DATNIBLE</td></tr></tbody></table>

Note. "-" = empty. What: 'D' = Data, 'T' = Tool, 'O' = Operation, 'C' = Constraint. How: 'T' = Tangible, 'G' = Graspable, 'I' = Intangible. 'WHT Name' = What-How Terminology name.

<caption>Table 5: Physical representations named for 14 applications from the literature, ordered by ascending years, from 2002 to 2016.</caption>
<table><thead><tr><th colspan="4">Application</th><th colspan="10">Physical Representation</th></tr><tr><th rowspan="2">#</th><th rowspan="2">Name</th><th rowspan="2">Year</th><th rowspan="2">References</th><th rowspan="2">Name or Description</th><th colspan="4">What</th><th colspan="3">How</th><th rowspan="2">WHT Name</th></tr><tr><th>D</th><th>T</th><th>O</th><th>C</th><th>T</th><th>G</th><th>I</th></tr></thead><tbody><tr><td>29.</td><td>CairnFORM</td><td>2019</td><td>[16]</td><td>Ring chart</td><td>X</td><td>-</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>DATABLE</td></tr><tr><td>30.</td><td>reSpire</td><td>2019</td><td>[10]</td><td>Shape-changing fabric</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>DATIBLE</td></tr><tr><td>31.</td><td>Embodied Axes</td><td>2020</td><td>[13]</td><td>Orthogonal arms (data axes)</td><td>-</td><td>-</td><td>X</td><td>-</td><td>X</td><td>-</td><td>-</td><td>OPIBLE</td></tr><tr><td>32.</td><td>CoDa</td><td>2020</td><td>[83]</td><td>Tokens (data points)<br>Interactive surface<br>Sidebar buttons (filter and analytic functions)<br>Data points<br>Analytical functions (lines)</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>DATIBLE<br>CONSTABLE<br>TOLABLE<br>DATNIBLE<br>DATNIBLE</td></tr><tr><td>33.</td><td>SABLIER</td><td>2022</td><td>[58]</td><td>Hourglass</td><td>X</td><td>-</td><td>-</td><td>-</td><td>X</td><td>-</td><td>-</td><td>OPIBLE</td></tr></tbody></table>

Note. ".-" = empty. What: 'D' = Data, 'T' = Tool, 'O' = Operation, 'C' = Constraint. How: 'T' = Tangible, 'G' = Graspable, 'I' = Intangible. 'WHT Name' = What-How Terminology name.

<table><caption>Table 6: Physical representations named for 5 applications from the literature, ordered by ascending years, from 2019 to 2022.</caption><thead><tr><th>#</th><th>Application</th><th>Hallmark</th><th>Class</th></tr></thead><tbody><tr><td>1.</td><td>Slot Machine</td><td>(0, 1, 1, 0, 0, 0, 0, 2, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>2.</td><td>CAAD 3D Modelling System</td><td>(0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>II</td></tr><tr><td>3.</td><td>Self-Builder Model (Segal Model)</td><td>(1, 0, 2, 0, 0, 0, 0, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>4.</td><td>Marble Answering Machine</td><td>(1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>5.</td><td>Head Prop</td><td>(1, 0, 1, 1, 0, 0, 0, 2, 0, 0, 0, 0)</td><td>II</td></tr><tr><td>6.</td><td>GraspDraw</td><td>(0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0)</td><td>III</td></tr><tr><td>7.</td><td>Tangible Geospace</td><td>(1, 0, 2, 0, 1, 0, 1, 0, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>8.</td><td>Build-IT</td><td>(0, 0, 3, 0, 1, 0, 0, 0, 2, 0, 1, 0)</td><td>III</td></tr><tr><td>9.</td><td>Pinwheels</td><td>(N, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>I</td></tr><tr><td>10.</td><td>Voodoo Dolls</td><td>(0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>II</td></tr><tr><td>11.</td><td>mediaBlocks</td><td>(0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0)</td><td>IV</td></tr><tr><td>12.</td><td>musicBottles</td><td>(0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1)</td><td>II</td></tr><tr><td>13.</td><td>Urp (Urban Planning Workbench)</td><td>(2, 0, 2, 2, 2, 0, 0, 2, 2, 0, 1, 0)</td><td>II</td></tr><tr><td>14.</td><td>Senseboard</td><td>(0, 1, 1, 0, 2, 0, 0, 0, 0, 0, 1, 1)</td><td>II</td></tr><tr><td>15.</td><td>Illuminating Clay</td><td>(1, 0, 3, 0, 0, 1, 0, 0, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>16.</td><td>AudioPad</td><td>(0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0)</td><td>II</td></tr><tr><td>17.</td><td>ReacTable</td><td>(0, 1, 3, 0, 4, 0, 0, 0, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>18.</td><td>IP Network Design Workbench</td><td>(0, 0, 2, 0, 2, 4, 0, 0, 2, 0, 1, 0)</td><td>III</td></tr><tr><td>19.</td><td>Query Shapes</td><td>(0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>II</td></tr><tr><td>20.</td><td>TUISTER</td><td>(0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0)</td><td>IV</td></tr><tr><td>21.</td><td>I/O Brush</td><td>(0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0)</td><td>III</td></tr><tr><td>22.</td><td>PICO</td><td>(0, 1, 1, 0, 0, 0, 0, 0, 3, 2, 0)</td><td>II</td></tr><tr><td>24.</td><td>GeoTUI</td><td>(0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0)</td><td>III</td></tr><tr><td>25.</td><td>Slurp</td><td>(0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0)</td><td>IV</td></tr><tr><td>26.</td><td>Relief</td><td>(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>I</td></tr><tr><td>27.</td><td>Teegi</td><td>(2, 0, 1, 1, 2, 0, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>28.</td><td>SoundFORMS</td><td>(1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0)</td><td>II</td></tr><tr><td>29.</td><td>CairnFORM</td><td>(0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>I</td></tr><tr><td>30.</td><td>reSpire</td><td>(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)</td><td>I</td></tr><tr><td>31.</td><td>Embodied Axes</td><td>(0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0)</td><td>IV</td></tr><tr><td>32.</td><td>CoDa</td><td>(1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 1, 0)</td><td>II</td></tr><tr><td>33.</td><td>SABLIER</td><td>(0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0)</td><td>IV</td></tr></tbody></table>

<caption>Table 7: Hallmarks and classes of the 33 applications.</caption>

Finally, the actuated interface SoundFORMS (#28) assigns digital roles to an array of actuated pins: pressing some opables ("trigger pins") starts playing datnibles ("sounds") that are bound to datibles ("soundwave pins").

In Class III applications, the tangible or graspable part of the user interface is not about data, but rather tools or operations. For example, the I/O Brush (#21) uses World as an opible ("palette"), to

draw with a toblite ("brush") on a datnible ("canvas"). The tabletop GraspDraw (#6) edits datnibles ("shapes") using a toblite ("pucks") that serve as physical anchors. Finally, the tabletop GeoTUI (#24) controls a toblite ("cutting line") with a toblite ("ruler") or a toblite ("two-puck prop") on a datnible ("top view map").

Artifacts of Class IV are designed to integrate across various applications, whose scopes can extend beyond tangible user interfaces (e.g., graphical user interfaces [90], augmented reality [13], and virtual reality caves [58]). For example, Slurp (#25) is an "eyedropper" that can extract and inject digital information. TUISTER (#20), Embodied Axes (#31), and SABLIER (#33) provide artifacts to browse hierarchical structures, control 3D positions, and navigate through space and time, respectively.

## 5 Mapping the Previous Terms

Figure 3 illustrates a mapping between the previous terms found in the literature and the new holistic terminology, achieved through a projection onto the two axes of the taxonomy that originated the terminology (i.e., "what" is physically represented and "how"). Only one term can cover the whole terminology. Except for this holistic term, the mapping reveals a focus of previous research on representing tangible data, which is describable through half of the terms. In contrast, constraints and intangible data are describable by only one term each, and intangible tools and operations are outside the conceptual perception.

The distorted focus on naming tangible data may have its origins in the community's search for paradigms and frameworks, as well as the influence of the data-centered viewpoint in the definition of tangible user interfaces. However, other roles than data are required to describe applicative tangible user interfaces.

The following section proposes to match the four tangibility classes with some previous classifications through the set of (orthogonal) applications provided by the corpus.

## 6 Cross-Matching Classifications

Some previous classifications have discriminated between tangible user interfaces according to their morphology (i.e., how they are physically shaped, such as surfaces [40], everyday objects [40], or discrete tangibles [44]) and behavior (i.e., how users interact with them and how they interact with users, such as assembly [40], deformation [44], or transformation [44]). Figure 4 bridges
these previous classifications and the tangibility classes through
the collection of applicative tangible user interfaces.

Matching previous classifications, the applications' tangibility
classes are split into five clusters. The present collection of
applications already shows that previous classifications and
tangibility classes have no direct correspondence. Whereas
"Artifacts & Objects" gathers applications coming from the four
classes, "Ambient Media" and "Constructive Assemblies" count
only one class each. Moreover, this initial matching may evolve
when more applications are included. Indeed, even if some genres
fit implicitly into certain classes well, based on their definition (e.g.,
interactive surfaces, such as tabletops, falign well with Class II and
Class III), some classes may appear in certain clusters. For example,
constructive assemblies could also be stackable audio or video filters
that are tools controlling intangible data (i.e., describing such an
application specimen would then require bridging Class III with
"Constructive Assemblies"). Furthermore, other clusters may appear
because certain genres of tangible user interfaces are absent from
the collection of applications (e.g., "Tangible Telepresence" [40] and
"Tangibles with Kinetic Memory" [40]). However, we believe that
the descriptive power of the four classes may provide a sufficient
design space to integrate any new applicative tangible user interface
and related physical user interfaces.

7 Limitations and Future Work

A limitation of this work is that analyses were not consolidated with
the specimens’ designers. Describing the 33 applications was mostly
straightforward, but sometimes required advanced questioning.
An example is when describing the Pinwheels specimen (#9),
which represents information flow in an ambient manner. At
first glance, Pinwheels must be datnible: airflow represents the
datum flow. However, this idea evolved from using airflow in the
ambientROOM [49]—merely needing air coming into the room
through a hole—to using visual “spinning pinwheels” [15, 47, 88].
Hence, Pinwheels are described as datibles<fnref n="7" />. Another example lies
in the vertical measurement graphs (i.e., usage rate, response time,
and running cost graphs) placed next to the IP Network Design
Workbench (#18). Some could have considered these graphs as
auxiliary GUIs. However, they were considered as opnibles (i.e.,
intangible measurement operations), as well as measurements in the
Urp (#13) that were considered as operations. Advanced questioning
can become tricky when design thoughts are not clearly detailed
in the articles. The community must benefit from future work
that describes previous physical user interface specimens in a
publicly shared database (e.g., [24]) and future articles that describe
specimens using holistic terminologies.

This work focused on the roles found in tangible and related
physical user interfaces without considering remote collaboration
conditions [7]. However, in such conditions, remote users may
be represented by the user interface through streams [39, 43, 55],
shapes [6, 8], or avatars [54] and even interact with physical entities
[6, 55]. Therefore, these conditions raise the question of defining
a role for remote users specifically (or considering them merely
as data or tools?), refining their body parts (e.g., head, forearms,

and hands), and defining terms that describe how they appear in
the user interface. These questions may extend the terminology to
interactions with robots and cobots in Human-Robot Interaction
[72] and Extended Reality conditions [29].

The description of bodied entities is refined on only two levels
of embodiment (i.e., tangible and graspable), which roughly
simplifies the twenty values of Fishkin' taxonomy [19]. However,
this article considered a level of embodiment for bodiless entities
(i.e., intangible), even if intangible entities can also encounter
various levels of representativeness (e.g., a beep or slang audio
feedback when sending a file to trash, versus paper-creasing noise).

This work separated hallmarks into classes. However, defining a
distance calculation between hallmarks could serve the needs of
automatic classification algorithms (e.g., cladistics [24]) that would
help sort, organize, and browse the vast collection of specimens
provided by the field of tangible user interfaces over the past three
decades.

Finally, this article introduces the concept of interactional entities
briefly, which future work will need to define more extensively.

8 Conclusion

This article introduces a “what–how” terminology (WHT) that
provides twelve terms to name physical representations and
controls: by taking a role-centered viewpoint (i.e., the digital role),
the resulting terms self-contain meaning about how digital entities
come into the physical world. Those terms are blended words,
whose logical structure from known words must make them easy
to learn, recollect, and understand. Mainly, this new terminology
outperforms the previous terms found in the literature for three
reasons. First, it defines all terms within a single namespace, free
from specific paradigms, thus allowing for cross-genre descriptions.
Second, it refines the descriptive level of physical user interface
components. Third, it better aligns with definitions, thereby
enhancing communication and understanding. This terminology
introduces the term “datable,” which matches the definition of
pure tangible user interfaces, along with providing eleven terms
that distinguish other representations. This terminology must
benefit at least three scopes: first, supporting ideation by providing
vocabulary that describes possibilities; second, offering a holistic
conceptual space for toolkit architectures; and finally, offering a
taxonomy usable for the exploration and classification of physical
user interfaces.

Acknowledgments

The author thanks the reviewers of the ACM CHI 2023 conference for their valuable comments, which helped revise and improve certain sections of this article, as well as extend its contributions.

References

[1] Robert Aish. 1979. 3D input for CAAD systems. *Computer-Aided Design* 11, 2 (March 1979), 66-70. https://doi.org/10.1016/0010-4485(79)90098-8

[2] Robert Aish and Peter Noakes. 1984. Architecture without numbers – CAAD based on a 3D modelling system. *Computer-Aided Design* 16, 6 (Nov. 1984), 321–328. https://doi.org/10.1016/0010-4485(84)90116-7

[3] Alejandro Barrena Jurado. 2019. A study on the 'wordgasm': the nature of blends' splinters. *Journal in English Lexicology* 14 (Dec. 2019), 49 pages. Issue on "Blending in English". https://doi.org/10.4000/lexis.3916

[4] Natalia Beliaeva. 2016. Blends at the intersection of addition and subtraction: Evidence from processing. *SKASE Journal of Theoretical Linguistics* 13, 2 (Sept. 2016). https://doi.org/10.4000/lexis.3916

<footnote n="7">⁷Treating single datnibles (airflow) as physical user interfaces would lead to reconsidering Class I definition or to creating a new class.</footnote>
2016), 23-45. Issue on selected papers from the "Word-Formation Theories Conference," Košice, Slovakia, June 26-28, 2015.

[5] Eran Ben-Joseph, Hiroshi Ishii, John Underkoffler, Ben Piper, and Luke Yeung. 2001. Urban Simulation and the Luminous Planning Table: Bridging the Gap between the Digital and the Tangible. *Journal of Planning Education and Research* 21, 2 (Dec. 2001), 196-203. https://doi.org/10.1177/0739456X0102100207

[6] Scott Brave and Andrew Dahley. 1997. inTouch: a medium for haptic interpersonal communication. In *CHI '97 Extended Abstracts on Human Factors in Computing Systems* (Atlanta, Georgia, USA, March 22-27, 1997) (*CHI EA '97*). Association for Computing Machinery, New York, NY, USA, 363-364. https://doi.org/10.1145/1120211.1120435

[7] Scott Brave, Hiroshi Ishii, and Andrew Dahley. 1998. Tangible interfaces for remote collaboration and communication. In *Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work* (Seattle, Washington, USA, November 14-18, 1998) (*CSCW '98*). Association for Computing Machinery, New York, NY, USA, 169-178. https://doi.org/10.1145/289444.289491

[8] Johanna Brewer, Amanda Williams, and Paul Dourish. 2007. A handle on what's going on: combining tangible interfaces and ambient displays for collaborative groups. In *Proceedings of the 1st International Conference on Tangible and Embedded Interaction* (Baton Rouge, Louisiana, February 15-17, 2007) (*TEI '07*). Association for Computing Machinery, New York, NY, USA, 3-10. https://doi.org/10.1145/1226969.1226971

[9] Andreas Butz, Markus Groß, and Antonio Krüger. 2004. TUISTER: A Tangible UI for Hierarchical Structures. In *Proceedings of the 9th International Conference on Intelligent User Interfaces* (Funchal, Madeira, Portugal, January 13-16, 2004) (*UII '04*). Association for Computing Machinery, New York, NY, USA, 223-225. https://doi.org/10.1145/964442.964486

[10] Kyung Yun Choi, Valentina Sumini, and Hiroshi Ishii. 2019. reSpire: Self-Awareness and Interpersonal Connectedness through Shape-Changing Fabric Display. In *Proceedings of the 2019 on Conference of Creativity and Cognition* (San Diego, California, USA, June 23-26, 2019) (*C&C '19*). Association for Computing Machinery, New York, NY, USA, 449-454. https://doi.org/10.1145/3325480.3329176

[11] Aubrey Colter, Patlapa Davivonga, Donald Derek Haddad, Halla Moore, Brian Tice, and Hiroshi Ishii. 2016. SoundFORMS: Manipulating Sound Through Touch. In *Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems* (San Jose, California, USA, May 7-12, 2016) (*CHI EA '16*). Association for Computing Machinery, New York, NY, USA, 2425-2430. https://doi.org/10.1145/2851581.2892414

[12] Patrick Connolly. 2013. The innovation and adoption of English lexical blends. *JournalLIPP* 2 (2013), 1-14. https://doi.org/10.5282/journalipp/68

[13] Maxime Cordeil, Benjamin Bach, Andrew Cunningham, Bastian Montoya, Ross T. Smith, Bruce H. Thomas, and Tim Dwyer. 2020. Embodied Axes: Tangible, Actuated Interaction for 3D Augmented Reality Data Spaces. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems* (Honolulu, Hawaii, USA, April 25-30, 2020) (*CHI '20*). Association for Computing Machinery, New York, NY, USA, Article 486, 12 pages. https://doi.org/10.1145/3313831.3376613

[14] Nadine Couture, Guillaume Rivière, and Patrick Reuter. 2008. GeoTUI: A Tangible User Interface for Geoscience. In *Proceedings of the 2nd International Conference on Tangible and Embedded Interaction* (Bonn, Germany, February 18-20, 2008) (*TEI '08*). Association for Computing Machinery, New York, NY, USA, 89-96. https://doi.org/10.1145/1347390.1347411

[15] Andrew Dahley, Craig Wisneski, and Hiroshi Ishii. 1998. Water Lamp and Pinwheels: Ambient Projection of Digital Information into Architectural Space. In *CHI '98 Conference Summary on Human Factors in Computing Systems* (Los Angeles, California, USA, April 18-23, 1998) (*CHI '98*). Association for Computing Machinery, New York, NY, USA, 269-270. https://doi.org/10.1145/286498.286750

[16] Maxime Daniel, Guillaume Rivière, and Nadine Couture. 2019. CairnFORM: A Shape-Changing Ring Chart Notifying Renewable Energy Availability in Peripheral Locations. In *Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction* (Tempe, Arizona, USA, March 17-20, 2019) (*TEI '19*). Association for Computing Machinery, New York, NY, USA, 275-286. https://doi.org/10.1145/3294109.3295634

[17] Emmanuel Dubois, Philip Gray, and Laurence Nigay. 2003. ASUR++: Supporting the design of mobile mixed systems. *Interacting with Computers* 15, 4 (Aug. 2003), 497-520. https://doi.org/10.1016/S0953-5438(03)00037-7

[18] Emmanuel Dubois, Philip D. Gray, and Laurence Nigay. 2002. ASUR++: A Design Notation for Mobile Mixed Systems. In *Proceedings of the 4th International Symposium on Mobile Human-Computer Interaction* (Pisa, Italy, September 18-20, 2002) (*Mobile HCI '02*). Springer-Verlag, Berlin, Heidelberg, 123-139.

[19] Kenneth P. Fishkin. 2004. A Taxonomy for and Analysis of Tangible Interfaces. *Personal Ubiquitous Comput.* 8, 5 (Sept. 2004), 347-358. https://doi.org/10.1007/s00779-004-0297-4

[20] Kenneth P. Fishkin, Thomas P. Moran, and Beverly L. Harrison. 1999. Embodied User Interfaces: Towards Invisible User Interfaces. In *Engineering for Human-Computer Interaction: IFIP TC2/TC13 WG2.7/WG13.4 Seventh Working Conference on Engineering for Human-Computer Interaction* (Heraklion, Crete, Greece, September 14-18, 1998) (*EHCI '98*), Stéphane Chatty and Prasun Dewan (Eds.).

Springer US, Boston, MA, USA, 1-18. https://doi.org/10.1007/978-0-387-35349-4_1

[21] George W. Fitzmaurice and William Buxton. 1997. An Empirical Evaluation of Graspable User Interfaces: Towards Specialized, Space-Multiplexed Input. In *Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems* (Atlanta, Georgia, USA, March 22-27, 1997) (*CHI '97*). Association for Computing Machinery, New York, NY, USA, 43-50. https://doi.org/10.1145/258549.258578

[22] George W. Fitzmaurice, Hiroshi Ishii, and William A. S. Buxton. 1995. Bricks: Laying the Foundations for Graspable User Interfaces. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Denver, Colorado, USA, May 7-11, 1995) (*CHI '95*). ACM Press/Addison-Wesley Publishing Co., USA, 442-449. https://doi.org/10.1145/223904.223964

[23] Morten Fjeld, Nicholas Ironmonger, Fred Voorhorst, Martin Bichsel, and Matthias Rautberg. 1999. Camera control in a planar, graspable interface. In *Proceedings of the 17th IASTED International Conference on Applied Informatics* (Innsbruck, Austria, February 15-18, 1999) (*AI'99*). IASTED/ACTA Press, Anaheim, California, USA, 242-245. https://doi.org/10.3929/ethz-a-010042423

[24] Stéphanie Fleck, Guillaume Rivière, Regina Ticona-Herrera, and Nadine Couture. 2018. Classifier les interfaces tangibles par la cladistique : critères et fiches pour un inventaire collaboratif [Classifying Tangible User Interfaces with Cladistics: Criteria and Forms for a Collaborative Inventory]. In *Proceedings of the 30th Conference on l'Interaction Homme-Machine* (Brest, France, October 23-26, 2018) (*IHM '18*). Association for Computing Machinery & AFIMH, New York, NY, USA & Grenoble, France, 209-218. https://doi.org/10.1145/3286689.3286705

[25] Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge, and Hiroshi Ishii. 2013. inFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation. In *Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology* (St. Andrews, Scotland, UK, October 8-11, 2013) (*UIST '13*). Association for Computing Machinery, New York, NY, USA, 417-426. https://doi.org/10.1145/2501988.2502032

[26] John H. Frazer and Peter A. Frazer. 1980. Intelligent Physical Three-Dimensional Modelling Systems. In *Proceedings of Computer Graphics'80* (Brighton, England, UK, August 13-15, 1980) (*CG '80*). Online Publications, Northwood Hills, Middlesex, UK, 359-370.

[27] John H. Frazer and Peter A. Frazer. 1982. Three-Dimensional Data Input Devices. In *Proceedings of Computer/Graphics in the Building Process* (Washington, District of Columbia, USA, March 22-26, 1982) (*BP '82*). National Academy of Sciences, Washington, District of Columbia, USA, 409-416.

[28] Jérémy Frey, Renaud Gervais, Stéphanie Fleck, Fabien Lotte, and Martin Hachet. 2014. Teegi: Tangible EEG Interface. In *Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology* (Honolulu, Hawaii, USA, October 5-8, 2014) (*UIST '14*). Association for Computing Machinery, New York, NY, USA, 301-308. https://doi.org/10.1145/2642918.2647368

[29] Javier Gonzalez-Sanchez and Rafael Guerra-Silva. 2025. Exploring Extended Reality to Optimize Human-Cobot Interaction. In *Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction* (Melbourne, Australia, March 4-6, 2025) (*HRI '25*). IEEE Press, USA, 1314-1318. https://doi.org/10.5555/3721488.3721670

[30] Saul Greenberg and Chester Fitchett. 2001. Phidgets: Easy Development of Physical Interfaces through Physical Widgets. In *Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology* (Orlando, Florida, USA, November 11-14, 2001) (*UIST '01*). Association for Computing Machinery, New York, NY, USA, 209-218. https://doi.org/10.1145/502348.502388

[31] Stefan Th. Gries. 2004. Shouldn't it be breakfunch? A quantitative analysis of blend structure in English. *Linguistics* 42, 3 (April 2004), 639-667. https://doi.org/10.1515/ling.2004.021

[32] Stefan Th. Gries. 2006. Cognitive determinants of subtractive word formation: A corpus-based perspective. *Cognitive Linguistics* 17, 4 (Dec. 2006), 535-558. https://doi.org/10.1515/COG.2006.017

[33] Beverly L. Harrison, Kenneth P. Fishkin, Anuj Gujar, Carlos Mochon, and Roy Want. 1998. Squeeze Me, Hold Me, Tilt Me! An Exploration of Manipulative User Interfaces. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Los Angeles, California, USA, April 18-23, 1998) (*CHI '98*). ACM Press/Addison-Wesley Publishing Co., USA, 17-24. https://doi.org/10.1145/274644.274647

[34] Ken Hinckley. 1996. *Haptic Issues for Virtual Manipulation*. Ph. D. Dissertation. University of Virginia, Charlottesville, Virginia, USA. Advisor(s) Randy Pausch. AAI9724701.

[35] Ken Hinckley, Randy Pausch, John C. Goble, and Neal F. Kassell. 1994. Passive Real-World Interface Props for Neurosurgical Visualization. In *Conference Companion on Human Factors in Computing Systems* (Boston, Massachusetts, USA, April 2-7, 1994) (*CHI '94*). Association for Computing Machinery, New York, NY, USA, 452-458. https://doi.org/10.1145/259963.260443

[36] Lars Erik Holmqvist, Johan Redström, and Peter Ljungstrand. 1999. Token-Based Access to Digital Information. In *Handheld and Ubiquitous Computing: Proceedings of the First International Symposium* (Karlsruhe, Germany, September 27-29, 1999) (*HUC '99*), Hans-Werner Gellersen (Ed.). Springer Berlin Heidelberg, Germany,
234-245. https://doi.org/10.1007/3-540-48157-5_22

[37] Elise van den Hoven and Berry Eggen. 2004. Tangible Computing in Everyday Life: Extending Current Frameworks for Tangible User Interfaces with Personal Objects. In *Ambient Intelligence: Second European Symposium, EUSAI 2004, Proceedings (Eindhoven, The Netherlands, November 8-11, 2004) (Lecture Notes in Computer Science, Vol. 3295)*, Panos Markopoulos, Berry Eggen, Emile Aarts, and James L. Crowley (Eds.). Springer Berlin Heidelberg, Germany, 230-242. https://doi.org/10.1007/978-3-540-30473-9_22

[38] Hiroyasu Ichida, Yuichi Itoh, Yoshifumi Kitamura, and Fumio Kishino. 2004. Interactive Retrieval of 3D Shape Models Using Physical Objects. In *Proceedings of the 12th Annual ACM International Conference on Multimedia (New York, NY, USA, October 10-16, 2004) (MULTIMEDIA '04)*. Association for Computing Machinery, New York, NY, USA, 692-699. https://doi.org/10.1145/1027527.1027685

[39] Hiroshi Ishii. 1990. TeamWorkStation: towards a seamless shared workspace. In *Proceedings of the 1990 ACM Conference on Computer-Supported Cooperative Work (Los Angeles, California, USA, October 7-10, 1990) (CSCW '90)*. Association for Computing Machinery, New York, NY, USA, 13-26. https://doi.org/10.1145/99332.99337

[40] Hiroshi Ishii. 2008. Tangible Bits: Beyond Pixels. In *Proceedings of the 2nd International Conference on Tangible and Embedded Interaction (Bonn, Germany, February 18-20, 2008) (TEI '08)*. Association for Computing Machinery, New York, NY, USA, xv-xxv. https://doi.org/10.1145/1347390.1347392

[41] Hiroshi Ishii, Eran Ben-Joseph, John Underkoffler, Luke Yeung, Dan Chak, Zahra Kanji, and Ben Piper. 2002. Augmented Urban Planning Workbench: Overlaying Drawings, Physical Models and Digital Simulation. In *Proceedings of the 1st International Symposium on Mixed and Augmented Reality (Darmstadt, Germany, September 30 - October 1, 2002) (ISMAR '02)*. IEEE Computer Society, USA, 203-211. https://doi.org/10.1109/ISMAR.2002.1115090

[42] Hiroshi Ishii, H. Rich Fletcher, Jay Lee, Seungho Choo, Joanna Maria Berzowska, Craig Wisneski, Charlie Cano, Andres Hernandez, and Colin Bulthaup. 1999. MusicBottles. In *ACM SIGGRAPH 99 Conference Abstracts and Applications (Los Angeles, California, USA, August 8-13, 1999) (SIGGRAPH '99)*. Association for Computing Machinery, New York, NY, USA, 172. https://doi.org/10.1145/311625.311976

[43] Hiroshi Ishii and Minoru Kobayashi. 1992. ClearBoard: a seamless medium for shared drawing and conversation with eye contact. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, USA, May 3-7, 1992) (CHI '92)*. Association for Computing Machinery, New York, NY, USA, 525-532. https://doi.org/10.1145/142750.1427977

[44] Hiroshi Ishii, David Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. 2012. Radical Atoms: Beyond Tangible Bits, toward Transformable Materials. *Interactions* 19, 1 (Jan. 2012), 38-51. https://doi.org/10.1145/2065327.2065337

[45] Hiroshi Ishii, Ali Mazalek, and Jay Lee. 2001. Bottles as a Minimal Interface to Access Digital Information. In *CHI '01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington, USA, March 31 - April 5, 2001) (CHI EA '01)*. Association for Computing Machinery, New York, NY, USA, 187-188. https://doi.org/10.1145/634067.634180

[46] Hiroshi Ishii, Carlo Ratti, Ben Piper, Yao Wang, Assaf Biderman, and Eran Ben-Joseph. 2004. Bringing Clay and Sand into Digital Design – Continuous Tangible User Interfaces. *BT Technology Journal* 22, 4 (Oct. 2004), 287-299. https://doi.org/10.1023/B:BTJT.0000047607.16164.16

[47] Hiroshi Ishii, Sandia Ren, and Phil Frei. 2001. Pinwheels: Visualizing Information Flow in an Architectural Space. In *CHI '01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington, USA, March 31 - April 5, 2001) (CHI EA '01)*. Association for Computing Machinery, New York, NY, USA, 111-112. https://doi.org/10.1145/634067.634135

[48] Hiroshi Ishii and Brygg Ullmer. 1997. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. In *Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA, March 22-27, 1997) (CHI '97)*. Association for Computing Machinery, New York, NY, USA, 234-241. https://doi.org/10.1145/258549.258715

[49] Hiroshi Ishii, Craig Wisneski, Scott Brave, Andrew Dahley, Matt Gorbet, Brygg Ullmer, and Paul Yarin. 1998. AmbientROOM: Integrating Ambient Media with Architectural Space. In *CHI '98 Conference Summary on Human Factors in Computing Systems (Los Angeles, California, USA, April 18-23, 1998) (CHI '98)*. Association for Computing Machinery, New York, NY, USA, 173-174. https://doi.org/10.1145/286498.286652

[50] John Isidoro and Stan Sclaroff. 1998. Active Voodoo Dolls: a Vision Based Input Device for Nonrigid Control. In *Proceedings Computer Animation '98 (Philadelphia, Pennsylvania, USA, June 8-10, 1998) (CA '98)*. IEEE Computer Society, USA, 137-143. https://doi.org/10.1109/CA.1998.681918

[51] Robert J. K. Jacob, Hiroshi Ishii, Gian Pangaro, and James Patten. 2002. A Tangible Interface for Organizing Information Using a Grid. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, April 20-25, 2002) (CHI '02)*. Association for Computing Machinery, New York, NY, USA, 339-346. https://doi.org/10.1145/503376.503437

[52] Sergi Jordà, Günter Geiger, Marcos Alonso, and Martin Kaltenbunner. 2007. The ReacTable: Exploring the Synergy between Live Music Performance and Tabletop

Tangible Interfaces. In *Proceedings of the 1st International Conference on Tangible and Embedded Interaction (Baton Rouge, Louisiana, USA, February 15-17, 2007) (TEI '07)*. Association for Computing Machinery, New York, NY, USA, 139-146. https://doi.org/10.1145/1226969.1226998

[53] Kazue Kobayashi, Mitsunori Hirano, Atsunobu Narita, and Hiroshi Ishii. 2003. A Tangible Interface for IP Network Simulation. In *CHI '03 Extended Abstracts on Human Factors in Computing Systems* (Ft. Lauderdale, Florida, USA, April 5-10, 2003) (*CHI EA '03*). Association for Computing Machinery, New York, NY, USA, 800-811. https://doi.org/10.1145/765891.766000

[54] Geonsun Lee, Yue Yang, Jennifer Healey, and Dinesh Manocha. 2025. Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-Engaging in Immersive VR Meetings. In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems* (Yokohama, Japan, April 26-May 01, 2025) (*CHI '25*). Association for Computing Machinery, New York, NY, USA, Article 785, 20 pages. https://doi.org/10.1145/3706598.3714078

[55] Daniel Leithinger, Sean Follmer, Alex Olwal, and Hiroshi Ishii. 2014. Physical telepresence: shape capture and display for embodied, computer-mediated remote collaboration. In *Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology* (Honolulu, Hawaii, USA, October 5-8, 2014) (*UIST '14*). Association for Computing Machinery, New York, NY, USA, 461-470. https://doi.org/10.1145/2642918.2647377

[56] Daniel Leithinger and Hiroshi Ishii. 2010. Relief: A Scalable Actuated Shape Display. In *Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction* (Cambridge, Massachusetts, USA, January 25-27, 2010) (*TEI '10*). Association for Computing Machinery, New York, NY, USA, 221-222. https://doi.org/10.1145/1709886.1709928

[57] Daniel Leithinger, David Lakatos, Anthony DeVincenzi, Matthew Blackshaw, and Hiroshi Ishii. 2011. Direct and Gestural Interaction with Relief: A 2.5D Shape Display. In *Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology* (Santa Barbara, California, USA, October 16-19, 2011) (*UIST '11*). Association for Computing Machinery, New York, NY, USA, 541-548. https://doi.org/10.1145/2047196.2047268

[58] Pierre Mahieux, Romain Biannic, Sébastien Kubicki, and Ronan Querrec. 2022. SABLIER: A Tangible Interactor to Navigate through Space and Time. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems* (New Orleans, Louisiana, USA, April 30 - May 5, 2022) (*CHI '22*). Association for Computing Machinery, New York, NY, USA, Article 503, 12 pages. https://doi.org/10.1145/3491102.3517567

[59] Ali Mazalek and Elise van den Hoven. 2009. Framing Tangible Interaction Frameworks. *Artif. Intell. Eng. Des. Anal. Manuf.* 23, 3 (Aug. 2009), 225-235. https://doi.org/10.1017/S0890060409000201

[60] James Patten and Hiroshi Ishii. 2007. Mechanical Constraints as Computational Constraints in Tabletop Tangible Interfaces. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (San Jose, California, USA, April 27 - May 3, 2007) (*CHI '07*). Association for Computing Machinery, New York, NY, USA, 809-818. https://doi.org/10.1145/1240624.1240746

[61] James Patten, Ben Recht, and Hiroshi Ishii. 2002. Audiopad: A Tag-Based Interface for Musical Performance. In *Proceedings of the 2002 Conference on New Interfaces for Musical Expression* (Dublin, Ireland, May 24-26, 2002) (*NIME '02*). National University of Singapore, Singapore, SGP, 1-6. https://doi.org/10.5281/zenodo.1176458

[62] James Patten, Ben Recht, and Hiroshi Ishii. 2006. Interaction Techniques for Musical Performance with Tabletop Tangible Interfaces. In *Proceedings of the 2006 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology* (Hollywood, California, USA, June 14-16, 2006) (*ACE '06*). Association for Computing Machinery, New York, NY, USA, 27-es. https://doi.org/10.1145/1178823.1178856

[63] Radia Perlman. 1976. Using Computer Technology to Provide a Creative Learning Environment for Preschool Children. *AI Memo* 360 (May 1976), 31 pages.

[64] Ben Piper, Carlo Ratti, and Hiroshi Ishii. 2002. Illuminating Clay: A 3-D Tangible Interface for Landscape Analysis. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Minneapolis, Minnesota, USA, April 20-25, 2002) (*CHI '02*). Association for Computing Machinery, New York, NY, USA, 355-362. https://doi.org/10.1145/503376.503439

[65] Matthias Rauterberg, Martin Bichsel, Uwe Leonhardt, and Markus Meier. 1997. BUILD-IT: a computer vision-based interaction technique of a planning tool for construction and design. In *IFIP TC13 International Conference on Human-Computer Interaction* (Sydney, New South Wales, Australia, July 14-18, 1997) (*INTERACT '97*), Steve Howard, Judy Hammond, and Gitte Lindgaard (Eds.). Springer US, Boston, Massachusetts, USA, 587-588. https://doi.org/10.1007/978-0-387-35175-9_95

[66] Matthias Rauterberg, Morten Fjeld, Helmut Krueger, Martin Bichsel, Uwe Leonhardt, and Markus Meier. 1998. BUILD-IT: A Planning Tool for Construction and Design. In *CHI '98 Conference Summary on Human Factors in Computing Systems* (Los Angeles, California, USA, April 18-23, 1998) (*CHI '98*). Association for Computing Machinery, New York, NY, USA, 177-178. https://doi.org/10.1145/286498.286657
[67] Patrick Reuter, Guillaume Rivière, Nadine Couture, Nicolas Sorraing, Loïc Espinasse, and Robert Vergnieux. 2007. ArcheoTUI - a Tangible User Interface for the Virtual Reassembly of Fractured Archeological Objects. In *Proceedings of the 8th International Conference on Virtual Reality, Archaeology and Intelligent Cultural Heritage* (Brighton, England, UK, November 27-29, 2007) (VAST'07). Eurographics Association, Goslar, Germany, 15-22. https://doi.org/10.2312/VAST/VAST07/015-022

[68] Kimiko Ryokai, Stefan Marti, and Hiroshi Ishii. 2004. I/O Brush: Drawing with Everyday Objects as Ink. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Vienna, Austria, April 24-29, 2004) (CHI '04). Association for Computing Machinery, New York, NY, USA, 303-310. https://doi.org/10.1145/985692.985731

[69] Kimiko Ryokai, Stefan Marti, and Hiroshi Ishii. 2007. I/O Brush: Beyond Static Collages. In *CHI '07 Extended Abstracts on Human Factors in Computing Systems* (San Jose, California, USA, April 27 - May 3, 2007) (CHI EA '07). Association for Computing Machinery, New York, NY, USA, 1995-2000. https://doi.org/10.1145/1240866.1240938

[70] Orit Shaer and Eva Hornecker. 2010. Tangible User Interfaces: Past, Present, and Future Directions. *Foundations and Trends*® in *Human-Computer Interaction* 3, 1-2 (Jan. 2010), 4-137. https://doi.org/10.1561/1100000026

[71] Orit Shaer, Nancy Leland, Eduardo H. Calvillo-Gámez, and Robert J. K. Jacob. 2004. The TAC paradigm: Specifying tangible user interfaces. *Personal Ubiquitous Comput.* 8, 5 (Sept. 2004), 359-369. https://doi.org/10.1007/s00779-004-0298-3

[72] Darja Stoeva, Andreas Kriegler, and Margrit Gelautz. 2024. Body Movement Mirroring and Synchrony in Human-Robot Interaction. *J. Hum.-Robot Interact.* 13, 4, Article 47 (Oct. 2024), 26 pages. https://doi.org/10.1145/3682074

[73] Steve Sutphen, Ehud Sharlin, Benjamin Watson, and John H. Frazer. 2000. Reviving a Tangible Interface Affording 3D Spatial Interaction. In *Proceedings of 11th Western Canadian Computer Graphics Symposium* (Panorama, British Columbia, Canada, March 26-29, 2000) (*SkiGraph'00*), 155-166.

[74] Brygg Ullmer. 2002. *Tangible Interfaces for Manipulating Aggregates of Digital Information*. Ph.D. Dissertation. Massachusetts Institute of Technology, Cambridge, Massachusetts, USA. Advisor(s) Hiroshi Ishii. AAI0804673.

[75] Brygg Ullmer and Hiroshi Ishii. 1997. The MetaDESK: Models and Prototypes for Tangible User Interfaces. In *Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology* (Banff, Alberta, Canada, October 14-17, 1997) (UIST '97). Association for Computing Machinery, New York, NY, USA, 223-232. https://doi.org/10.1145/263407.263551

[76] Brygg Ullmer and Hiroshi Ishii. 1999. MediaBlocks: Tangible Interfaces for Online Media. In *CHI '99 Extended Abstracts on Human Factors in Computing Systems* (Pittsburgh, Pennsylvania, USA, May 15-20, 1999) (CHI EA '99). Association for Computing Machinery, New York, NY, USA, 31-32. https://doi.org/10.1145/632716.632739

[77] Brygg Ullmer and Hiroshi Ishii. 2000. Emerging Frameworks for Tangible User Interfaces. *IBM Syst. J.* 39, 3-4 (July 2000), 915-931. https://doi.org/10.1147/sj.393.0915

[78] Brygg Ullmer, Hiroshi Ishii, and Dylan Glas. 1998. MediaBlocks: Physical Containers, Transports, and Controls for Online Media. In *Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques* (Orlando, Florida, USA, July 19-24, 1998) (*SIGGRAPH '98*). Association for Computing Machinery, New York, NY, USA, 379-386. https://doi.org/10.1145/280814.280940

[79] Brygg Ullmer, Hiroshi Ishii, and Robert J. K. Jacob. 2005. Token+constraint Systems for Tangible Interaction with Digital Information. *ACM Trans. Comput.-Hum. Interact.* 12, 1 (March 2005), 81-118. https://doi.org/10.1145/1057237.1057242

[80] Brygg Ullmer, Rajesh Sankaran, Srikanth Jandhyala, Blake Tregre, Cornelius Toole, Karun Kallakuri, Christopher Laan, Matthew Hess, Farid Harhad, Urban Wiggins, and Shining Sun. 2008. Tangible Menus and Interaction Trays: Core Tangibles for Common Physical/Digital Activities. In *Proceedings of the 2nd International Conference on Tangible and Embedded Interaction* (Bonn, Germany, February 18-20, 2008) (TEI '08). Association for Computing Machinery, New York, NY, USA, 209-212. https://doi.org/10.1145/1347390.1347436

[81] Brygg Ullmer, Orit Shaer, Ali Mazalek, and Caroline Hummels. 2022. *Weaving Fire into Form: Aspirations for Tangible and Embodied Interaction* (1st ed.). Vol. 44. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3544564

[82] John Underkoffler and Hiroshi Ishii. 1999. Urp: A Luminous-Tangible Workbench for Urban Planning and Design. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Pittsburgh, Pennsylvania, USA, May 15-20, 1999) (CHI '99). Association for Computing Machinery, New York, NY, USA, 386-393. https://doi.org/10.1145/302979.303114

[83] Annemiek Veldhuis, Rong-Hao Liang, and Tilde Bekker. 2020. CoDa: Collaborative Data Interpretation Through an Interactive Tangible Scatterplot. In *Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction* (Sydney, New South Wales, Australia, February 9-12, 2020) (TEI '20). Association for Computing Machinery, New York, NY, USA, 323-336. https://doi.org/10.1145/3374920.3374934

[84] Mark Weiser. 1991. The Computer for the 21st Century. *Scientific American* 265, 3 (Sept. 1991), 94-104. https://doi.org/10.1038/scientificamerican0991-94

[85] Mark Weiser. 1999. The Computer for the 21st Century. *SIGMOBILE Mob. Comput. Commun.* Rev. 3, 3 (July 1999), 3-11. https://doi.org/10.1145/329124.329126

[86] Pierre Wellner. 1991. The DigitalDesk Calculator: Tangible Manipulation on a Desk Top Display. In *Proceedings of the 4th Annual ACM Symposium on User Interface Software and Technology* (Hilton Head, South Carolina, USA, November 11-13, 1991) (UIST '91). Association for Computing Machinery, New York, NY, USA, 27-33. https://doi.org/10.1145/120782.120785

[87] Pierre Wellner. 1993. Interacting with Paper on the DigitalDesk. *Commun. ACM* 36, 7 (July 1993), 87-96. https://doi.org/10.1145/159544.159630

[88] Craig Wisneski, Hiroshi Ishii, Andrew Dahley, Matthew G. Gorbet, Scott Brave, Brygg Ullmer, and Paul Yarin. 1998. Ambient Displays: Turning Architectural Space into an Interface between People and Digital Information. In *Proceedings of the First International Workshop on Cooperative Buildings, Integrating Information, Organization, and Architecture* (Darmstadt, Germany, February 25-26, 1998) (CoBuild '98). Springer-Verlag, Berlin/Heidelberg, Germany, 22-32. https://doi.org/10.5555/645968.674740

[89] Andy Wu, Sam Mendenhall, Jayraj Jog, Loring Scotty Hoag, and Ali Mazalek. 2012. A Nested API Structure to Simplify Cross-Device Communication. In *Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction* (Kingston, Ontario, Canada, February 19-22, 2012) (TEI '12). Association for Computing Machinery, New York, NY, USA, 225-232. https://doi.org/10.1145/2148131.2148180

[90] Jamie Zigelbaum, Adam Kumpf, Alejandro Vazquez, and Hiroshi Ishii. 2008. Slurp: Tangibility Spatiality and an Eyedropper. In *CHI '08 Extended Abstracts on Human Factors in Computing Systems* (Florence, Italy, May 5-10, 2008) (CHI EA '08). Association for Computing Machinery, New York, NY, USA, 2565-2574. https://doi.org/10.1145/1358628.1358713
<figure><img src="image_4.png" /><figcaption>(a) Relief: Terrain dynamic exploration (#26).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) SABLIER: Time navigation in cultural heritage scopes (#33).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(c) Head Prop: Neurosurgical visualization (#5).</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(d) ArcheoTUI: Virtual archaeology fragments' assembly (#23).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(e) GraspDraw: 2D shape edition (#6).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(f) GeoTUI: Cutting line selection for subsoil exploration (#24).</figcaption></figure>

Licenses: (b) Castle 3D model: Ramon Duran, CC0; (c) Doll's head 3D model: David Gilman, CC BY 4.0 - Brain MRI picture: Rudy Liggett, CC0; (d) Vase 3D model: Robert Tolliver, CC BY 4.0;

(f) Topographic map: Gordon Dylan Johnson, CC0; (c,d,e,f) Chair 3D model: Zach Steindler, MIT License.

<caption>**Figure 2:** Pointing out physical representations and controls for six depicted example specimen applications.</caption>
<figure><img src="image_10.png" /><figcaption>Figure 3: Mapping thirteen previous terms for physical representations and controls on the new conceptual space.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Figure 4: Matching previous classifications with what-how tangibility classes through a collection of applications.</figcaption></figure>