# Information Holodeck: Thinking in Technology Ecologies

Sharon Lynn Chu, Francis Quek

► To cite this version:

Sharon Lynn Chu, Francis Quek. Information Holodeck: Thinking in Technology Ecologies. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.167-184, 10.1007/978-3-642-40483-2_12. hal-01497434

HAL Id: hal-01497434

https://inria.hal.science/hal-01497434

Submitted on 28 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Information Holodeck: Thinking in Technology Ecologies

Sharon Lynn Chu¹, and Francis Quek¹

¹ 2202 KnowledgeWorks II, Kraft Drive

{sharilyn, quek}@vt.edu

**Abstract.** Information can be persistently represented on a multitude of devices beyond a single screen and session. This paper explores how technological *dis- play and device ecosystems (DDEs)* may support human thinking, learning and sensemaking. We propose a theoretical foundation that extends Vygotsky's sign mediation triangle to include digital information. Through a process we call *ob- jectification*, perceivable objects, e.g. interface objects, tangible technologies, can be associated with signs to support thinking. We present a qualitative study of learning in a testbed *DDE* with 12 graduate students. We developed a meth- od that traces digital objects within 'thinking episodes' to help us evaluate how technology configurations support *objectification*. Our findings relate two story- lines of how *DDE* technologies may afford objectification. Our work advances a method informed by psychological theory to examine device ecologies for their potential for learning, and articulates affordances for the design of tech- nology that can help to support higher thought.

**Keywords:** Ecology, technology, devices, displays, thinking, sensemaking, ob- jectification, embodied interaction, design

## 1 Introduction

In the TV Sci-Fi series, Star Trek Voyager, there is an Emergency Medical Hologram Mk 1 (EMH) doctor played by Robert Picardo. Although the EMH was designed to simulate a personality, at the beginning of the series he/it was just a program restrict- ed to the Sick Bay (which had ‘Holo-emitters’ to support his materialization), and could take physical form for periods no longer than necessary to see an emergency patient. But because Voyager was stranded far from home without medical support, the technical staff ‘stabilized his matrix’, so he could remain persistently corporeal. Over time, they extended the technology coverage to enable him to materialize in larger portions of the spaceship, and acquired a ‘mobile holo-emitter’ that enabled him to be material continuously. He became ‘real’.

This technology tale is relevant to our deployment and use of technology because just like the doctor, information with which we interact is at the same time digital bits and materialized representations on screens and devices. Just as an ecosystem of tech- nological enablements support the materialization of the doctor, a multitude of devic- es are increasingly able to support persistent representations of information materiali-
zations beyond the restrictions of a single screen and a single session. The question is
how may our information become ‘real’ and why this may be important for learning.

We have a good understanding of how technology augments our human physical
skills (e.g., digging), and our cognitive skills (e.g., memory) to a certain extent. It has
proven more difficult for us to understand how technology can augment the higher
human thought. Investigations on information, communication, artifact and media
ecologies, have addressed mostly philosophical, behavioral, design and technical
perspectives of these new environments. This paper explores how these technological
display and device ecosystems (DDEs) in the physical world may function to support
human thinking, learning and sensemaking.

First, we review and synthesize the literature on technology ecologies, and then
present a model to make sense of the process of thinking in DDEs derived from the
theories of the Russian psychologist, Lev Vygotsky. We describe a study that uses our
theoretical framework to investigate how students think in the DDEs that they form,
and conclude the paper by discussing technology affordances and their implications
for the design of DDEs for learning.

## 2 A Review of Technology Ecosystems

A range of research has hitherto applied the metaphor of a biological ecosystem to human activities with technology for illustrative purposes and to stimulate intellectual discussions. However, this body of work does not always form a coherent whole, and it is a challenging undertaking to present a comprehensive synthesized account. We reviewed many technology ecology notions in the literature, but in the interest of space, we shall simply list them and expand only on the most relevant ones here.

The overall message underlying the different positions in the literature is that artifacts, devices, systems and products cannot be studied in isolation but can only be truly understood when seen in the broader perspective of the universe they inhabit. Depending on the position taken, the universe can consist of one's physical context, other artifacts used, or one's practices and culture using technology. We classify the perspectives into three categories: philosophical positions, empirical study results and technical frameworks.

Among the theoretical or philosophical positions taken on technology ecology concepts, one can find the ‘media ecology’ by McLuhan [1], Altheide’s [2] ‘ecology of communication’ or ‘communicative ecology’, the ‘information ecology’ by Nardi & O’Day [3], Tungare et al.’s [4] ‘personal information ecosystem’, Krippendorf’s [5] ‘ecology of artifacts’, and Rick’s [6] proposition of a ‘classroom ecology of devices’. Gibson [7], Suchman [8] and Norman [9] have also used the ecological metaphor.

Research of technology ecologies that present an empirical study of some sort include Huang, Mynatt & Trimble's [10] 'display ecology', Enquist, Tollmar & Corry's [11] 'interaction ecology', Dearman & Pierce's [12] 'computing with multiple devices', Forlizzi's [13] 'product ecology', Jung et al.'s [14] 'personal ecology of interactive artifacts', and Coughlan et al.'s [15] 'device ecology'. From the management sciences, Bailey & Barley [16] present an extensive 'shadowing' study of 'teaching-
learning ecologies', tracking knowledge as it moves through six engineering firms. Their focus however was on people and not technology.

Some of the technical frameworks that have been proposed to implement technological ecologies are Loke & Ling's [17] use of petri nets to represent the state of devices, the 'task migration framework' by Pyla et al. [18], and Pierce & Nichols' [19] framework based on instant messaging to enable multi-device user experiences.

Among the different conceptions of ecologies that we reviewed, only Coughlan et al. [15] and Jung et al. [14] presented formal empirical studies about technology use in a learning context. Rick [6] points out the importance of a classroom ecology, but does not provide any supporting study. Coughlan et al.'s investigation informs the design of ecologies by studying transitions in foci across devices (a tabletop computer with a mirrored projection, laptops, a telephone) in three short controlled activities, carried out in a "technologically-enhanced indoor space". Communication across devices was provided by a Central Management System, and instant messaging. The focus of their study was on how device ecologies can support collaboration. Study results presented a set of "seams" that represent disconnects in a device ecology that can affect users' behaviors. Their study however gave little indication of how one can understand whether or how learning has occurred within the context they constructed.

Jung et al. [14] studied one's network of personal artifacts through the lens of 'factors' and 'layers' within a 'personal ecology of interactive artifacts', described as a "set of all physical artifacts with some level of interactivity enabled by digital technology that a person owns, has access to, and uses". They make use of two methods called the *Personal Inventory*, based on a simplified version of the Repertory Grid Technique, and the *Ecology Map*, which consists of sketching using sticky notes to probe about a person's device ecology. Their exploratory study with ten graduate students found that perceived attributes of an artifact can be classified into two categories, designed properties (physical, functional, informational, interactive aspects) and subjective values (experiential, emotional, social). They further specify the different types of relations that artifacts in a personal ecology can have, based on: purpose of use, context of use, or subjective meaning. Their study results, although very helpful to understand the nature and types of technological ecologies, again do not consider the process of learning.

We conceive of a technology ecology where devices function, not as individual gadgets, but in ecosystems to deliver an experience. As such, we define a *DDE* as a *mesh of interacting displays and devices that enable the manifestation and manipulation of digital information to deliver a cohesive learning experience* [20].

## 3 A Model of Thinking in DDEs

In the context of our definition of a *DDE* above, we present a high-level model of how thinking can be understood with regards to interactions with digital information through physical technological devices. Figure 1 illustrates our model. People's thoughts (what we label as *Thought objects*, TOs), can be encoded into *information*, that we describe as *digital objects* (DOs). Through technology in the *DDE*, DOs can
be *externalized* as *Manifest Objects* (MOs) that are percep-
tible to the human senses. Ex-
amples of MOs can be dis-
played documents, images, or
file icons. These MOs, however,
also have the capacity to medi-
ate further *thinking* – a process
we call *objectification*. Exter-

nalization of TOs as MOs allows the user’s perceptual and spatial abilities to partici-
pate in the thinking process. This idea is related to that of ‘distributed cognition’ [21],
which states that human knowledge and cognitive processes are offloaded into the
environment as external representations. Our contribution is that we suggest the
mechanism by which cognition is distributed into the environment through the *DDE*.
In the next section, we make use of Vygotsky’s sign mediation theory to describe a
mechanism by which the different processes in our high-level model take place. Other
perhaps more commonly known theories that have been derived from or are closely
associated with Vygotsky’s theory include, for instance, activity theory and
distributed cognition.

The Sign Mediation Triangle and
Materialized Thought

We turn to Vygotsky's theories to
understand thinking in *DDEs* be-
cause he proposes a way by which
things in the environment may be

brought into the very process of thinking. According to sign mediation theory, lan-
guage is conceived of as a psychological tool by which both cultural (interpersonal)
and psychological (intrapersonal) thought are ‘mediated’ [22, 23]. Signs are self-
generated linguistic stimuli [22] that extend the operation of human cognition beyond
the confines of the strictly biological system. Figure 2 illustrates Vygotsky’s sign
mediation theory. For example, a student in algebra may be introduced to the summa-
tion concept, the TO: a + b + c .... She understands and is able to perform the opera-
tion. However, if she had to think of details of the concept each time she applies it,
the limits of her memory, attention, and mental processing would make further ad-
vancement untenable. Thus, she encodes this concept as a mental ‘sign’ – the concept
of ‘summation’. She is able to think of the operation simply as Σ, and to employ this
in further learning (e.g. Σ<sup>6</sup><sub>n=1</sub>(n<sup>2</sup> − 1)). As the sign becomes ‘internalized’ it becomes
in essence the object in her thinking. She can ‘unpack’ the sign as needed to attend to
the details.

In Vygotsky's model, *signs* may take the form of both internal or external symbols (a stick between a child's legs becomes his horse, and a block represents an idea [22]), or as an abstract entity grounded in language. Externally instantiated signs are referred to as *material carriers* (MC) of thought [24] (see Figure 2). Any perceiva-

<figure><img src="image_2.png" /><figcaption>Figure 2 Vygotsky's sign mediation theory</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 1 High-Level Model of Thinking in DDEs</figcaption></figure>
ble *object* (spatial location, gestures, objects or even sounds) in the environment can opportunistically and temporally be appropriated for use as MCs to assist thinking by bringing spatial ability and perception into play. In theory, the MC can be anything that may or may not resemble the mental object. In our example, the MC for convolution can be a specific hand gesture or a written * symbol.

In the early 1900s, Vygotsky
obviously never encountered the
magic of computation and mod-
ern display technology. The
number of MCs one can entertain
at any one time is limited by the
meanings one can assign and
recall for amorphous objects and
space. We advance a theoretical
framework, in Figure 3, that ex-
tends the model of the sign and
MC, to include the ‘magic’ of

digitality. The top of Figure 3 from *thought object* (TO) to mental *sign* and *material carrier* (MC) replicates Vygotsky’s mediation theory. In the information world, a TO can be *encoded* in a *digital object* (DO). The DO can at times even *extend* the TO, such as with information about how the convolution formula was derived. The mental *sign*, as well, thus becomes *associated* with the DO. Through DDE technologies, the DO is *expressed* as a MO, e.g. as a website displayed in a browser on a mobile device. If one mentally appropriates this MO in the process of thinking, in essence shedding ‘intentional regard’ [25] to the object, the MO becomes synonymous with the MC, i.e. the binding process of *objectification*.

Our model expands the
power of MOs to support
thinking through MCs in two
ways. First, MOs are iconic
and provide mnemonic refer-
ence in ways that arbitrary
objects and space cannot. This
potentially expands the number
of MCs one can employ over
longer periods of time. Second,

digital media that are encodings of one's *thought objects* (and hence associated with one's mediating *signs*) can serve as an external long-term detailed representation of a piece of knowledge, thus extending the depth of thinking one can handle. Our model describes a thinking process that is different from simply opening a document on a screen to refresh one's memory about an idea because then the document simply becomes something one queries for information rather than wields in the process of thinking. With multiple users, the MO, being the perceivable component of the model, becomes (in co-located situations) or may be made (in remote scenarios) the shared object among all users (Figure 4). We envision users taking advantage of the

<figure><img src="image_4.png" /><figcaption>Figure 3 Extension to the sign triangle</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4 Thinking in groups</figcaption></figure>
broader interactions made possible by a *DDE* to place, organize, and possibly step back and view networks of MCs.

# 4 The Study

We conducted a qualitative study of the manifestation of the thinking process in *DDEs*. Adopting the hermeneutic perspective, the inquiry was to seek “understanding rather than explanation” and to uncover systems of meaning from the participant’s vantage point [26]. We follow the methodology specified by Patterson & Williams [26] who advance that a hermeneutic approach begins with establishing a point-of-view (our ‘objectification framework’), from which an ‘organizing system’ is derived to meaningfully organize, interpret and present data [27].

## Study Methodology

The study enabled a set of students to experience a *DDE* for the purpose of doing a knowledge discovery assignment over a two-month period, and capturing their experience through self-reports and interviews. The participants were 12 computer science students in a graduate class. The assignment, to be completed in teams of three, was to research and write a report about the emergent field of ‘Physical Computing’.

<figure><img src="image_6.png" /><figcaption>Figure 5 Suite of devices available to study participants</figcaption></figure>

The students were each given a ‘testbed suite’ of devices comprising an *iPod Touch*, an *iPad*, and a 27” *iMac* to use as their own throughout a semester. The *iMac*, embedded in a custom casing that allowed it to be laid on the table horizontally or at an angle, was endowed with a touch overlay to enable touch interaction (see Figure 5). The rationale for the dissemination of devices of various form factors was that we wanted to provide the students with an experience of a heterogeneous *DDE*. Additionally, to provide a basic information architecture that crosses devices in the ecology, we installed the free file sharing service *Dropbox*, the notetaking service, *Evernote*, with a paid subscription, and the paid PDF reader *GoodReader*<fnref n="1" /> on the *iPad*. The first two services are based on the cloud, and *GoodReader* can be paired with *Dropbox* to allow data synchronization and transfer. We provided the students with a tutorial session on how to use the three services/applications prior to the study.

The class was held in our research center, which contains several large display screens spread out in different meeting rooms, and a large interactive vision-based

<footnote n="1">¹ www.dropbox.com, www.evernote.com http://www.goodreader.net/goodreader.html</footnote>
touch screen prototype [28]. All students had constant access to the building. They were asked to use technology as much as possible while doing the assignment, including the device suite given to them, the large display screens around the center, and any other devices that they own such as laptops and other desktops. They could also use the devices freely for any other purposes.

## Data Collection

We conducted semi-structured interviews and surveys to capture the students' experiences with technology before they were given our suite of devices (pre-experience), and after they have used them for the semester (post-experience). Each interview lasted for about an hour and was audio recorded. The purpose of doing a pre- and post-interview was not to perform an experimental comparison, but rather, to be cognizant of the initial conditions under which our participants joined the study.

The pre-experience interviews and surveys collected data about the students' behaviors and 'strategies of use' of the devices, as well as their attitudes toward and perceptions of the devices and processes. More specifically, the survey, which was completed in the presence of the interviewer, asked about the list of devices used and owned, the duration and frequency of use, familiarity with devices, purpose of use, perceived usability of devices (measured on a 7-point likert scale by adapting the IBM usability questionnaire [29] with dimensions like ease-of-use, comfort, efficiency, satisfaction), and data sharing methods. Qualitative comments were also encouraged during the completion of the survey. The interview addressed similar themes as the survey, and added questions about the role of devices in their idea generation and paper writing processes, and device interactions in their existing ecologies.

The post-experience interviews asked about the process of assignment completion; impact of the devices on practice; problems with devices, information sharing, and the writing process; context/situations of devices use; influencing factors of information use; personal sensemaking; work distribution and team coordination; use of file/data sharing services; longer term device use; general assessment of the ecology; desired changes; meeting contexts; idea generation process; and, information presentation.

## Data Analysis

All pre- and post-interviews were fully transcribed with timestamps. Survey data were imported into a spreadsheet for analysis. To separate relevant from irrelevant verbiage in the transcripts, two coders did a first round of analysis of all the transcripts to identify 'thinking episodes'. Following Barker's ([30] in [16]) concept of "behavioral episodes", we define a thinking episode as a "coherent run of [thinking] in which the constituent [processes] have a constant direction, a purpose". Some examples include 'filtering out important keywords', 'searching for information', and 'annotating papers'. We acknowledge that thinking is a process with no explicit end point or necessary resulting outcome. However, the method of tracking thinking episodes allowed us to identify possible situations when the students engaged in thinking, and was thus presented with opportunities for learning. It is not our interest in this paper to evaluate
the uptake of knowledge or retention per se. Moreover, in this paper, we understand
the commonly used term of sensemaking as being a necessary part of thinking.

<figure><img src="image_7.png" /><figcaption>Figure 6. Left. Analysis procedures; Right. Example analysis</figcaption></figure>

In order to make sense of our data on the students’ experiences of their ecologies,
we engaged in what we call a ‘DO (Digital Object)-path analysis’. The steps involved
in our analysis, illustrated in Figure 6 (Left) together with an example analysis
(Right), were as follows: each thinking episode was analyzed to distinguish the differ-
ent components of TOs, DOs, MOs, and technologies used. We noticed that certain
objects had a high frequency of occurrence in our collated list. **To focus our analysis
thus, we decided to trace the use of three of the common DOs: GoogleDocs doc-
uments, Evernote notes, and PDF documents.** We shall call these our three focus
objects for our study. All thinking episodes where any of the three DOs were mani-
fested as MOs were copied to a separate spreadsheet for analysis. We then analyzed
the different technology configurations in which the DO in question was used in each
of the relevant thinking episodes.

By tracing the model components in context of use, we determined whether there
were indications that *objectification* had occurred in any form for the particular task at
hand. Two separate coders identified *objectification* indications by making a judg-
ment as to whether the MO (e.g. *Evernote* note displayed on a tablet screen) brought
spatial ability and perception into the process of thinking. Subsequently, a feature
extraction was done, whereby we identified the characteristics of the technologies, or
technology configurations, that seemed to have supported or hindered the *objectifica-
tion* process. The characteristics were further categorized along uncovered themes.

# 5 Findings

To frame our findings, we will first describe the starting context of our participants
from results of the pre-interviews/surveys, with an emphasis on processes of thinking
with devices. This also conveys an understanding of the work strategies of the partici-
pants in the testbed *DDE* of the study, since practices, as we anticipated, did not
change significantly over the two months. We then describe case scenarios from our DO-path analysis, before expounding on derived technology affordances for *DDEs*.

## Initial Experience of DDEs

It was evident that some of the students already had a rather extensive ecology of devices prior to the study. The most common devices that they possessed were laptops, desktops and smartphones. Fewer owned tablets, music players and large displays (Figure 7). Among those who owned them, laptops and tablets were used daily. Desktops were mostly used only two to three times per week. All, apart from two, indicated that they used their smartphones everyday. The use of music players and large displays was more sporadic.

Laptops and smartphones were mostly ranked first in familiarity, followed by tablets and music players. Students were generally least familiar with

<figure><img src="image_8.png" /><figcaption>Figure 7 Distribution of devices owned at start of study</figcaption></figure>

desktops and large displays. Their use of the devices can be classified into three types: *instantaneous* use, e.g. checking email, calendar, news, social networks (tablet, smartphone); *extended* use, e.g. reading papers, video streaming (laptop, tablet, large display, music player); and *long-term* use, e.g. backup of work, managing Internet (desktop, music player).

All except one had experience with *Dropbox* prior to the study, with an average duration of use of a year. Only three of the students had used *Evernote*, with an average of two months of use. All of the others however made use of other notetaking applications such as *GoogleDocs* and *TextMate*. Among those who already had a tablet, only one of them made use of *GoodReader*. The other students used other PDF readers/annotators such as *Adobe Reader* and *QuickOffice*. In terms of usability, the laptop (*M* = 2.57) was ranked the highest, followed by the tablet (*M* = 2.41), the desktop (*M* = 2.39), the smartphone (*M* = 1.92), the music player (*M* = 1.88), and finally the large display (*M* = 1.85).

In the existing *DDEs* of the students, the thinking process for assignments mostly followed the standard loops of foraging and sensemaking [31], with the prominent use of the laptop and desktop throughout. Actions in which a thinking process was evident on these devices included for instance, “just open[ing] up Microsoft Word and start typing in ideas”, taking notes in a text file on *Dropbox* “when I’m surfing and I find something interesting”, “categorizing my papers through...folders”, and “start[ing] to kind of do a treelike structure from the cited references”.

Those who had a tablet reported making use of it only occasionally for purposes of work. We could not identify many thinking episodes in their accounts of use of tablets. Among the few that we found such as for annotating readings, reading information in the browser or notetaking in the native ‘Notes’ application, the annotations were only “like an intermediary step to a final annotation”, the website was “just [to
take] a look at”, and the notes (e.g. “on page X second paragraph is interesting”) would remain on the tablet.

The use of non-digital materials, such as sticky notes, notebooks and pieces of paper, was evident mostly in the ‘foraging’ part of the process. Paper materials appeared to have been used only as quick, temporary MCs and rarely had their content transferred to digital in the same exact form. One participant, for instance, recounted that she would write down ‘idea fragments’ on paper, and then “when I finally get my idea, I would put it on TextEdit”, after which she would throw away the paper pieces. Or for reading papers, another participant “tend to print them out”, “take some notes on it. And then turn it into some thrash”. A number of the participants also reported relying only on memorization, “I remember some keywords of it, so I can get it when I want to”, without the use of any devices.

## Objectification as Manifested in Testbed DDEs

Recall that our analysis traces three kinds of ‘focus objects’ (PDFs, Evernote notes, & GoogleDocs) used by our participants as DOs. Our DO-path analysis of the post-interviews brought to light different ways in which the *objectification* process occurred in the *DDEs* that the students experienced. We follow one of our three ‘focus’ objects as it was involved in two example storylines extracted from thinking episodes.

**Storyline 1: Identifying relevant sources for the assignment.** One team employed a strategy by which they brainstormed a set of keywords and organized these on a whiteboard as a concept map, after attempting to use the vision-based interactive touch screen and failing. From this, they selected a subset of keywords to explore, and used these for literature searches. One of our participants related the following process, which we map to our model:

1. He searched for sources using the keywords and left these in tabs in his laptop browser. These were not saved to disk nor named. The browser tabs served as a ‘bag of finds’ whose relevance was yet to be determined (“I didn’t save them on my computer. They were opened in my browser actually, because I wanted to filter what I have and see if I need to *take these* or not.”).

2. He brought his bag of finds on his laptop to the next meeting and presented them to the team, and selected some to *take* (“I just renamed the paper to keep track of *which is which*. If I saw robotics ... so this is *the robotics paper*. If there is a lot of robotics papers, I will say *robotics 1, 2, 3*, and or I would tie it to the name of the author.”). It is at this point that the participant associates particular PDFs with *signs*. Relating this to Figure 3, the ‘concept of robotics paper as relevant to physical computing’ was the *thought object*, the *sign* was the name of the file, and the *manifest object* was either the displayed PDF document or the named file icon in the folder.

3. He would copy the PDFs to his iPad for further reading, and refined the naming of the PDF according to his understanding of the paper (“So after you read the paper ... OK ... this paper is about *this*. A way I used to do is to just rename the paper to keep track of *which is which*.”).

In this storyline, we observe the formation of *signs* and the use of *manifest objects* as MCs to help the participant make sense of how each paper (representing a specific
concept) fits in 'Physical Computing'. This strategy was seen in two other partici-
pants. One had put her 'bag of papers' on her iPad and had them open in GoodReader
tabs to share and discuss with her teammates ('... papers that were open on it ... since
we were talking about what we researched. I was able to say I found this paper, it
talks about this and this'). She was using the iPad on which the PDF was material-
ized as an MO to support her discussion and thinking (the team did not actually read
the paper on the iPad screen, the participant merely used it as an objectified sign to
reference the information the PDF contained). A third participant employed an almost
identical strategy with a slight difference in that he did an initial filtering of his 'bags
of finds' on his laptop browser tabs, and winnowed these to 15 - 20 papers that he
saved and named in his Dropbox. He copied these to his iPad for further reading and
annotation. There was also evidence of his employment of filenames as meaningful
signs to give him mnemonic access to the PDFs, and his use of the displayed PDFs as
MCs in the process of reasoning about 'Physical Computing'.

**Storyline 2: Creating the assignment report.** One of the participants organized his paper building from an outline written in a *GoogleDocs* document. This outline served as the conceptual frame on which he worked: "I have the outline and each point I know which PDF is linked to what, so I would read the PDF and while I am reading, I get an idea, and I start writing on the paper." He read the paper on his *iPad* and wrote in *GoogleDocs* on his laptop simultaneously. He mentally associated each outline item in his *GoogleDocs* outline with several PDFs.

Here, we see that he has a two-level conceptualization of the PDF paper he is read-
ing. The idea of an outline item is a *thought object*. Both the mnemonic name of the
PDF and the concomitant name of the outline point serve as *signs* for the TO. The TO
has two different simultaneous MOs. The outline item in the opened *GoogleDocs*
document is a MO with which he associates the TO, but it is also clear that he "knows
which PDF [it] is linked" to. A second MO is thus the display of the PDF on his *iPad*.
Further, since the PDF summoned to the *iPad* is open to reading and inquiry, the con-
tents of the PDF themselves become another level of TOs, *signs*, and MOs. At each
level, *objectification* may occur to allow the use of the MO as an MC for reasoning. It
is precisely this nature of *signs* that are able to hold entire concepts for thinking, and
be unpacked into its contents for inquiry at a finer degree of abstraction that makes
the theory a potent vehicle to understand thinking in *DDEs*.

This same two-level strategy was employed by another participant who used PDFs
in *Dropbox* folders to support conceptualization. The mnemonic names of the PDFs
served as the *signs* and the preview display of the PDFs (quick look feature in the
*Macbook*) in the *Dropbox* folder served as the MO or MC when he referred to it while
writing. The participant wished he could add more information to the *sign* at the level
of the whole PDF, stating "Originally we discussed annotating them (the PDFs), but
we found that it was kind of difficult to actually do it because we just wanted to have
a summary of each paper but Dropbox on the iPad would not let us create a text file
[of the same name] next to the papers." This shows that the participant was thinking
at the level of the PDF as a whole, and wanted to associate meta-information at the
level of PDF within the folder structure.
Another three participants exhibited the same two-level conceptual structure using yet other methods to provide overarching structure of how the *sign* associated with each PDF relates to the paper as a whole. Two of them employed a citation manager, e.g. *Endnote*, *Mendeley*, to maintain citations and to add an annotation at the level of the whole citation for each PDF. An entire PDF thus could be used as a contained unit of thinking for paper writings. Our last participant recorded to use this strategy employed *Evernote* for her higher-level organization. She had an *Evernote* note for each PDF document, describing its contents at the level of its place in the whole 'Physical Computing' paper. *Evernote* allowed her to attach the PDF directly to the note, enabling the same two-level *sign-MC* strategy.

## Technology Affordances of DDEs

To recap, our theoretical framework used for analysis is that digital objects may be materialized through DDEs into manifest objects, and that these MOs may be appropriated through a process of objectification to serve as material carriers that support thinking. We extracted 50 thinking episodes that related to GoogleDocs documents, Evernote notes, and PDF documents. Our DO-path analysis traced the three 'focus' MOs in terms of thinking processes, TOs, DOs, MOs, technologies used, relevant quotes relating to affordances and experience, and the occurrence of an MO-MC binding (coded as hindrance or support).

The second part of the DO-path analysis, of which a sample is shown in Figure 8, coded the quotes extracted in the first part of the analysis for objectification indicators and 'support' (in the moment or/and in the long-term) or 'lacking' characteristics of the technology configurations used. Indicators included for

example referencing materials as thoughts, reports of pointing to objects as ideas, and use of space as an organizing structure. Figure 9 shows the distribution of thinking episodes in which the MO was able to be coded as either

<table><caption>Figure 8 Objectification indicators and Technology affordances from DO-path analysis</caption><thead><tr><th>Thinking Process</th><th>Technology Configuration</th><th>Objectification Indications</th><th>Support Characteristics</th><th>Lacking Characteristics</th></tr></thead><tbody><tr><td>Concept formation/Outlining</td><td>• A laptop with EVN hooked to a large display</td><td>• Reference to points listed in Evernote</td><td>• Common ground<br>• Simultaneous focus</td><td>• None</td></tr><tr><td>Brainstorming/Concept formation</td><td>• Three co-located personal laptops, each with EVN</td><td>• None</td><td>• None</td><td>• Simultaneous focus<br>• Common ground</td></tr><tr><td>Concept formation</td><td>• iMac with EVN<br>• One personal laptop</td><td>• Gesturing and pointing to picture in Evernote</td><td>• Spatiality<br>• Accessibility<br>• Immediacy</td><td>• None</td></tr><tr><td>Presenting ideas</td><td>• Tablet with EVN</td><td>• Reference to points in Evernote while presenting</td><td>• Transparency<br>• Immediacy<br>• Persistence</td><td>• None</td></tr><tr><td>Sharing information</td><td>• Tablet with EVN</td><td>• None</td><td>• None</td><td>• Transparency<br>• Consistency<br>• Expectation of Interaction</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 9 Role of Manifest Objects in thinking</figcaption></figure>

having a support or hindrance role in relation to the students' thinking processes. The synthesis of the technology affordances that we found supported, were lacking, or hindered objectification to occur is presented in Figure 10. The features were derived from quote-supported storylines like Storyline 1 and 2 above.

Storyline 1, for example, shows evidence of the creation of mental *signs* as part of the reasoning workflow, and how *objectification* may have occurred by which MOs,
materialized from the
PDFs, served as MCs to
support thinking. The
*persistence* of the ‘bags
of finds’ in the laptop
browser allowed the
participants to think of
these finds as ‘stuff I
found’ without necessari-
ly forcing them to a

premature commitment to meaning or interpretation. The mobility of the iPads supported opportunistic action so that the bags of finds and the named PDFs could be used to support thinking. The *transparency* of action between searching, renaming, and reading allowed for “a complete experience”.

In Storyline 2, all the participants regarded the entire PDF as the DO related to a particular TO at the level of the organization of the entire paper being written. The participants all employed a strategy of encapsulating this level of conceptualization of the paper with a very short description that served as the MO. This allowed the participant to bring perception into the process of thinking by using the MO to serve as an MC. The extent at which this *objectification* process occurred however was highly dependent on the *transparency* of interaction to annotate/associate meta-data to the high-level description of the TO, and the *persistence* of the display of the MO. The *immediacy* of being able to quickly drill down to the content of the PDF document as well supported the unwrapping of successive layers of abstraction and encapsulation.

<figure><img src="image_10.png" /><figcaption>Figure 10 Technology affordances for objectification</figcaption></figure>

# 6 Discussion

We organize the discussion of our work under three themes: the *objectification* framework as a method for the study of technology ecologies; the experiences of students in *DDEs*; and technology affordances to aid in the design of *DDEs*.

## An Objectification Framework for Thinking in DDEs

Digital information is ‘magical’. We can duplicate, copy, summon to visual presentation, search, etc. almost at will. Yet, despite this malleability of pixels on screen, we often revert to physical, material things to help us make sense of information because they more readily help us to objectify thought. Starting from Vygotsky’s sign media tion theory, we proposed that *thought objects* can be related to *digital objects* that can be *materialized* in DDEs as *manifest objects*. Through *objectification*, the student appropriates the MO to serve as a *material carrier*, thereby bringing the MO into the loop of thinking. Our DO-path analysis, as presented in the two example storylines, showed the promise of our *objectification* framework, and affordances in the *DDE* that appears to support *objectification*. We successfully traced the components of our model in different situations in our data transcripts to elucidate the thinking process.
Other similar ‘path analysis’ methods have been proposed before for the study of technology use, notably in works that argue for the importance of context in HCI. For example, in ‘activity theory’ (AT), the unit of analysis is human activity, mediated by the use of artifacts or tools. One traces the subject, object, actions and operations in an analysis approach using AT [32, 33]. In a similar vein, situated action models identify the “activity of persons-acting in setting” as the focus of study. Using the approach, one traces the relations between individual and environment at a “very fine-grained level” as the human engages in dynamic uses of artifacts [32]. Distributed cognition (Dcog) looks at the “cognition of a system in terms of its function”. It requires one to trace “representation [states] inside and outside the head” as the central unit of analysis [34]. Our method of tracing DOs does not exclude artifacts (devices and displays), individuals (students), or their activities (thinking processes). Adopting a different focus, it provides a way instead to make sense of the flow of information, literally ‘food for thought’, from the environment to the mind of the individual (or vice-versa).

## Experiencing the DDEs

With currently available technologies, students already build ecologies with their personal devices in daily life. As highlighted by Jung et al. [14], “every newly designed interactive artifact will inevitably become a part of someone’s ecology”. The question is however whether it will be an *effective* part of his/her ecology. We found many situations when the student was faced with ‘gaps’ in his/her DDE, which hampered the smooth flow of his/her thinking. He/she then had to find other technologies that could bridge the ‘gaps’. In this sense, the DDEs that one constructs today are ‘add hoc ecologies’: the different components do not function as a synergistic whole by design, but places the burden of synchronization and file control on the user [4].

Furthermore, we found that the use of the individual devices was mostly characterized by what we call the ‘portal thinking’ effect. We posit that many current interfaces and devices are designed based on the Cartesian view that external things and thinking processes are mutually exclusive. The result may be that the screen then acts only as a conduit through which information is summoned. With computational advancements (e.g. larger screens, mobile systems) we may have multiple ‘portals’, but they remain data straws to draw information to view. It is unfortunate thus that the potential of interface, visual and physical components of devices to be MCs for thinking often fails to be harnessed, and instead, technology is used as mere input, output or ‘projection’ devices.

## Designing DDEs for A Learning Experience

We believe that there is a lack of designed support to optimize technology’s role of augmenting humans’ higher thought. From our findings, we proposed technology affordances that appears to help the process of *objectification*. These can be classified into seven groups, although they are all interrelated at certain levels. Below, we discuss the central theme of each group, while providing considerations for system design features that can potentially embody each theme. It must be noted however that
our proposition of technology affordances in *DDEs* for learning does not imply that one can design such a complete ecosystem for learning. Variations in individual learning styles and the diversity of subject matter to be learned would preclude such rigidly constructed ecosystems. Drawing insight from the inveterate paper ecology that has supported learning since Gutenberg’s printing press, we suggest that learning ecologies, be they physical or digital, have to remain ad-hoc but with designed support. The paper ecology, that includes writing implements, tables, bookshelves and libraries, books, and the myriad paper clips, staplers, rulers, and folders, has evolved over more than half a millennium and provides a vast set of components that are flexible enough for the individual to appropriate them to construct his/her environment for specific learning experiences. The digital ecology as well, needs to evolve technologies that possess affordances but yet are malleable enough to support learning.

**Affordance 1: Expectation of interaction (includes transparency, shareability):**
It is key that components of *DDEs* are able to not only interoperate in some way, but also provide an expectation of interaction to users. Our writing implements for example can be expected to write with few impediments on paper. Of course, the paper ecology is constrained by materials and physical laws (e.g., we do not have to worry about book 2.0 falling through the surface of table 3.1 because of incompatibilities) while all interactions in *DDEs* have to be designed and implemented. Also, the cultural longevity of paper has built expectations and constraints (e.g., pencils do not work on leather portfolios) into the user community that digital technologies cannot always rely on. In our study for instance, the students reported that they decided to use the whiteboard for brainstorming particularly because they knew that they would be able to take a picture of it with their *iPad* cameras later on. Conversely, a clear example of failure of this aspect in our study is one instance where the students spent one entire meeting only to set up shared *Dropbox* folders and *Evernote* notebooks. Work in middleware and system interoperability, as well as direct interaction methods, are important to enable this affordance of ‘expectation of interaction’ to take place.

**Affordance 2: Immediacy (includes persistence, reference, situatedness, accessibility, multitasking, opportunistic action):** *Immediacy* concerns features that allow the user to display, manipulate, and use DOs across devices without going through one or more indirect actions. For example, if a user has a physical paper that she wants a friend to read, she drops it on the table in front of him. However, if she had the document on her laptop, she may have to put it into *Dropbox* and tell him where to get it before they can discuss its contents. The lack of *immediacy* in this scenario hinders the *opportunistic* use of the document as a focus of discussion or thought. *Immediacy* is closely associated with *transparency*. Transparent interoperability across platforms supports immediate action as do *consistency* of interaction techniques (consistent ways to move and manipulate MOs across platforms is critical to support their use as MCs). In our study, participants used *persistence* to allow information to stay immediate and more easily participate in their thinking: on the laptop, they aligned their *Word* document and the PDF papers side by side. Others used their *iPad* as a ‘persistent’ secondary display for the PDFs.

**Affordance 3: Concreteness (includes dedicated use, spatiality, tangibility):**
Components that possess the characteristic of ‘concreteness’ in a form appears to
better support *objectification*. In our study we saw that the whiteboard that affords
the use of space and the iPad that opens applications using the full screen real estate,
for example, aid information to become what Heidegger [35] calls a ‘thing’, some-
thing tangible for the student to grasp in her thinking process. A technological envi-
ronment that requires the user to hold thoughts in mind while looking for the appro-
priate device to record them offers little support for objectification. Work on embod-
ied interaction, physicality, and tangible user interfaces that address how to enable the
user to easily assign thoughts to concrete materials contribute to this affordance.

**Affordance 4: Iconicity (includes structure, mnemonicity, atomization, customization):** Objects to which we have assigned meaning become more easily internalized than neutral objects. We have seen the use of file renaming and organization of folder structure as instances when *iconicity* has enabled MOs to be used as MCs of thought. The few works in HCI that have looked at familiarity of interfaces are potent informants of this affordance. *Atomization* is a related feature that supports the association of DOs to TOs. A TO is typically an atomic idea at some level of abstraction, in the same way that a ‘unit of analysis’ specifies an entity that is a coherent whole at a certain level. Take the idea of convolution in our earlier example. A Wikipedia page on convolution would be an apropos DO for the concept, but a whole book on signal processing would not (even if it contains a section on convolution). Features to support *atomization* (e.g., bookmark individual components of larger text documents) have been investigated for example in hypermedia research.

**Affordance 5: Common ground (includes simultaneous focus and control):** To be able to focus on thinking in groupwork, the technology should provide support for students to easily create shared MCs. A common MO may not necessarily lead to the same mental *signs* for two different people. This is essentially the question of inter-subjectivity [36]. Physical things inherently allows for several users to have simultaneous focus and control. A page on the table can be seen by everyone around the table; several users can write on the whiteboard at the same time. In digital technologies however, the students always needed a separate ‘situating channel’ (e.g. speech, instant messaging, comment lines) to set common ground together with the ‘information channel’, where work is carried out. This factor also appeared to have prompted the more intensive use of *GoogleDocs*, which has a simultaneous editing capability, over *Evernote* for notetaking. Work in computer-supported collaborative would be relevant to further inform the design of this affordance into technologies.

# 7 Conclusion

This paper contributes significantly to the area of technology ecologies, focusing on
how they may help us to think and to learn. We identified a gap in the literature of the
plethora of technology ecology/ecosystem concepts previously proposed in terms of
investigating learning. Second, we extend Vygotsky's sign mediation theory to the
digital world by proposing how *thought objects* may relate to *digital objects*, and their
manifestation through technology ecologies. Third, we propose a way through the
concept of *objectification* and method of DO-path analysis to examine device ecolo-
gies for their potential for learning. And fourth, we articulate affordances that can
inform either design guidelines or evaluation metrics for the development of *device
and display ecologies* that allow us to think effectively.

One limitation of our study may be that it was conducted with computer science students who may have had a different perspective on technology. Researchers from other domains like history may require additional support to piece together fragmentary information. Such extension can be a rich domain for future research. Although we do not claim to propose a holistic explanatory theory of cognition, we do believe that the *objectification* framework manages to uncover basic thinking processes irrespective of the student’s domain expertise. Knowledge discovery is foundational to any form of learning. In a future study, we plan to employ our model to make sense of students’ in-situ speech on top of their self-reported experiences of learning using technology. In the face of changing perspectives, HCI researchers should begin investigating and designing technologies that help us think, or in our Star Trek tale, that make the *doctor* ‘real’.

8 Acknowledgements

This project has been supported by NSF grant No. IIS 1059389.

References

1. Strate, L., Studying Media As Media: McLuhan and the Media Ecology Approach. MediaTropes, 2008. I: p. 127-142.

2. Altheide, D., An Ecology of Communication: Toward a Mapping of the Effective Environment. The Sociological Quarterly, 1994. 35(4): p. 665-683.

3. Nardi, B. and V. O'Day, Information Ecologies: Using Technology with Heart. First Monday, 1999. 4(5).

4. Tungare, M., et al., Personal Information Ecosystems and Implications for Design, in Computing Research Repository (CoRR) cs/06120812006.

5. Krippendorff, K., The Semantic Turn: A New Foundation for Design 2005: CRC Press. 368.

6. Rick, J. Towards a classroom ecology of devices: Interfaces for collaborative scripts. in Wksp at CSCL 2009: "Scripted vs. Free CS collaboration: alternatives and paths for adaptable and flexible CS scripted collaboration". 2009. Rhodes, Greece.

7. Gibson, J.J., The ecological approach to visual perception 1979, Lawrence Erlbaum Associates.

8. Suchman, L.A., Human-Machine Reconfigurations: Plans and Situated Actions 2007. Cambridge U. Press.

9. Norman, D., The Psychology of Everyday Things 1988, New York: Basic Books.

10. Huang, E., E. Mynatt, and J. Trimble, Displays in the Wild: Understanding the Dynamics and Evolution of a Display Ecology. Pervasive, 2006. p. 321-336.

11. Enquist, H., K. Tollmar, and A. Corry, Interaction Ecologies, in Wksp at Pervasive Computing 2007.

12. Dearman, D. and J. Pierce. "It's on my other Comp-uter!": Computing with Multiple Devices. CHI 2008.
13. Forlizzi, J., The Product Ecology: Understanding Social Product Use and Supporting Design Culture. International Journal of Design, 2008. 2(1): p. 11-20.

14. Jung, H., et al., Toward a Framework for Ecologies of Artifacts: How are Digital Artifacts Interconnected within a Personal Life?, in NordiCHI 2008. p. 201-210.

15. Coughlan, T., et al., The Conceptual Framing, Design and Evaluation of Device Ecologies for Collaborative Activities. IJHCS, 2012. 70(10): p. 765-779.

16. Bailey, D. and S. Barley, *Teaching-Learning Ecologies: Mapping the Environment to Structure Through Action*. Organization Science, 2011. 22(1): p. 262-285.

17. Loke, S. and S. Ling. Analyzing Observable Behaviours of Device Ecology Workflows. in 6th International Conf. Enterprise Information Systems. 2004.

18. Pyla, P., et al., Continuous User Interfaces for Seamless Task Migration. HCI, 2009. p. 77-85.

19. Pierce, J. and J. Nichols. An Infrastructure for Extending Applications' User Experiences Across Multiple Personal Devices. in UIST 2008. ACM.

20. Chu Yew Yee, S., et al. The Physicality of Technological Devices in Education: Building a digital experience for learning. in ICALT 2012.

21. Hollan, J., E. Hutchins, and D. Kirsh, Distributed Cognition: Toward a New Foundation for HCI Research. TOCHI, 2000. 7(2): p. 174-196.

22. Vygotsky, L.S., Mind in society: The development of higher psychological processes, ed. M. Cole, et al. 1978, Harvard U. Press.

23. Vygotsky, L.S., ed. Thought and Language, (Edited and translated by E. Hanfmann and G. Vakar) ed. A. Kozulin 1987, MIT Press: Cambridge.

24. McNeill, D., Gesture and Thought 2005, Chicago: University of Chicago Press.

25. Dourish, P., Intentionality, in Where the Action Is: The Foundations of Embodied Interaction 2004, MIT Press: Boston, MA. p. 134-138.

26. Patterson, M. and D. Williams, Collecting and Analyzing Qualitative Data: Hermeneutic Principles, Methods, and Case Examples. Advances in Tourism Applications 2002: Sagamore Publishing.

27. Tesch, R., Qualitative Research: Analysis Types and Software Tools 1990, New York: Falmer Press.

28. Verdie, Y., B. Fang, and F. Quek, Mirrortrack: tracking with reflection - comparison with top-down approach, in ICMI-MLMI 2009. p. 247-250.

29. Lewis, J.R., IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. IJHCI, 1995. 7(1): p. 57-78.

30. Barker, R.G., The Stream of Behavior 1963, New York: Appleton-Century-Crofts.

31. Card, S. and P. Piroli, Sensemaking Processes of Intelligence Analysts and Possible Leverage Points as Identified Through Cognitive Task Analysis. in Int'l Conf. on Intelligence Analysis. 2005.

32. Nardi, B. Studying context: A comparison of activity theory, situated action models and distributed cognition. in East-West Conf. on HCI 1992. St. Petersburg, Russia.

33. Kaptelinin, V., B. Nardi, and C. Macaulay, The Activity Checklist: A Tool for Representing the "Space" of Context, in Interactions 1999. p. 27-39.

34. Halverson, C.A., Activity Theory and Distributed Cognition: Or What Does CSCW Need to Do with Theories? CSCW, 2002. 11: p. 243-267.

35. Heidegger, M., Being and Time, ed. J. Macquarrie 1962, New York, NY: Harper & Row Publishers.

36. Gillespie, A. and F. Cornish, Intersubjectivity: Toward a dialogical analysis. J. Theory of Social Behaviour, 2010. 40: p. 19-46.