Modeling Cognitive Processes in Design from the Designer’s Perspective
Towards a computational model for implementing intentional capacities in digital systems○ Yuval KAHLON *1 and Haruyuki FUJII*2*1 Assistant Professor, Tokyo Institute of Technology, D.Eng.
*2 Professor, Tokyo Institute of Technology, D.Eng.
Summary: Inquiry into design cognition is important for the progress of artificial intelligence. This research aims to model a wellknown cognitive phenomenon often referred to as “seeing as”, which plays an important role in designing. In short, seeing as
consists of the ability of a human subject to attribute various meanings to the same object. This phenomenon is tightly linked
with the cognitive process of interpretation - a major component of human design processes. Accordingly, its elucidation is
expected to help implement human-like design capabilities within computational systems. In this study, we collect and analyze
empirical data from a design task, developed to enable us to closely observe of this phenomenon. We then present an approachfor systematically modeling seeing as, which serves as a first step toward its implementation within computational design
systems. Our model integrates state-of-the-art frameworks for design cognition with the highly important notion of
intentionality from the philosophy of mind. As such, it serves as an invitation for constructing a bridge between these rather
disparate disciplines. By applying our model, we suggest implications for studying design cognition phenomena, towards theconstruction of artificial agents which can design in a human-like manner.
Keywords: Design science; intelligent systems; intentionality; architectural design.
1. Introduction

potential directions for realizing such capacities in digital design

Since design can be regarded as a highly intelligent humansystems in the future.

activity1), its study is important for the development of artificial
intelligence (AI). One field of design science which is relatively

2. Backgroundwell-developed is the study of human interpretation processes.

2.1. MODELING VISUAL INTERPRETATION IN DESIGN

Specifically, much research was dedicated for understanding how

Interpretation is a fundamental process in design3). In this work,architects interpret visual representations, as they engage in

we focus on a major aspect of human interpretation processes,

various architectural design tasks. As a key example, see thewhich is the phenomenon of “seeing as” (SA)4). Simply put,famous work by Schon and Wiggins2) , in which various kinds ofpeople can look at one thing and (somewhat intuitively) interpretseeing were explored, in the context of designing simple

it in various manners (Fig.1).

architectural layouts.
Past research has shed light on how designers give meaning tovisual objects, mainly by identifying and describing various
cognitive phenomena related with this activity. However, tomobilize these insights into AI, the knowledge gained from suchstudies should be both synthesized and formalized. This researchtakes a step forward in this direction, by attempting to construct

Figure 1. The phenomenon of SA: a set of yellow square-shaped

a computational model for design interpretation processes. To dowall tiles can be seen either as a downward pointing arrow or asso, we not only draw on past studies, but also on empirical data

a left pointing arrow.

gathered specifically for our goals.
In this work, we closely observe, document and analyze

One important role of SA in design is to enable designers tointerpretation activity, in the context of architectural design. A

escape situations of fixation, by envisioning design alternatives.

model for this activity is proposed, as a step towards

Consequently, it facilitates high flexibility in terms of thinking

implementation of human-like interpretation capabilities inand action, which characterizes human design processes. We aim

computational design systems. Our model is presented andto replicate this valuable human ability in digital design agents,

discussed. In this, we point out key limitations, as well as outline

as to facilitate a high degree of adaptivity, which is essential for

日本建築学会情報システム技術委員会
第 44 回情報･システム･利用･技術シンポジウム論文集， 215-220 ， 2021 年 12 月，京都
Proceedings of the 44th Symposium on Computer Technology of Information ，Systems and Applications ， AIJ ， 215-220 ， Dec. ， 2021 ， Kyoto－ 215 －

dealing with complex design tasks. To date, research on this topic

3. Methodology

mainly provided us with qualitative descriptions of SA, via
observation of human design processes2),4). However, taking the

3.1. OVERVIEW
We devised a simple design task that will enable observing

computational approach to mind, we believe that a deep

interpretation by human designers, as they engage in architectural

understanding of SA would require to go beyond observation anddesign (which is the dominant design domain utilized in existing

into modeling this practice, for its implementation in astudies of such phenomena, as previously mentioned). The

computationalaspecific context selected for this purpose is that of Japanese rockcomputational model of SA, by drawing on state of the art

garden (JRG) design, for several reasons: primarily, JRGs consist

frameworks in design cognition research.

of forms (rocks) which are meaningful in the sense that theysystem.

Accordingly,

westrivefor

In this work, we utilize Gero and Kannengiesser’s situatedencourage “Mitate ( 見 立 て )”, i.e. SA, in design (Fig. 2).

function-behavior-structure framework (situated FBS)5), as a

Additionally, their minimalistic nature enables deep observation

model for systematically describing human design processes. In

and analysis of this highly complex phenomenon, in a mannersituated FBS, cognitive activity in design is distributed acrosswhich is extremely difficult (in fact, potentially impossible) tothree worlds: 1) external world: in which things outside of theachieve in real-world design scenarios. The possibility of such adesigner are noticed; 2) interpreted world: in which objects are

deep inquiry is further facilitated by their emphasis on a single

assigned with meaning by the designer; 3) expected world: infunctional aspect (that of appreciating visual forms), which

which the designer imagines the desired state of the world in theserves to reduce the immense complexity involved infuture, to be achieved by designing.
As the situated FBS framework acknowledges the aspect ofsimultaneous consideration of multiple conflicting aspects andinterpretation, we find it useful for our purposes. However, thecontexts and tasks, such as designing residential units etc.).

requirements (characteristic of many other types of design

framework can be enriched via the inclusion of a central aspectof human cognition – that of intentionality (taken from thephilosophy of mind, see 2.2). This work takes a step in thisdirection of constructing a bridge between design cognition andintentionality, as a means for providing a systematic account of
how we engage in interpretation.
2.2. INTENTIONALITY
Simply put, all thought is directed towards something, be it
concrete or abstract. Such directedness towards objects isreferred in the philosophy of mind as our “intentional capacity”
or simply “intentionality”6). It is critical that any framework fordesign cognition will include this notion in some form, since it isa fundamental property of the human mind.
As explained by Searle, intentionality theory attempts toprovide an account of how the mind relates with the world.7) Two

Figure 2. SA in a Japanese garden; large rock representing a cow

concepts proposed by Searle are important here: “intentional

resting on grass; Nijojo, Kyoto (photo by Jiang Ming; with

content” (what one thinks about) and “direction of fit”. The latter

permission).

was introduced to emphasize the difference between mental
states like perceiving, in which we fit our thoughts to what exists
in the world outside of us (therefore, we say that perception has

3.2. TASK AND DATA COLLECTION
Since the cognitive ability of SA is not limited to design

the direction of fit of “world-to-mind”; WTM) and desiring, in

specialists1, we did not place strong emphasis on collecting data

which we wish to change the state of the world based on thespecifically from highly skilled designers, considering thecontents of our thoughts (thus having the direction of fit called

exploratory phase of this work. Ten architectural designers aged

“mind-to-world”; MTW). The above concepts are used in this22-30 were recruited for the study (male: 7, female: 3) in varying

study to model interpretation activity, by noting the mental

levels of skill and education. The minimal requirement was that

content held at certain moments in time by a human designer, andof a Bachelor’s in architecture or in a related design discipline.

how it relates with the world at that instant.

論文 R47

Each subject was requested to design a miniature rock garden－ 216 －

to his/her liking, while verbalizing one’s thoughts by designingthese, we drew insights for modeling the phenomenon underin a “think aloud” style. Several rocks in various forms wereconsideration, towards its elucidation and implementation within

available as design materials, and a small wooden tray with a bed

computational systems.

of white sand was provided as the “site” for the design. Eachsession was capped at one hour, where in several cases the subject
wished to continue and work on his/her designs for a short while

4. Results
Utterances in a single session amounted to ~2050 words on

beyond the allotted time. We captured: 1) audio data of subjects’

average. Surprisingly, while subjects were not instructed toutterances, and 2) video data of the physical state of the design,

employ SA, all sessions included at least one SA occurrence. To

as it developed throughout the session.

explain our proposed modeling approach, we present one such
example. A subject formed a composition of a person riding a

3.3. DATA PROCESSING AND ANALYSISAs a basis for analysis, data was processed as follows: 1) fullturtle, which he later associated with the famous Japanese fairy

transcriptions of utterances were prepared, including all(Urashima Taro) rides a turtle into a hidden kingdom found in theutterances both by the research subject and by the moderator; 2)

depths of the ocean.

tale of Urashima Taro (Fig. 4). In this fairytale, the protagonist

the video data was used for extracting images which represent

By examining this short segment we can readily identify

changes to the state of the physical model, as to form sequences

several consecutive events of SA. Reading the figure from top toof representative frames; 3) utterances which accompanied eachbottom, we notice that the designer first sees two similar rocks as

step were added next to the corresponding image, resulting in“twins” (1). Then he sees another rock as “a turtle” (2), leadingvisual protocols.

him to search for a “head” (3) and see another rock as “a turtle’s

To identify events of SA and extract interpretations assigned

foot” (4) etc. Breaking such interpretation processes down into

by designers to the design materials (rocks, sand etc.), we thenseries of small steps enables us to gradually trace the chain ofreviewed the visual protocols and marked places where SA

events which characterized the formation and elaboration of theevents occurred. Identification is done either based on explicit

SA that was employed in each case. In the following section, weutterances, in which similarity between a rock and a referent wasapply our model to this small segment, and demonstrate how

pointed out (e.g. “this rock looks like a tree”), or by metaphoricalclose observation can aid in developing a deep and systematicutterances with respect to how the designer views the rock (e.g.

understanding of the processes involved in SA, towards its“this rock is a mountain”). SA and non-SA events were thenmodeling and implementation in computational systems.

organized sequentially within the three worlds of situated FBS(Fig. 3). From these we derived the relevant entities which need
to be accounted for in order to replicate this process in acomputational design system.

Figure 3. Example for dissection into events using situated FBS.5)
The mapping method developed throughout the analysis wasused to create visual representations of the designer’s perspectiveat a given time, which enable to derive insights regarding
important events characterizing the process of SA. Based on

論文 R47

Figure 4. Example for an interpretive process driven by SA,
extracted from a design session of a novice research subject.
5. The proposed model
5.1. KEY TERMS AND DEFINITIONS
We outline key terms which serve as the basis for our modeling－ 217 －

approach, demonstrated in the following sub-section. First, we(m1) appears in the Mind. This mental image serves as a means

distinguish between two main design worlds (“World”, “Mind”)

for finding a rock that can serve as the “head”2. This brief searchaccording to the work by Searle (introduced in 2.2) and relate

episode is characterized by a two-sided direction of fit (f3, f4),them with the situated FBS framework (introduced in 2.1). We

i.e. both WTM and MTW (more in 6). Finally, a candidate rockthen introduce key entities which we view as essential forfor the head of the turtle is found (ob2), and consequently themodelling SA processes. Important definitions are given below

mental image of the imagined “head” fades away, as it is noin Table 1. Note that, in addition to these, the term “direction of

longer of use to the designer, at least for the time being.

fit” is used in accordance with the definition by Searle7),

This application of our model enables us to closely examine

preserving his basic distinction between “world-to-mind”

the components and processes involved in visual interpretation

direction and “mind-to-world” direction (WTM and MTW,

and ask essential questions regarding the nature of SA as aaccordingly; the reader may refer back to 2.2).

cognitive process. In the next section, we raise several suchquestions, and hypothesize regarding potential answers for these,
based on our insights from this study thus far.

Table 1. Key terms and definitions.
Term

Definition

World

External reality, which is accessible to an
agent

throughhissenses;generallycorresponds with the external world in
situated FBS.
Mind

The totality of internal mental content at any
moment (containing both the interpreted andexpected worlds in situated FBS).

State

A time duration in which change is notobserved.

Concept

A linguistic (symbolic) expression which
identifies & classifies an object.

Interpretation

Meaning given by an agent to an object,
given as a symbolic linguistic expression.

MentalNon-symbolic mental content held by an

image

agent at a certain moment in time as means
for search.

5.2. APPLYING OUR MODEL
We apply our model on the short segment previously presented
in Fig. 4. Specifically, we focus on the second and third steps, inwhich the designer notices that a rock can be seen as a turtle, andthen finds another rock to represent the turtle’s head. This process
is visualized in Figure 5, which should be read from top to bottom.
Table 2 serves as complementary to this figure, and summarizes
the values associated with each entity in the selected segment.
As a first step (state 1) an object (ob1) in the World is noticed(or1) and recognized as a rock (c1). Next (state 2) the rock is said
to be seen as a “turtle” (I1). However, later we discover that theintended meaning was that it looks like a turtle’s body or its shell.
Therefore, the actual interpretation is such that the rock is seen
as a “turtle’s body” (i1; an interpretation which remains implicit
yet can be safely inferred). This interpretation is projected onto
the real rock (f2). Following this the subject is “looking for the
turtle’s head” (state 3). In this case, we see that a mental image

論文 R47

Figure 5. Close examination of events in SA using our model.
Table 2. Values associated with the entities in Figure 5.
Entity Associated value
c1“rock”
I1“like a turtle”
i1
turtle’s body
f1
perception
f2
projection (MTW)X
future “turtle’s head”－ 218 －

m1
f3
f4

search image; “turtle’s head”
evaluation in search (MTW)
perception in search (WTM)

6. Discussion
6.1 IMPLICATIONS FOR MODELING SA
When examining the above visualization, the following fact isstriking – there seems to be no necessary connection3 between
states 1 and 2. In state 1 the designer picked up a rock, and instate 2 he declared it looked “like a turtle”. The key question that
we should raise here is this: why did the designer engage in SA
in state 2? Did he consciously decide to do so, or was driven tothis by a trigger of some sort (visual/haptic/other)? These
questions are highly important, since we cannot construct a
computational design agent which interprets objects in a humanlike manner if we do not know how behavior is regulated when
designing. In other words, we need to know how (or when) do
designers decide to utilize one cognitive strategy (SA) rather thananother one (for instance, focusing on the aesthetic dimension
without interpreting the rock using any referent).
Another important observation which can be made byexamining our visualization, is the following - state 2 has led tothe production of a mental image which directed (if not governed)
the action in state 3. This mental image, which specified how a“turtle's head” should be represented is unique to the subject inseveral ways: first, it depends on his past knowledge, which is inturn derived from his personal background etc. Second, and more
obviously, it cannot be experienced by anyone else, i.e. it ispurely subjective. The fact that such subjective mental images(which are inaccessible to direct scientific inquiry to date) drive
action means that we can never realistically model humandesigning without providing an explanatory model for
subjectivity, and the ways in which it shapes thinking and action.
Therefore, we must develop methods for externalizing suchmental images so that they can be studied and modeled ascomputational processes as well.
Finally, extending the previous point, notice that the activity offinding the desired rock using the mental image can neither becharacterized solely as a WTM or MTW process. Instead, we seea mind-to-world-to-mind loop, as the designer not only imposes
the mental image on the world when looking for a rock which fitsit (f3), but also considers potential candidates in the world which
are not initially included in the mental image (f4). By jumping
back and forth between the desired object and the available ones,
the designer can locate a successful candidate to fill the design
requirement and make progress in the design process. This
exemplifies the potential of our model as a tool for making senseof complex processes involved with SA, via their description in

論文 R47

terms of simpler processes, which can be inquired into via
intentionality theory.
6.2. LIMITATIONS AND FUTURE WORKFirst, the current model deals with organizing and representing
the mental content of designers and its relation to external reality.
However, in practice, mental content is almost always associatedwith a certain type of mental state. Therefore, our account mustalso include the aspect of types of mental states, and the way inwhich they relate with mental content. One potential direction isto employ well-established models from AI, such as the beliefdesire-intention architecture, or simply BDI8).
Second, while currently we model SA by capturing key
components within symbolic terms, human cognition has a subsymbolic aspect as well9). For example, when our subject uttered
the word “turtle” in SA, the word was further associated withmental images which cannot be reduced to any single linguistic
or symbolic term. Such aspects may be integrated into the modelby adding a computational layer to serve as grounding forsymbolic terms, facilitating their assignment and manipulation.
To make progress in this direction, we may employ recent
methods in AI, which enable us to capture patterns without
explicit representation, and associate them with single symbolic
terms in the memory (e.g., by using supervised learning methods).
Further, note that in Searle’s theory we do not only have mental
states with a type and content, but also a “background” which
they operate against. The difficulty with integrating this idea into
the work lies in the fact that, for Searle, the background is “nonrepresentational”. However, recently it was proposed by Schmitz
that, rather than non-representational, it may be better to regard
it as non-conceptual10). In this manner, and by drawing on theidea of including a sub-symbolic layer mentioned above, thebackground may be subsumed into future models forcomputational design agents. We believe that striving in thisdirection would require modeling not only mental content but
also additional aspects of the agent, such as embodiment, social
environment, etc.
Lastly, it should be made explicit that the current model does
not consider the aspect of functionality, which is central todesigning. As previously noted, we have resolved to use JRGs forthis study in order to simplify the highly complex process ofinterpretation. JRGs enabled to achieve this, at the cost ofoversimplifying the aspect of functionality (as one of theirprimary functions is a representational one; i.e. function is almost
identical with interpretation). To further develop the model andintegrate this important aspect, we intend to examine such design
activity in the context of Chinese traditional gardens, as these not
only include rocks, but also enable the visitor to interact with－ 219 －

them (by touching, climbing etc.).
6. Conclusion
An approach for modeling SA, which is a key cognitive ability indesign, was proposed and demonstrated. Our findings’
importance is twofold: 1) it was demonstrated that certain aspectsof SA can be described using symbolic terms, which may serveas an explanation of how animals retrieve desired objects11).
3. The idea of necessary connection has long been a challenging
topic for philosophical debate. Hume has even argued that it ismerely an illusion, which stems for the frequent observation ofseeming regularity in the world12). For all practical purposes
however, we can establish that a necessary connection exists
when causal relations can be assumed with a high degree ofconfidence. For example, between the sun rising and the fact that
there is light.

as a basis for future computational implementations; 2) the
analysis conducted using our method also points to the necessityof developing methods for studying subjective aspects ofcognitive phenomena in design, as an essential step towards
modeling for human design processes. In the absence of suchmethods, any model which attempts to explain human behavior
in design, and specifically the important ability to engage in SA,
will remain partial at best. The proposed model serves as astarting point in this journey, in two key-aspects: in identifying
and elaborating the components of SA processes, and in aiding toexpose gaps in our current understanding. As such, it can be used
to construct a road map for achieving human-level interpretation
capabilities in AI systems which target design tasks, and direct
our inquiry as we move into this uncharted territory.
Acknowledgements
We thank our research subjects for taking time out of their busy schedules
to participate in this study.
Endnotes
1. In fact, in a previous experiment we have witnessed children as
young as eight naturally engage in SA, when presenting them witha set of rocks in various forms. These children did not have any
educational background in design.
2. This image can be considered as a “search image” – a term
proposed in the thought-provoking work by Jakob von Uexküll,

論文 R47

References
1) Kalay YE.: 2004, Architecture’s new media: Principles, theories,
and methods of computer-aided design, MIT press, Cambridge.
2) Schön D, Wiggins G.: 1992, Kinds of seeing and their functions
in designing, Design Studies, 13(2), 135–56.
3) Goldschmidt G.: 1988, Interpretation: its role in architecturaldesigning, Design Studies, 9(4), 235–45.
4) Goldschmidt G.: 1991, The Dialectics of Sketching, Creativity
Research Journal, 4(2), 123–43.
5) Gero J, Kannengiesser U.: 2004, The situated function–
behaviour–structure framework, Design Studies, 25(4), 373–91.
6) Crane TT.: 2003, The Intentional Structure of Consciousness, in
A. Jokic and Q. Smith (eds.), Consciousness: New Philosophical
Perspectives, Oxford University Press, New York, 1-27.
7) Searle J.: 2004, Mind: A Brief Introduction, Oxford University
Press, New York.
8) Müller J.: 1996, The Design of Intelligent Agents: A Layered
Approach, Springer-Verlag Berlin Heidelberg, New York.
9) Fujii H.: MAAI: 2019, Dynamic Equilibrium in Architecture. New
Generation Computing, Springer Nature, 37(3), 285–305.
10) Schmitz M.: 2012, The Background as Intentional, Conscious, andNonconceptual, in Radman Z. (eds), Knowing without Thinking:
New Directions in Philosophy and Cognitive Science, Palgrave
Macmillan, London, 57-82.
11) Von Uexküll J.: 1934, A Foray into the Worlds of Animals andHumans, University of Minnesota Press, Minneapolis.
12) Hume D.: 1896, A Treatise of Human Nature, Selby-Bigge LA(eds), Clarendon Press, Oxford.

－ 220 －