# Finding-NEVO: Toward Radical Design in HCI

Sharon Lynn Chu, Francis Quek, Yao Wang, Rex Hartson

► To cite this version:

Sharon Lynn Chu, Francis Quek, Yao Wang, Rex Hartson. Finding-NEVO: Toward Radical Design in HCI. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.471-478, 10.1007/978-3-642-40483-2_33. hal-01497455

HAL Id: hal-01497455

https://inria.hal.science/hal-01497455

Submitted on 28 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Finding-NEVO: toward radical design in HCI

Sharon Lynn Chu, Francis Quek, Yao Wang and Rex Hartson

Center for Human-Computer Interaction,
Virginia Tech

{sharilyn, quek, yxw5167}@vt.edu; rex.hartson@gmail.com

**Abstract.** We address the methodology of design-oriented research in HCI, whereby researchers design and implement technology to test concepts. The task is to produce a testable prototype (that we call *NEVO*, *Non-Embarrassing Version One*) that faithfully embodies the concept. We probed leading HCI researchers and CHI authors about the challenge of *Finding NEVO*. We found uncertainty on how to design prototypes that allow for both design and scientific contributions. We propose the *Finding-NEVO* model that articulates a process yielding prototypes that are faithful to the rationale and idea being studied. We conclude by discussing our theoretical and methodological contributions.

**Keywords:** Radical design, design method, innovation, HCI

## 1 Introduction

HCI is an interdisciplinary field with a foot each in the doors of science and design [1-3], causing an inherent tension since “[s]cience and design have different principal objectives” [3]. While design values innovation and is generative in nature [4], science builds on prior advances in verifiable steps. Approaches like *theory-based design (TbD)*, *design-oriented research (DoR)*, and *research through design (RtD)* have been proposed to resolve this conflict. In *DoR* [2], design and technology implementations serve to test and validate the research concepts. Similarly, *TbD* [3] and *RtD* [5] see the value of design in the light of artifacts embodying some form of knowledge. Conversely the design of commercial products focuses on design practice, or solving problems and real-world obstacles, that Fallman calls *research-oriented design (RoD)* [2]. Our research looks at process in the context of the DoR tradition in HCI.

To properly answer a research question, DoR prototypes have to embody the ideas to be tested. Yet, there is little concrete guidance on how to design such research prototypes. Our goal is to gain an understanding of prominent design processes in HCI through interviews with HCI researchers and domain experts, and to suggest a plausible process model for the design of prototypes in research.

## 2 Design of Research Prototypes

We employed a general model (Figure 1), as a probe for interviews, of how research prototypes are designed. This model is similar to Gaver’s [4] idea of designs
occupying points in the design space or creating a design space around themselves. The HCI researcher advances a concept from some *conceptual rationale* (label A), and produces a *seed idea* (B). The *conceptual rationale* may be a theoretical construct, a new technology, a research intuition, or a user need, and the *seed idea* is an instantiation of how the theoretical construct may be embodied in a system. This *seed idea* is situated in the *space of possible designs* of the domain in question (C), which may be populated by a set of *existing* designs (D). If the *conceptual rationale* is novel, and the *design space* is not densely populated, the new *seed idea* would likely be far from *existing designs*. We call such a design a *potentially radical design* (E) because of this difference from existing solutions. The *seed idea* does not fully specify a testable system. Many *pathways* of design choices or decisions have to be made to develop a system (F). Some of these design choices flow directly from the *seed idea*, and some may be necessary to realize the system but do not find guidance from the *seed idea*. All of these design choices have potential impacts on the testing of the *conceptual rationale* of the research. The term *NEVO* (*Non-Embarrassing Version One*) designates the first testable prototype that faithfully embodies a concept.

There is an inherent tension, that we seek to address, when realizing the *design pathways* (F). This tension can be characterized by different types of ‘creeps’ (G) that threaten fidelity to the original conceptual rationale. Prior experience with existing systems by designers, developers, and study participants introduces forces to move the design to resemble existing solutions. We call this *experience creep*. Also, our tendency to resort to the most convenient way for implementation and design using tools and frameworks at hand can introduce *convenience creep*.

## 3 Contextualizing our Work

**Design of Research Prototypes:** There are few well-defined, systematic process methods to develop good research prototypes. We briefly

describe two frameworks that are particularly relevant to the problem of finding *NEVO*. Carroll's & Kellogg's [3] *theory-based design (TbD)* proposes that successful HCI designs embody psychological claims in contexts of use, and advances that claims coalescing together in the implementation of a system projects “a model world”, “a believable illusion” to the user. The careful recording of this ‘design rationale’ can inform the design of future artifacts. Keyson and Alonso's [6] *Empirical Research through Design method* embeds interaction design hypotheses into working prototypes to contribute to design knowledge by creating experimental variability. The method lists several broad guidelines (e.g., “prototype variability has to be carefully defined so as not to confound the research question at hand”), and specifies the use of design iteration and techniques, but presents no clear process model.

**Process for Radical Design:** Work on radical innovations tends to come overwhelmingly from the management sciences. Many either emphasize the importance of the individual with passion and vision, the “human side” (e.g. teamwork, networks,

<figure><img src="image_2.png" /><figcaption>**Figure 1. Going from concept to testable prototype**</figcaption></figure>
roles [7]) and organizational mechanisms (e.g. rewards, management policies), or
advocate the approach of concept refinement through the involvement of users (e.g.
[8, 9]). Management science is concerned with the potential of a product to be adopt-
ed by users, thus defining radical products "in the sense that they imply changes in
consumers' everyday lives" [9]. In contrast, our concept of radicalness in HCI re-
search is that it is valuable because it is an extension to knowledge in and of itself.

Closely aligned with Figure 1, a ‘hill-climbing’ model was proposed by Norman
[10]. He characterizes the design space for a particular domain as being occupied by
multi-dimensional ‘design hills’. A particular design sits on the slope of a hill, and
ascending the hill constitutes a design improvement. Human-centered design (HCD)
approaches allow designers to climb a particular hill by an iterative incremental pro-
cess of design improvements driven by lessons learned from user studies. Norman
states that while HCD enables a design to ascend its hill, it cannot move the design to
another hill with a higher peak. Such a jump to a different hill would constitute a radi-
cal design shift. HCD is thus “only suited for incremental innovation”, to improve
existing products according to the user’s contextual needs. Norman & Verganti [10]
argue that potentially radical *seed ideas* have to be driven by technology or meaning
change. For radical innovation, they suggest that HCD must admit the “simultaneous
development of multiple ideas and prototypes” (essentially many *seeds*).

4 Study Methodology & Data Analysis

We carried out hour-long semi-structured interviews with two sets of HCI people: 1. *Meta-Interviewees* (MIs) and 2. *System Builders* (SBs). The MIs are leading researchers from prominent HCI research labs throughout the world who have done substantial conceptual or theoretical work in design methodology. The SBs, researchers who develop technology using the DoR approach, were selected by combing through CHI papers published in the last decade. The selection, done by three members of our team, identified papers that 1. Adopt an approach where a system is built to investigate a research question; 2. Have a recognizable *seed idea* and an explicit or implied design process; and, 3. Present some form of user testing. The selection process yielded 97 papers across HCI domains (e.g. accessibility, web search, ubiquitous computing, embodiment, social interactions, 3D interaction).

An email request for interview was sent to the first authors of all the papers. If we
did not receive a reply, a request was sent to the second authors of the papers. Three
local researchers (2 SBs and 1 MI) were used as pilot interviewees. Twenty-two HCI
researchers, 20 from university research labs and two from industry were interviewed
as SBs. Eight leading HCI researchers were interviewed as MIs. Five of the eight MIs
also did the SB interview. Our interviewees were from the US, Canada, the UK, Ja-
pan, Singapore, South Korea, and Australia. Eighteen participants were interviewed
through video teleconferencing, two by telephone, and two in face-to-face sessions.
Oral consent for audio recording was first obtained from each interviewee.

The SB interview guide comprised 32 questions (multiple-choice survey and rat-
ing, and open-ended questions) on four main themes: 1. *Description of the research*
undertaken specifically in the CHI paper; 2. Methods used to design the system; 3. Testing of the system; 4. Design in HCI in general. The MIs' interview was divided into two sections: 1. The first six questions probed on research of the MI in general, and on her perspective on design in HCI. 2. We explained the purpose of our study using Figure 1. Nine questions guided the discussion and prompted the interviewees about design processes to move from an idea to a research prototype and the role of user testing in radical design. We asked the permission of some MIs to quote their responses non-anonymously, and they were given the opportunity to review the paper.

All interviews were transcribed and coded for themes of importance in several iter-
ative passes. The research process used by each interviewee was identified. A prelim-
inary process model was constructed through a first high-level analysis. We refined
our model in subsequent iterations through deeper-level analyses and discussion of
the insights gleaned. Survey data collected in the interviews were collated on a
spreadsheet and summarized. These helped us to get a sense of how each component
of our model was manifested in each project. We present a few relevant study find-
ings below in a synthesized account of the qualitative and survey data.

5 Findings

The basis used by most of our SB in-
terviewees to make decisions on what to
include in a design was their own intui-
tion and experience with their original
conceptual rationale, or by generating a
list of design requirements based on
studies of users. Figure 2 shows the dif-
ferent categories of how researchers made design choices.

However, none of our SBs could provide a reliable method of how to interpret
study results (e.g. unexpected study results could be because the original design idea
is unsuitable for the particular user group or the system was improperly implement-
ed). All interviewees who provided an answer said that an *iterative design* process
was needed, and some said that using *low-fidelity prototypes* for user studies could
enable the researcher to make an informed guess “based on the reaction we got from
people”. Elements of *intuition and experience* were also put forward by some MIs as
key to making judgments about study results.

All MIs agreed that there is currently no process to guide researchers on how to
embody a *seed idea* in a *design*, resulting in many ineffective designs. The MIs’ per-
spectives can be summarized into six ‘high-level strategies’, all pointing to the uncer-
tainty inherent in the ad-hoc way in which HCI research is conducted: **1. One needs to**
*try “existing solutions first so that one doesn’t innovate unnecessarily. When you fail with existing solutions, that’s when you innovate and fly.” (Scott Klemmer, Stanford U.)*
**2. One should “stick” to one’s idea and persist: “You simply have to face your own ideas and go forth and most of the time you will fail. But the few times you succeed make up for the failures.” (Norman, Norman Nielsen Group)**
**3. One should gen-

<figure><img src="image_3.png" /><figcaption>Figure 2. Basis for design decisions</figcaption></figure>
erate many designs and test often: “The principle is to try to do this (minimal prototyping) with little effort as possible. So you can try a lot of designs” (Jacob, Tufts U.). **4.** One should make use of intuition, skills and experience. But for Norman, “this is where the huge experiences and skills and intuitions of the designers come in. Is that a method? No.” David Frohlich (U. of Surrey) contrasted “inspiration for design” and “requirements for design”. In the former, one may “generate ten times more designs” than the latter which employs UCD to produce just one or two from the requirements. Frohlich states that intuition is the only way to bridge these two approaches. **5.** One should be pragmatic and be flexible to adopt any method that makes sense without limiting oneself to a “fixed methodology”. **6.** One should persist based on faith. Norman suggested that the only evidence that the researcher needs to move forward with her idea is to see (from test results) whether at least “one or two people…really believe deeply in what you are doing”.

**6 Proposed Model**

Through iterative modeling of current methods we extracted in our interview data, we derived a model (Figure 3) of how design decisions (Label F in Figure 1) may be made to better arrive at NEVO. As in Figure 1, the radical design process begins with a *conceptual rationale* in the form of an integrated *Research Aim, Motivation or Rationale* from which a *seed idea* (A) is generated. A set of *idea-defining characteristics* is produced (B) to guide the design process to further specify the original concept. Exhaustive articulation of these *characteristics*, however, is not always possible. More importantly, over-specification of *idea-defining characteristics* may even have the undesirable side effect of over-constraining creativity in the design process. The model addresses this by recognizing the need for an informal *Team Understanding and Consensus* (B). Together, the *Idea-Defining Characteristics* and the *Team Understanding and Consensus* embody the *seed idea*, which then guides the *Gatekeeper* pro-

<figure><img src="image_4.png" /><figcaption>**Figure 3. Finding-NEVO model**</figcaption></figure>
cess. The *Gatekeeper* process (D) is used to vet individual *design ideas* in advance of
development. These *design ideas* (C) may be the product of methods such as brain-
storming and ideation, with respect to a particular design issue (which may be identi-
fied through methods like ethnographic studies, design principles, experience or sim-
ple intuition).

Our key contribution is in the insertion of a formalized ‘evaluation’ at block D into
the accretion process of the *seed idea* into a full design. We do not prescribe any hard
and fast way to implement the *gatekeeper*. This could be consensus among a group of
stakeholders, the judgment of a single individual, or discussion in the research team.
Each *design idea* may be determined by the *Gatekeeper* process to be: *positive* (it is in
line with, supports or even extends the specified *seed idea*) or *negative* (it violates the
premises of the *seed idea*). In the event of a negative judgment, either an *acceptable
compromise* is found (e.g., owing to implementation and other pragmatic reasons such
as time limitations, a modified version of the design idea may be judged the best al-
ternative), or if the violation is so severe as to contradict the original research question
and no compromise can rectify this, then the design idea is rejected.

Additionally, the *gatekeeper* may decide that the *design idea* is orthogonal to the *seed idea*. Here, a second choice needs to be made. In DoR, an unnecessary design component can introduce confounds into the study results. If the design idea is a necessary aspect of the system, the process proceeds to development, if not, it is rejected. The *prototyping process* (E) is typically done together with *iterative user testing*. As discussed earlier, this is an incremental process, that in Norman's parlance, climbs the design hill on which the *seed idea* is planted [10]. The result of this process is a system (*NEVO*) that can be used to answer the original research questions in formal user studies (F).

# 7 Discussion

Our model is distinct from previously proposed models in a number of ways. TbD
argues for the use of psychological theories to inform the design of artifacts; the main
aim is to produce new artifacts. In contrast, the purpose of *Finding-NEVO* is to reveal
knowledge and understanding; it uses design in the service of research. Moreover,
while the focus of TbD is on repeatability (i.e., being able to analyze, understand and
maybe reproduce artifacts), ours is on validity (i.e., how to stay true to one's idea
when testing it through the use of artifacts). TbD then can be considered to be a *de-
sign* method (others being ideation, contextual design, etc.) that can be used in Box C
in our model to generate design ideas.

Much of our model is grounded in similar assertions as those of Norman's 'hill-
climbing' model. However, we believe that even if the *seed idea* is potentially radical
(through meaning change, technology push or random tinkering) and sits at the base
of a higher 'hill', it may or may not develop to the top of the 'hill' using incremental
HCD methods because of creeps (*convenience, experience, feature and user input
creeps* described before). More importantly, for these same reasons, the design may
jump back to some other 'hill' of a more mature but ultimately limited design trajec-
tory. Our approach ensures that a potentially radical *seed idea* is allowed to blossom.
To be clear, our model is not inconsistent with Norman & Verganti's position. We simply suggest an additional *gatekeeper* process that may make radical designs more likely without necessarily posing HCD as irrelevant to radical innovation.

Interestingly, the five MIs who were also interviewed as SBs gave differing perspectives on HCD depending on their roles. As MIs, they were candid concerning the inadequacies of HCD in bringing about radical designs needed for *DoR*, but as SBs, they employed and justified their approaches within an HCD framework. We posit two possible reasons. First, the field is in need of new models and methodological warrants to break out of the HCD paradigm. Second, as SBs/researchers they needed to publish their work, and employing HCD may make their work more acceptable to reviewers. Both interpretations suggest that the field of HCI needs to be evaluated its dependence on the HCD paradigm. This paper contributes by adding to this discourse.

Beside the contributions above, this paper enriches our understanding of HCI as a scientific domain, and contributes to research and design methodologies in HCI:

**Contributing to HCI as a field:** HCI is a relatively young field that is still being defined. Grudin asked the question “Is HCI homeless?” [11]. Much discussion has occurred on how to come to terms with the interdisciplinary nature of HCI (e.g. [12]). Our model provides a way for different disciplines to be integrated in one HCI research project. The *conceptual rationale* and *seed idea* (Boxes A and B) can come from any domain (e.g. social sciences/humanities). Methods from the creative disciplines (e.g. art/design) are employed in Box C, *design thinking*, to generate ideas for a prototype. If technical implementation is desired, engineering knowledge is required at Box E for prototype development, testing and integration. And finally research methods from the sciences are needed to carry the project through at Box F. HCI has also found it challenging to bridge many ‘gaps’ [13]. As we mentioned in the introduction, the conflict between the needs of science and those of design is very evident in HCI, and yet HCI is touted as being “the science of design” [14]. We believe this paper facilitates a better reconciliation of the two.

**Contributing to HCI methodology:** *Finding-NEVO* is a formalized process model based on current methodological practices of researchers and refined to enable HCI research to be done more transparently. The inclusion of a gatekeeping procedure not only serves to infuse systematicity into the research process, but also provides a roadmap for new researchers to the field to know how to proceed. Even among our interviewees, many found it difficult to articulate clearly the steps that they took in their research project. Some stated that “it was messy”. Moreover, our approach can provide a common mental model of research and common terminology among project team members with differing backgrounds and perspectives. Broadly, the core contributions of *Finding-NEVO* is not only in the methodological systematicity it allows, but also in the common basis of understanding that it can provide to the field.

Gaver [4] lamented that the *RtD* approach may be seen as unscientific since the criterion of falsifiability cannot be applied: the “synthetic nature of design is incompatible with the controlled experiments useful for theory testing”. The danger that threatens the scientific validity of the approach is precisely that design is generative [4]. However, setting up a constant gatekeeper in the system development process enables one to stay ‘on track’ with regards to the initial *conceptual rationale*. I.e., two re-
searchers starting out with the same *rationale* may not end up with the same systems,
but following our model, there is greater chance for the systems to be truthful embod-
iments of the original idea, leading to similar results in final scientific user tests.

8 Conclusion

The *Finding-NEVO* model for *DoR* articulates a methodology to select design ide-
as that yield prototypes that are faithful to a *conceptual rationale* and *seed idea*. To
the degree that the *rationale* and *seed* are novel, the method is likely to produce radi-
cal designs that may impact design for the marketplace. Our model is constructed
through an analysis of current practice to understand both the applicability and fail-
ings of current methodology. In doing so, we contribute both theoretically and meth-
odologically to HCI. Future work includes the validation of the *Finding-NEVO* model
across various HCI projects through longitudinal adoption.

References

1. Fallman, D., *Design-oriented Human-Computer Interaction*, in CHI. 2003, p. 225-232.

2. Fallman, D., *Why Research-Oriented Design Isn't Design-Oriented Research: On the Tensions Between Design and Research in an Implicit Design Discipline*. Knowledge, Technology & Policy, 2007. 20: p. 193-200.

3. Carroll, J.M. and W.A. Kellogg. *Artifact as Theory-Nexus: Hermeneutics meets Theory-Based Design*. in CHI 1989.

4. Gaver, W., *What Should We Expect From Research Through Design?*, in CHI. 2012, p. 937-946.

5. Zimmerman, J., E. Stolterman, and J. Forlizzi. *An Analysis and Critique of Research through Design: Towards a Formalization of a Research Approach*. in DIS. 2010.

6. Keyson, D.V. and M. Bruns. *Empirical research through design*. in In'l Assoc. of Soc. of Design Res. Conf. (IASDR'09). 2009.

7. O'Connor, G.C. and C.M. McDermott, *The Human Side of Radical Innovation*. Journal of Engineering and Technology Management, 2004. 21: p. 11-30.

8. Lettl, C., C. Herstatt, and H.G. Gemuenden, *Users' Contributions to Radical Innovation: Evidence from Four Cases in the Field of Medical Equipment Technology*. R&D Management, 2006. 36(3): p. 251-272.

9. Heiskanen, E., et al. *User involvement in radical innovation: are consumers conservative?*
European Journal of Innovation Management, 2007. 10(4): p. 489-509.

10. Norman, D. and R. Verganti, *Incremental and Radical Innovation: Design Research Versus Technology and Meaning Change*. Design Issues, 2012.

11. Grudin, J., *Is HCI Homeless? In Search of Inter-Disciplinary Status*, in Interactions. 2006, p. 54-59.

12. Carroll, J.M., *HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science*. 2003, San Francisco, CA: Morgan Kaufmann.

13. Lazar, J., J. Feng, and H. Hochheiser, *Inherent Conflicts in HCI*, in *Research Methods in Human-Computer Interaction*. 2010, John Wiley & Sons. p. 9-15.

14. Carroll, J.M., *Human-computer Interaction: Psychology as a Science of Design*. Int. J. of Human-Computer Studies, 1997. 46(4): p. 501-522.