# Designing Interactive Applications to Support Novel Activities

Authors: Hyowon Lee; Nazlena Mohamad Ali; Lynda Hardman
Journal: Advances in Human-Computer Interaction
 (2013)

---

## Abstract

R&D in media-related technologies including multimedia, information retrieval, computer vision, and the semantic web is experimenting on a variety of computational tools that, if sufficiently matured, could support many novel activities that are not practiced today. Interactive technology demonstration systems produced typically at the end of their projects show great potential for taking advantage of technological possibilities. These demo systems or “demonstrators” are, even if crude or farfetched, a significant manifestation of the technologists’ visions in transforming emerging technologies into novel usage scenarios and applications. In this paper, we reflect on design processes and crucial design decisions made while designing some successful, web-based interactive demonstrators developed by the authors. We identify methodological issues in applying today’s requirement-driven usability engineering method to designing this type of novel applications and solicit a clearer distinction between designing mainstream applications and designing novel applications. More solution-oriented approaches leveraging design thinking are required, and more pragmatic evaluation criteria is needed that assess the role of the system in exploiting the technological possibilities to provoke further brainstorming and discussion. Such an approach will support a more efficient channelling of the technology-to-application transformation which are becoming increasingly crucial in today’s context of rich technological possibilities.


---

## Body

## 1. Introduction

Technological advancements are at an unprecedented pace. Supported by ever-increasing computing power, storage capacity, network infrastructure, and scalability and further fuelled by the general public’s awareness of technology and their increasing willingness to try new services and the consequent marketing opportunities, many technology research laboratories around the world are fiercely investigating and experimenting on technological possibilities as never before. Multimedia, computer vision, information retrieval, artificial intelligence, and language technology are some examples of computational technology fields that are leading this advancement boom, promising a high-impact outcome that will shape the way we interact with technology as well as how we interact with each other in the coming years.In dominantly technically focused projects that have long been developed in these fields, the end of a project often sees a “demonstrator” or “demo system” that showcases possible end-user interactivity with the developed piece of technology. For example, the VideOlympics [1] is an annual event in the multimedia research community to showcase and promote various outcomes of video retrieval research, where the researchers bring to the event their interactive systems and demonstrators that incorporate cutting-edge video retrieval techniques and algorithms. These demonstrators highlight interactivity in which an end user might engage to search, browse, and be entertained by video content in ways that have not been tried before. The technological possibilities witnessed in this event are extremely intriguing and inspirational, although often they display low-quality interaction design and poor usability due to the reasons that will be addressed later in this paper.Exemplified by these technology demonstration systems arenovel applications—applications that are new and have no existing user base or usage practice today. Opposed to these are the mainstream or conventional applications—applications that support existing practices with existing groups of users, for example, word processing software, library management systems, museum kiosks, and online travel websites. Most of the software industry today is naturally geared towards developing conventional applications to support their current customers’ needs.Very successful in incrementally refining the details of user-interfaces, theuser centred design approach that focuses on fully understanding and documenting the target users and their work practices into detailed requirements before designing the system is one of the most significant contributions of the human-computer interaction (HCI) discipline with its provision of methods, tools, and procedures to today’s software industry. This requirements-driven design methodology works well when developing mainstream applications because it aims to make the system fit as closely as possible to their target users’ practices and contexts.Conversely, applications created in technology research labs as demonstrators and exploratory proof-of-concept prototypes have quite a different purpose to their creation. Novel applications are developed to demonstrate technological possibilities and how they might manifest in user interactivity, help inspire the research community, foster brainstorming and discussion exploring more possibilities from the demonstration, and ground and guide future research directions to more feasible scenarios. Though this type of novel application development effort is an important investment for the future, little understanding of designing for these purposes exists today. With the conventional wisdom of understanding end users and their contexts in the mainstream application development as amply emphasised by HCI and practiced in the industry today, many technology research groups try to adopt user-centred and requirements-driven design methods to guide the development of their demonstrator systems. However, they tend to result in wasted resources due to (i) insufficient information sources from the beginning, notably the absence of user base, and the lack of usage practice and (ii) insufficient know-how and the lack of methodological support for converting the project’s technological agenda into effective end-user features.Given the immense potential of these demonstration systems in shedding light on how the technological advancements could shape our future interactivity with technology, a proper facilitation of suitable tools and methods in place to guide the design process for these novel systems becomes a significant issue.This paper presents a reflective analysis on some recently completed technology projects. Based on the decade-long, first-hand experience in designing novel demonstrators and applications that incorporate various emerging technological tools, the reflections in this paper focus on how some of the crucial design decisions were made and how they steered the design process, especially in shaping the unidentified user needs and trying to satisfy them. The contributions of this paper are:(i)
to highlight the existence of the on-going work in designing novel applications in technology-related R&D laboratories and to emphasise the significance of this line of applications in helping shape people’s interactivity with technologies in the near future;(ii)
to inform the current situation which many technology R&D communities are facing where the usability engineering approach in the requirements-centred tradition of HCI design methodology they take does not effectively support designing novel applications;(iii)
to characterise the design process and evaluation criteria suitable for this line of applications by reflecting on past projects that resulted in successful novel application development.Through these contributions, we hope that the activity of designing technology demonstrators will gain the attention that they deserve, a suitable design methodology will be instrumented for technology R&D sectors, and eventually, the pace of innovation in these sectors in turning the emerging technological advancements into feasible and usable application scenarios will accelerate.


## 2. Designing Novel Application: Three Examples

Within the past 12 years, the authors of this paper have designed over 50 novel interactive systems incorporating some of the advanced and emerging computational technologies, 33 of which resulted in concrete and complete user-interaction strategies, 21 of them were formally user-tested, and 8 of them were deployed and used by people over time. In this section, three examples from them were chosen to illustrate the ways in which novel systems are designed. The examples chosen represent the novel technology systems with 3 quite different degrees of novelty in terms of supporting what people do or do not do today and how much these are expected to change the way people engage in the activities in the near future, that is, from a less novel (shifting the sequence of tasks that people used to do) to a somewhat novel (adding an additional task to enhance the value of what people used to do) to a highly novel (supporting a completely new activity that people have not done before). The chosen examples in this section are on purpose limited to the applications on the same interaction platform, that is, web-based desktop PC applications, in order to reduce the possible discussion points on the affordances of different interaction modalities (and the design decisions to be made for these), so as to help focus on the comparative design decisions made for the respective examples. Through these examples, characteristics of novel application design projects will be highlighted and how their design process differs from a more conventional design process.


### 2.1. “My Friends’ Faces”—Exploiting Face Recognition

This project aimed to develop an application that leveragesautomatic face detection and recognition techniques, which have been popular topics in computer vision for many years. Face detection typically uses a combination of visual features (e.g., colour, shape, and texture) and a classifier trained on example faces in photos to determine the existence of a face in a new photo. Face recognition then uses the face detection output to calculate the visual similarity between the detected faces to establish whether two faces belong to the same person. If one detected face had been initially labelled with a person’s name, then all other detected faces with the similarity level above a certain threshold can be labelled with the same name. Such techniques can, if accurately performed, considerably reduce the user’s manual annotation burden when the number of photos to deal with is large.The ideation of the application started with the usage context of personal photo management, but it particularly focused on exploiting the face detection and recognition techniques to highlight their power and to show how such techniques might be used to provide new ways of photo consumption that have not been featured before.One obvious strategy that arose early in the design process was to plug in the face detection and recognition techniques at the back end of any typical personal photo management service such as Flickr or Picasa. When a user visits the website and browses photos as normal, she notices that the photos have already been annotated or tagged with people’s names. In this case, the face detection and recognition techniques are there simply toautomate the manual annotation task without requiring any major interaction shift or change in design on the front end. Many automatic media indexing tools currently being researched and developed have the potential to be used in this way, as they try to automate what human users or indexers had to do conventionally.Given our focus on emphasising incorporated techniques and providing a feature that had not been tried before, we reexamined the overall idea of a conventional, Flickr-like photo browsing interaction that typically starts with a user selecting a group of photos organised by events, date, and time and then selecting a photo in an event to view an enlarged photo. This led us to the assumption that the need for browsing their photos stems from people’s desire to see their friends or family members captured in various situations, and the photos themselves are merely a means of doing it.We changed the interaction sequence as follows: a user first browses a “face index,” which is a list of thumbnail-size faces (face icons) of people who regularly appear in the user’s collection (see the left side of Figure1). The face icons in this list are fully automatically selected and cropped by the system from various photos in the user’s collection. Beside each face icon, the name of that person and the frequency of that person’s appearance in the collection are indicated. The face index is initially sorted by appearance frequency in the user’s collection. Selecting a face icon will retrieve all photos and events that contain that particular person in increasing order of the number of people present in the photos. Figure 1 shows that the user selected “Georgina” in the face index and clicked the “Go” button at the bottom. The result is a panel of photos in the middle of the screen, starting with 1-person photos of Georgina, before proceeding to 2-person photos including Georgina and so on. When the user clicks on any of the retrieved photos, an enlarged version of the photo with its detailed information will appear similar to other conventional personal photo management services.Figure 1
Primary photo access point is the “face index” as fully automatically prepared by the system.The interaction difference between this and other conventional photoware might seem subtle, but the novelty of this interaction is the automatically prepared face index and appearance frequencyas the primary access pointandstarting point of interaction.The user interface has undergone a dozen refinement iterations with informal testing, a cognitive walkthrough, discussions, and brainstorming throughout the design process, with overall layout and sequencing, individual features and the look and feel modified and refined accordingly. At the end of this project, a formal user evaluation was conducted with 4 test users to identify usability problems and ascertain overall opinions and comments on the application [2]. Among many useful findings, the face index was overall perceived as only “slightly useful,” partly due to some missing faces and incorrectly annotated names. When the main premise of the application (i.e., face detection and recognition) does not perform perfectly, the user interface that exploits that feature will reveal such an inaccuracy and the trust of the system by the users decreases immediately. In addition, having used popular photo services such as Flickr and Picasa for many years, our test users naturally wanted interaction styles similar to those.The application was considered a state-of-the-art system in terms of demonstrating the face recognition technology that revised the conventional photoware interaction to take advantage of the technology and served as a prop in brainstorming for new projects.


### 2.2. “Mo Músaem Fíorúil” (My Virtual Museum)—Exploiting Object Matching

Identifying objects in images and videos and then comparing the visual characteristics among the identified objects are on-going research topics in the field of multimedia. The seed idea for this application was to exploit an object matching algorithm called the scale-invariant feature transform (SIFT) [3], which uses multiple visual keypoints around the outlines of objects in two images and can determine whether they belong to the same object, even when the two objects show different angles and scales. Before a series of brainstorming sessions and discussions commenced, we had a vague notion of a museum usage scenario where a user visits a museum and takes many photos of museum artefacts and wants to find detailed information about the artefacts after returning home. We developed this notion into a detailed interaction sequence for this application.We incorporated the SIFT technique’s capability in two sequential ways during the user interaction: (1) the system automatically groups the users’ photos by unique artefacts, and (2) it matches each group of photos to an authoritative artefact photo from the museum. In the designed scenario, the user comes home and uploads all the photos to the web-based application, whereupon the uploaded photos are presented in groups of unique artefacts. In Figure2, a total of 11 photos uploaded are shown automatically grouped into 5 rows (on the left column), with each row representing a unique artefact.Figure 2
User’s photos are automatically grouped by unique artefacts and then matched to authoritative photos with the relevant information presented.The accuracy of the automatic grouping by unique artefacts depends on the performance of the SIFT technique. If the technique did not perform correctly, the user can manually move a misplaced photo from one group into another simply by drag-and-drop action, whereupon the back-end SIFT technique will recalculate and improve the performance for the next stage of interaction in real time.The user can then select a group, and the most similar artefact candidates from the museum’s archive are presented (5 items with circular green icons in the middle column in Figure2). Here again, the technique’s accuracy becomes a design issue: if the SIFT performance was 100% accurate, the stage of presenting the top 5 candidates would have been unnecessary and selecting one group of photos could immediately show the details of that unique artefact matched from the museum’s database. Selecting one candidate of the matched artefacts will then present detailed information about that particular artefact (right side of Figure 2). Blighe et al. [4] present more details on this application.There is no tool today that supports an interaction feature such as this, though there are many museum visitors who take photos of museum artefacts. Potential users, if interviewed, would not have expected or expressed such a feature because it was a purely technological possibility that facilitated such an application scenario. The design process was more geared towards properly incorporating the SIFT technique in the user interactivity given its imperfect performance, rather than incorporating users’ known behaviours and needs.Though born of technical inspiration with no substantial end-user engagement, the application has become a valuable demonstrator within the multimedia research community as well as more application-focused field such as human-computer interaction that effectively highlighted a technical possibility resulting in a novel usage scenario.


### 2.3. “My Visual Diary”—Exploiting Event Detection from Lifelog Photos

Visual and continuous lifelogging is an active research area today, but no member of the general public has actually practised a lifelogging activity in any significant way. Using a passive photo capture device such as SenseCam or Vicon Revue (Vicon Revue, Memories for Life,http://viconrevue.com/), our day-to-day activities can be visually recorded on a continuous and long-term basis. Amounting to 2,000–3,000 photos a day (approximately 1 million photos a year, if captured every day), these devices pose a considerable challenge for accessing the captured photos in an easy and meaningful way.A conventional way to access them would be if, each day, the lifelog user could review that day’s photos by quickly slide showing all of the photos temporally, which typically takes 10–30 minutes depending on the speed of the slide show.Content-based image indexing techniques can automatically group a day’s photos by distinctive “events” that happened throughout the day, pick the most representative photos from each event, and determine which events were more important than others that day. While experimenting with various technical possibilities for automatically indexing and structuring months and years of SenseCam photos collected by some of our group members, a visual summarisation idea emerged.In Figure3, the main part of the screen presents 19 photos, each representing a major event that happened that day, as automatically chosen by the system. The photo size is proportional to the importance of that event, as calculated by the system by looking at all of the events within the past week and comparing the uniqueness of each event [5]. Using a packing algorithm to display images in a more condensed way [6], the system composes an intriguing comic book-style layout, visually emphasising the outcome of event importance calculation.Figure 3
Reviewing the day’s highlights and viewing the past events similar to a particular event from today.For any event presented, clicking on “Find Similar” button retrieves a group of past events that are visually similar to the selected event (in Figure3, the retrieved similar events are presented on the right side, all of which show the user chatting with a colleague in a lab environment).As the concept of lifelogging and visual reviewing of one’s day after continuous photo capture becomes more widespread and capture devices become cheaper, smaller, and more convenient to carry, people will start experimenting and engaging in such an activity in the near future. This application was created for that time, and it will be at that time when the application could be deployed and user-tested and its features and be refined to better fit to what people at that time will need and want. The application has served as an excellent brainstorming tool to obtain feedback, engage in further discussions, and guide future agendas for content-based indexing research, and the variations derived from this application have been developed for more specific target user groups, including a simplified touch-sensitive version for elderly users [7].


## 2.1. “My Friends’ Faces”—Exploiting Face Recognition

This project aimed to develop an application that leveragesautomatic face detection and recognition techniques, which have been popular topics in computer vision for many years. Face detection typically uses a combination of visual features (e.g., colour, shape, and texture) and a classifier trained on example faces in photos to determine the existence of a face in a new photo. Face recognition then uses the face detection output to calculate the visual similarity between the detected faces to establish whether two faces belong to the same person. If one detected face had been initially labelled with a person’s name, then all other detected faces with the similarity level above a certain threshold can be labelled with the same name. Such techniques can, if accurately performed, considerably reduce the user’s manual annotation burden when the number of photos to deal with is large.The ideation of the application started with the usage context of personal photo management, but it particularly focused on exploiting the face detection and recognition techniques to highlight their power and to show how such techniques might be used to provide new ways of photo consumption that have not been featured before.One obvious strategy that arose early in the design process was to plug in the face detection and recognition techniques at the back end of any typical personal photo management service such as Flickr or Picasa. When a user visits the website and browses photos as normal, she notices that the photos have already been annotated or tagged with people’s names. In this case, the face detection and recognition techniques are there simply toautomate the manual annotation task without requiring any major interaction shift or change in design on the front end. Many automatic media indexing tools currently being researched and developed have the potential to be used in this way, as they try to automate what human users or indexers had to do conventionally.Given our focus on emphasising incorporated techniques and providing a feature that had not been tried before, we reexamined the overall idea of a conventional, Flickr-like photo browsing interaction that typically starts with a user selecting a group of photos organised by events, date, and time and then selecting a photo in an event to view an enlarged photo. This led us to the assumption that the need for browsing their photos stems from people’s desire to see their friends or family members captured in various situations, and the photos themselves are merely a means of doing it.We changed the interaction sequence as follows: a user first browses a “face index,” which is a list of thumbnail-size faces (face icons) of people who regularly appear in the user’s collection (see the left side of Figure1). The face icons in this list are fully automatically selected and cropped by the system from various photos in the user’s collection. Beside each face icon, the name of that person and the frequency of that person’s appearance in the collection are indicated. The face index is initially sorted by appearance frequency in the user’s collection. Selecting a face icon will retrieve all photos and events that contain that particular person in increasing order of the number of people present in the photos. Figure 1 shows that the user selected “Georgina” in the face index and clicked the “Go” button at the bottom. The result is a panel of photos in the middle of the screen, starting with 1-person photos of Georgina, before proceeding to 2-person photos including Georgina and so on. When the user clicks on any of the retrieved photos, an enlarged version of the photo with its detailed information will appear similar to other conventional personal photo management services.Figure 1
Primary photo access point is the “face index” as fully automatically prepared by the system.The interaction difference between this and other conventional photoware might seem subtle, but the novelty of this interaction is the automatically prepared face index and appearance frequencyas the primary access pointandstarting point of interaction.The user interface has undergone a dozen refinement iterations with informal testing, a cognitive walkthrough, discussions, and brainstorming throughout the design process, with overall layout and sequencing, individual features and the look and feel modified and refined accordingly. At the end of this project, a formal user evaluation was conducted with 4 test users to identify usability problems and ascertain overall opinions and comments on the application [2]. Among many useful findings, the face index was overall perceived as only “slightly useful,” partly due to some missing faces and incorrectly annotated names. When the main premise of the application (i.e., face detection and recognition) does not perform perfectly, the user interface that exploits that feature will reveal such an inaccuracy and the trust of the system by the users decreases immediately. In addition, having used popular photo services such as Flickr and Picasa for many years, our test users naturally wanted interaction styles similar to those.The application was considered a state-of-the-art system in terms of demonstrating the face recognition technology that revised the conventional photoware interaction to take advantage of the technology and served as a prop in brainstorming for new projects.


## 2.2. “Mo Músaem Fíorúil” (My Virtual Museum)—Exploiting Object Matching

Identifying objects in images and videos and then comparing the visual characteristics among the identified objects are on-going research topics in the field of multimedia. The seed idea for this application was to exploit an object matching algorithm called the scale-invariant feature transform (SIFT) [3], which uses multiple visual keypoints around the outlines of objects in two images and can determine whether they belong to the same object, even when the two objects show different angles and scales. Before a series of brainstorming sessions and discussions commenced, we had a vague notion of a museum usage scenario where a user visits a museum and takes many photos of museum artefacts and wants to find detailed information about the artefacts after returning home. We developed this notion into a detailed interaction sequence for this application.We incorporated the SIFT technique’s capability in two sequential ways during the user interaction: (1) the system automatically groups the users’ photos by unique artefacts, and (2) it matches each group of photos to an authoritative artefact photo from the museum. In the designed scenario, the user comes home and uploads all the photos to the web-based application, whereupon the uploaded photos are presented in groups of unique artefacts. In Figure2, a total of 11 photos uploaded are shown automatically grouped into 5 rows (on the left column), with each row representing a unique artefact.Figure 2
User’s photos are automatically grouped by unique artefacts and then matched to authoritative photos with the relevant information presented.The accuracy of the automatic grouping by unique artefacts depends on the performance of the SIFT technique. If the technique did not perform correctly, the user can manually move a misplaced photo from one group into another simply by drag-and-drop action, whereupon the back-end SIFT technique will recalculate and improve the performance for the next stage of interaction in real time.The user can then select a group, and the most similar artefact candidates from the museum’s archive are presented (5 items with circular green icons in the middle column in Figure2). Here again, the technique’s accuracy becomes a design issue: if the SIFT performance was 100% accurate, the stage of presenting the top 5 candidates would have been unnecessary and selecting one group of photos could immediately show the details of that unique artefact matched from the museum’s database. Selecting one candidate of the matched artefacts will then present detailed information about that particular artefact (right side of Figure 2). Blighe et al. [4] present more details on this application.There is no tool today that supports an interaction feature such as this, though there are many museum visitors who take photos of museum artefacts. Potential users, if interviewed, would not have expected or expressed such a feature because it was a purely technological possibility that facilitated such an application scenario. The design process was more geared towards properly incorporating the SIFT technique in the user interactivity given its imperfect performance, rather than incorporating users’ known behaviours and needs.Though born of technical inspiration with no substantial end-user engagement, the application has become a valuable demonstrator within the multimedia research community as well as more application-focused field such as human-computer interaction that effectively highlighted a technical possibility resulting in a novel usage scenario.


## 2.3. “My Visual Diary”—Exploiting Event Detection from Lifelog Photos

Visual and continuous lifelogging is an active research area today, but no member of the general public has actually practised a lifelogging activity in any significant way. Using a passive photo capture device such as SenseCam or Vicon Revue (Vicon Revue, Memories for Life,http://viconrevue.com/), our day-to-day activities can be visually recorded on a continuous and long-term basis. Amounting to 2,000–3,000 photos a day (approximately 1 million photos a year, if captured every day), these devices pose a considerable challenge for accessing the captured photos in an easy and meaningful way.A conventional way to access them would be if, each day, the lifelog user could review that day’s photos by quickly slide showing all of the photos temporally, which typically takes 10–30 minutes depending on the speed of the slide show.Content-based image indexing techniques can automatically group a day’s photos by distinctive “events” that happened throughout the day, pick the most representative photos from each event, and determine which events were more important than others that day. While experimenting with various technical possibilities for automatically indexing and structuring months and years of SenseCam photos collected by some of our group members, a visual summarisation idea emerged.In Figure3, the main part of the screen presents 19 photos, each representing a major event that happened that day, as automatically chosen by the system. The photo size is proportional to the importance of that event, as calculated by the system by looking at all of the events within the past week and comparing the uniqueness of each event [5]. Using a packing algorithm to display images in a more condensed way [6], the system composes an intriguing comic book-style layout, visually emphasising the outcome of event importance calculation.Figure 3
Reviewing the day’s highlights and viewing the past events similar to a particular event from today.For any event presented, clicking on “Find Similar” button retrieves a group of past events that are visually similar to the selected event (in Figure3, the retrieved similar events are presented on the right side, all of which show the user chatting with a colleague in a lab environment).As the concept of lifelogging and visual reviewing of one’s day after continuous photo capture becomes more widespread and capture devices become cheaper, smaller, and more convenient to carry, people will start experimenting and engaging in such an activity in the near future. This application was created for that time, and it will be at that time when the application could be deployed and user-tested and its features and be refined to better fit to what people at that time will need and want. The application has served as an excellent brainstorming tool to obtain feedback, engage in further discussions, and guide future agendas for content-based indexing research, and the variations derived from this application have been developed for more specific target user groups, including a simplified touch-sensitive version for elderly users [7].


## 3. Discussion

The previously applications introduced have a number of design aspects in common.(i)
Their usage scenario and interactivity is novel in that the tasks that the applications support are not something people practice today.(ii)
Their design decisions were driven by the aim of exploiting particular technological tools rather than fitting the technology to specific user needs.(iii)
They were developed with a strong design discipline approach rather than a usability engineering approach when establishing solutions.(iv)
They are valuable for demonstrating technical possibilities and provoking further discussions on how they might be applied to create novel activities.These points will be discussed in more detail below.


### 3.1. Underlying Needs and Novelty

A novel application implies that there is anovel activity that the application can support or the type of activity that is new to people today. The intention of coming up with a novel application, then, is to invent a tool to create a new activity that people can start engaging in to benefit their lives in some way. For example, the activity of tweeting had not existed until the tool Twitter was created, and subsequently people started using it, and the value of using it became apparent. The activities of online video sharing and voting, blogging, social networking, mobile texting, internet shopping, or even TV watching are among the many examples of innovative systems or services that created novel activities (rather than supporting existing activities) at the time of their invention.At the early stage of the development of the three novel applications in Section2, establishing clear user needs was problematic because the premise of the projects was to exploit an emerging computational tool in a way that has not been used before. However, there were different degrees in which this problem of lacking the understanding of user needs affected the design process. In the case of My Friends’ Faces, once the role the back-end computational tool to be exploited (face detection and recognition) had been mapped to the novel feature of the iconised face index as the interaction starting point, the rest of the design process proceeded in a relatively straightforward manner. In the case of My Visual Diary, however, the uncertainty of user needs and operating the project under that uncertainty strongly influenced the design and evaluation throughout the project, making the undertone of the project quite different from My Friends’ Faces. For example, in analysing the user evaluation of My Friends’ Faces, it was possible to solicit the test users’ views and opinions on the novel feature in comparison to their prior experiences of using other more conventional photo management services because, even with the novel face index mechanism as the default interaction starting point, they were more or less agreeing on the overall purpose and the value of the system under testing; it was not the case for My Visual Diary.There are different degrees of novelty in the novel applications. Blogging, for example, was a novel activity when it first started, but one might argue that people’s inherent need for expressing themselves had existed and manifested in the forms of writing letters and keeping diaries, until the timely online tool came out and satisfied our needs more conveniently and with bigger impact (by being able to reach a wider audience); tweeting activities may be traced to the need for staying connected with people, hitherto satisfied with shorter blog entries or mobile texting, or face-to-face small talk; online video sharing may be traced back to physically posting a parcel of video tapes or CDs that contain family videos; lifelogging activities may be comparable to an extreme case of proactive photo-taking or obsessive note taking during special events. Depending on how novel the need supported by the invented application is compared to any known or existing need, we can draw a spectrum of novelty. Figure4 depicts some novel activities invented in approximately the last decade and located on the novelty spectrum by the degree of novelty at the time of invention (for they have now become mainstream activities and are thus not considered novel anymore).Figure 4
Novelty spectrum with example novel activities.For example, online shopping is a less novel activity because the need for shopping had been well known before any online shopping service was invented. Mobile texting is more novel than blogging because the need for instant messaging had not been explicitly known beforehand, while the need for diary keeping had been known.Also plotted in this spectrum are the three example applications from the previous section. In the case of My Friends’ Faces, though the overall activity of online photo browsing and sharing is not novel today, the way the application invites the user to browse the list of faces (instead of a group of photos) as the starting point of interaction is the novel aspect of this application afforded by the fully automatic face detection and recognition techniques. More novel than this is a museum visitor being able to browse his/her photos to gain detailed information about the artefacts. While people do visit museums and take photos of artefacts today, the need for uploading them to a system to organise them by unique artefacts and learn more details was unknown and most likely nonexistent. My Visual Diary, for visual lifelog users, is much more novel, as the need for lifelogging and reviewing one’s day is not a familiar concept as a “need” by the general public today.In some cases, especially those on the left side of the spectrum in Figure4, we may be able to create novel activities by observing people’s existing activities and identifying the inherent needs of those activities before trying to link those needs with recent or emerging technical possibilities. Here, proxy activities and proxy users could be used to identify useful information as the basis for inventing more novel activities and tools to support them (e.g., studying active photo bloggers to inform the design of a novel photoware system). Conversely, for those activities that are more novel (those on the right side of the spectrum), observation of people’s existing activities is likely to have less benefit because what is to be created can greatly differ from what people need and practise today. In this sense, those activities on the left side of the novelty spectrum tend to have a greater chance of benefiting from user studies (including interviews and observing people’s behaviour) than those activities on the right side, when creating new applications to support them.The tension between creating novel products and conventional ones has certainly been a reality in industry and a topic of interest in academia, witnessed from the early most advanced yet acceptable (MAYA) principle by the well-known industrial designer Raymond Loewy (Raymond Loewy, The Father of Industrial Design,http://www.raymondloewy.com/) to offer a novel solution but acceptable to current consumers, to a more recent view on how to interpret the phenomenon of incremental versus radical innovations [8]. In a series of consumer studies, Hekkert et al. [9] found that consumers preferred novel designs but not so much novel as to hinder the typical use of it, implying a careful balancing act required by a designer between novelty and typicality. Similar findings are reported in a study of chair design where the correlation between the level of novelty and aesthetic preference by people was observed as an inverted-U curve [10], implying a moderate level of novelty to be perceived best. More clearly categorising design activities by the purpose of design also comes into the picture: “design exploration” to find out what is possible  and what would be desirable or ideal, as opposed to “design practice” to build a system that satisfies a specific group of users in a specific context [11]. These different kinds of design activities suggested can be seen as targeting to create applications at different points on the novelty spectrum. The transfer scenarios [12, 13] try to “combine the best of both worlds” of invention (idea generation) and inquiry (study of people) by studying the people with marginal or unusual practices and transferring the main features discovered to a different domain in order to design novel systems or services. Similarly, “matchmaking” method [14] tries to match the already-known, existing, and intended technology to a specific work domain that is not yet clearly defined or established.Because it is difficult to suddenly come up with a novel activity or novel application that supports such an activity, there have been attempts to explore the methodological solutions to somehow balance between technologically oriented invention (right side of the spectrum) and grounded user study (left side of the spectrum). For example, a framework to align “blue-sky” research with real user interests [15], a playful participatory design to inspire new ideas [16], speed dating method to rapidly and cheaply compare design opportunities well before any prototyping effort is undertaken [17], “technology probe” [18] and other application studies that followed, a participatory envisioning and enactment to make the design-evaluation feedback cycle as dynamic and frequent as possible [19], and recent guidelines on when and how to use such a user enactment [20] are the examples in which the aspects of both users and technology were utilised in trying to come up with new applications on varying points on the novelty spectrum. Each of the methods as mentioned here will be better at creating novel applications on a different range or different points on the novelty spectrum. For example, those methods that involve participatory design approach will more likely end up creating novel applications on the left side, rather than right side, of the spectrum.The novelty spectrum illustrated here is far from objective or accurate, but one of the aspects it highlights is that designing a novel interactive system can range from switching around the initial information access point (My Friends’ Faces) to providing a value-added activity in conjunction with an existing activity (My Virtual Museum) to providing a completely novel activity that might currently be considered obsessive or even ridiculous (My Visual Diary). The spectrum can be used to roughly gauge any group of system or application development in terms of their level of novelty and to guide the kind of approach, tools, and overall mindset required to optimally engage in such development. For example, most of the demonstrator systems showcased at the recent Video Browser Showdown (Video Browser Showdown, at the 19th International Conference on Multimedia Modeling, 7–9 January 2013, available online athttp://mmm2013.org/Video_browser_showdown.htm) where all sorts of novel interactive video search systems had an open competition at a conference venue fall into the left side of the spectrum where the end-user need had been more or less clear from the beginning (quickly search for video clips) although the search features in the systems exploiting the computer vision and information retrieval techniques made them novel and innovative. On the other hand, many prototype systems exhibited at the laboratories such as MIT Media Lab and Interaction Research Studio at the University of London Goldsmiths try to experiment with new affordances that emerging technologies allow even when the result is not what people today have experience about (e.g., “History Tablecloth” [21] shows the trail of objects that had been on the table but moved away later. Nobody actually voiced that such a feature would be useful in our daily lives, but by being able to show such a history of where things had been on the table before, they are trying to discover or create a completely new, potentially beneficial use and need). We can locate this group of prototypes on the right side of the spectrum. Locating any of the developing systems by different research groups or communities on the spectrum provides a rough indication of their stance or agenda in exploring novel possibilities and the kind of weight they place between the knowledge on the end-user needs and the use of technology.Another aspect the spectrum highlights is that, in general, the less novel is an application, the more useful it is in today’s context and vice versa. As we strive to explore more novel applications (the right side of the spectrum), they will be obviously viewed as less useful in today’s context because of the unknown nature of the need and activity that the application tries to support. Judging the value of a designed application purely based on its usefulness in today’s context is thus not a particularly desirable way to think about future applications.


### 3.2. Exploiting a Promising but Immature Technology

In the three design examples, some emerging computational techniques served as the seed idea and starting point of the development process. This seems in stark contrast with the way conventional applications are supposed to be designed, but unlike supporting existing needs and known activities, identifying new needs and creating novel activities cannot solely depend on existing practices and known concepts, as implied in the previous section on the novelty spectrum. The design aim of these projects was to exploit a piece of emerging technological advancements as much as possible in the provision of new interaction features, instead of fitting the technology to an identified user need.One difficulty in doing so is that the technological advancement to be exploited is, by definition, an immature and less-understood technology currently being researched in laboratories. A problematic issue with taking a premature piece of technology out of the lab and developing an application based on it is that it does not perform accurately, reliably or robustly. The developed novel application should be showcased and even user-tested to help explore more avenues for usage scenarios and interaction possibilities. However, when the very element that the design of the application tries to exploit does not function as expected, the value of the application in the eyes of test users inevitably drops [22] as was the case with My Friends’ Faces.As the technological tools become more mature and accurate as R&D in these areas continues, there will be fewer such issues arising for that application. However, instead of simply expecting the technology to mature in a few years, a more constructive strategy is to design to support such inaccuracy. More and more computational technology research communities are realising that user interactions with the applications could support simple and easy manual correction mechanisms or other work-around solutions to alleviate this problem. For example, My Friends’ Faces provides a simple manual name correction feature when the system predicts a person’s name incorrectly; in My Virtual Museum, the user interface provides a drag-and-drop feature for a user to manually regroup the photos and a list of candidate photos from which the user can choose, for possible SIFT technique failures, and My Visual Diary has a minislider bar to adjust the number of events presented on the comic-book style panel if the system did not pick an important event initially. These interaction mechanisms have been designed in from the beginning of the development process and thus are seamlessly and gracefully incorporated as part of the overall interaction flow. Leveraging human users’ intentional input by encouraging more user feedback and other types of emerging online crowdsourcing mechanisms such as Amazon Mechanical Turk (Amazon Mechanical Turk, artificial artificial intelligence,https://www.mturk.com/) can thus be understood as a way of compensating for the inaccuracy in fully automatic computations and is now increasingly featured in the demonstrators developed by computational technology communities. A unique situation the media technology community is facing today is that in trying to exploit computational tools initially intended to automate the labour-intensive tasks, they are beginning to bring back human users on the scene (or behind the scene, to be more accurate) in the form of a large number of anonymous human labours afforded by web connectivity, replacing or complementing the automatic methods.


### 3.3. Design Ideas Unfold as Project Progresses: Reflective Conversation with the Situation

There is a great deal of general design knowledge base that we can use to develop interaction features. Design principles, guidelines, and heuristics that have been generated, accumulated, and evolved over the past 20 years or so were adhered to in the design of the above examples and used to provide detailed interaction strategies, visual layout, sequencing, and emphasis which were correct and usable [23]. Emphasising the general design knowledge rather than the specific, contextual knowledge where the system will operate will lead to designing applications more “open to interpretation” in many ways [24]. By designing not to fit specific user needs or contexts that we do not know but instead to make the interactivity affordances clear, we have better chances to explore where the technical possibilities exploited in the application could further lead. This contrasts to how a system or an application is to be designed to support existing or conventional activities that people do today where a specific set of requirements and a specific context of usage should be targeted in order to satisfy a specific user needs.However, using the general design knowledge base as the starting point for novel application design does not guarantee a successful execution of the project. In the three example design cases and indeed most other novel application design projects we have experienced, the absence of a user base and lack of information on the situations and contexts where an application is to operate forced us to take an approach that is strongly design disciplinary rather than scientific or problem solving. Having only vague initial visions on usage with stronger technical intentions (to exploit a particular technological tool), it was not possible to rigorously define and clarify the requirements that would typically drive a design in a mainstream design project. Rather, the earlier part of the design process proceeded mainly by filling in sensible or possible scenario elements by guessing, imagining, and framing; the latter part of the process proceeded by sketching the interaction and continuously revising by going through the sketches while receiving feedback from the members of the project providing the technical possibilities. The sketching activity was strongly helped by the general design knowledge base mentioned above to ensure general usability and clarity in its interaction affordances, but the crucial guidance of the design progress, especially in shaping the overall interaction strategies, occurred through seeing the sketches and critically reflecting on them to make revisions before reflecting again. In this sense, research through design [25] with its propositional, future-oriented and meaning-after-action approach of inquiry, is one of the ways to explain how these novel applications have been designed and experimented. Understanding the mechanisms of designing novel applications may well form an important body of case studies for formalising the Research through Design into a proper methodology as more theoretical and rigorous foundations are being shaped. A move towards “constructive design research” [26] where the construction of product, system or media itself takes the central place in accumulating knowledge will also contribute to shaping the agreed-upon foundation for understanding how novel applications are to be designed.In our examples of novel applications, the design ideas, sketching, and ways to improve them unfolded and developed as the project progressed, becoming more concrete and detailed as the process continued. It can be said that the required knowledge was created bit by bit along the way, rather than available in large quantity at the beginning. It was impossible to collect all necessary information about the interaction at the beginning to set a definite “problem” and then to find the “solution.” In most cases, we faced more elements of solutions readily available to us (technological tools that we wanted to exploit and general design principles and guidelines) than of problems (where the system will be used and what the users want to do in what context).For My Visual Diary, for example, it was not even clear whether our future users would benefit from seeing a list of past events that were similar to a given event that happened today or, for that matter, whether they will want to review their days every evening. Having great uncertainty in the problem space and yet being fixated on finding solutions regardless via continuous reflective conversation with the sketching [27] is a typical characteristic of design practise in domains such as architecture and industrial design [28, 29]. With regard to novel interactive application design, this design practise characteristic seems to be even more prominent and crucial. For example, the typical design characteristic of quickly generating possible solutions then putting efforts into refining them, instead of investing in fully understanding the problem first, renders itself well for novel application design situations where the weak initial information without definite and clear target contexts force the designer to dwell on the solution space rather than the problem space. In particular, design expertise exhibits the ability to work well with problems that do not lend themselves to exhaustive analysis [30], to consider different levels of abstraction at the same time [28, 29], and to innovate across disciplines [31]. Harnessing these design abilities into the methodological framework specifically tailored for designing novel applications allows more effective and successful technology-to-application transformations.One implication of this shaky starting point in a novel application design (due to the lack of understanding of the eventual usage) and the continuous reflective exploration as the project progresses is that it is very difficult to prescribe a general step-by-step procedure which, when rigorously followed, will guarantee a successful creation of a novel application. The activity of designing a novel application requires a series of tentative exploration at the current situation with a limited information and understanding in order to unfold a more enlightened situation; the subsequent step is very much dependent on what has been found at the previous step and how to frame it for the next. However, reflecting on the example design projects as introduced in Section2, a rough formation of the procedural aspect of this type of design project might take such steps as the following:(1)
identify a specific technological possibility afforded by a recently emerging computation or a combination of it;(2)
by discussing with the researcher or the group who developed that specific technology, find out the overall assumption or projection behind that piece of technology, however vague or far fetched;(3)
using the assumption revealed as the initial clue, start a series of sketching of possible usage scenarios scoping as generic and open as possible. The sketching sessions will be continuous feedback loops between the designer and the technology provider, making up the core creative stage of the design project. This stage will ask the technology provider on the characteristics and properties of the computational technology to be exploited and their implications for the end-user features (e.g., can the computation be real time; can it cope with different domain data; can it cope with small amount of input from the end users, and if so, what will be the minimum amount that could achieve an acceptable accuracy; can the computation run on the server or client; can it run on various interaction devices and gadgets available today or soon-to-be available ones; can the accuracy be enhanced during the interaction by the end-user’s manual input, etc.) and will intensively iterate between sketching and feedback. The result is a series of mockups or sketches of the application to demonstrate the concept and the power of the specific computation incorporated in the concept;(4)
implement a working version of the mockup in order to user-test its interactivity and obtain feedback from the end users.Once a working prototype is available, then many conventional usability engineering methods currently practiced can be utilised to observe, monitor, and get the sense of the usage. As can be seen in these steps, the core of the methodology is in the art of exploiting the computation in concern in the suitable and innovative way (i.e., step 3) rather than in diligently adhering to a prescribed steps. Further prescribing a general procedure within step 3 may be less meaningful due to the many unpredictable factors at play in each specific project and the synthesis-oriented nature of the step perhaps more requiring the designer’s inherent quality of trying to frame the situation from his or her experience and to match a portfolio of prior design solutions, thereby increasing the certainty for the next iteration. In this sense, recently proposed practical methodologies to support quick turnaround for innovative product design in business and industry such as O’Reilly’s Lean UX [32] may be some early examples of how this design thinking-driven, iteration-heavy, and technology-inspired approach might eventually take a more procedural form.


### 3.4. Value of Novel Applications: Evaluation Criteria

For all three example applications described in Section2, the motivation for conducting the user evaluation sessions was less about verifying whether the systems satisfied a set of identified requirements or supported the users’ preconceived wishes but was more about witnessing how they perceived these systems to be beneficial to them and how they might see the novel interaction features could be adopted to their other existing tools and services or to their current lifestyle, although many detailed usability issues and widget-level problems on their user interfaces were also identified along the way as a side effect.Greenberg and Buxton [33] warn against blindly following the evaluation doctrine available today, and Olsen Jr. [34] advises us to avoid the trap of only creating what a usability test today can measure. These are the points particularly relevant with regard to developing novel applications. Conventional application designs are driven by initially identified user needs and requirements, and thus their evaluation is also geared towards assessing whether the designed application satisfies user needs and requirements. The goal of designing novel applications, as demonstrated in this paper, is not so much to fit existing user needs and requirements. Therefore, there is a limitation when its evaluation is driven by the prevailing idea of how well the application satisfies the user needs and requirements.People’s initial reactions to a novel feature could be negative simply because it is not something with which they are familiar or had expected. For example, in My Friends’ Faces, our test users’ opinions about using the application were generally low simply because they expected concepts and styles similar to their familiar photo services. It is impossible to remove such a bias in conducting a one-off user evaluation and asking their opinions about novel features.More appropriate criteria for the evaluation of this type of novel applications would include the following.(i)
How well does the application exploit and represent the back-end technique?(ii)
How well does the application exhibit the generic affordances as to how to interact with it?(iii)
How well does the application provoke brainstorming and discussion?In other words, what should be evaluated is the role of the application in helping explore the connection between technical possibilities and novel usage, not the role of the eventual, situated application with its detailed feature provision to fit a specific usage context. Fitting the application to a specific context is a task that can be performed much later, after major technical possibilities and exploitation efforts have been explored. Once we draw a clearer distinction between design practise (context-driven, supporting real-world practise) and design exploration (idealistic, supporting invention of new artefacts) [11] in terms of the design methodology, then we will be able to devise and offer more specialised, more optimal, and more cost-effective design processes tailored for different kinds of design activities. There is no reason why the technology R&D community should try to create a perfect application in one go that exhibits novel functionalities and fulfills specific user needs at the same time.


## 3.1. Underlying Needs and Novelty

A novel application implies that there is anovel activity that the application can support or the type of activity that is new to people today. The intention of coming up with a novel application, then, is to invent a tool to create a new activity that people can start engaging in to benefit their lives in some way. For example, the activity of tweeting had not existed until the tool Twitter was created, and subsequently people started using it, and the value of using it became apparent. The activities of online video sharing and voting, blogging, social networking, mobile texting, internet shopping, or even TV watching are among the many examples of innovative systems or services that created novel activities (rather than supporting existing activities) at the time of their invention.At the early stage of the development of the three novel applications in Section2, establishing clear user needs was problematic because the premise of the projects was to exploit an emerging computational tool in a way that has not been used before. However, there were different degrees in which this problem of lacking the understanding of user needs affected the design process. In the case of My Friends’ Faces, once the role the back-end computational tool to be exploited (face detection and recognition) had been mapped to the novel feature of the iconised face index as the interaction starting point, the rest of the design process proceeded in a relatively straightforward manner. In the case of My Visual Diary, however, the uncertainty of user needs and operating the project under that uncertainty strongly influenced the design and evaluation throughout the project, making the undertone of the project quite different from My Friends’ Faces. For example, in analysing the user evaluation of My Friends’ Faces, it was possible to solicit the test users’ views and opinions on the novel feature in comparison to their prior experiences of using other more conventional photo management services because, even with the novel face index mechanism as the default interaction starting point, they were more or less agreeing on the overall purpose and the value of the system under testing; it was not the case for My Visual Diary.There are different degrees of novelty in the novel applications. Blogging, for example, was a novel activity when it first started, but one might argue that people’s inherent need for expressing themselves had existed and manifested in the forms of writing letters and keeping diaries, until the timely online tool came out and satisfied our needs more conveniently and with bigger impact (by being able to reach a wider audience); tweeting activities may be traced to the need for staying connected with people, hitherto satisfied with shorter blog entries or mobile texting, or face-to-face small talk; online video sharing may be traced back to physically posting a parcel of video tapes or CDs that contain family videos; lifelogging activities may be comparable to an extreme case of proactive photo-taking or obsessive note taking during special events. Depending on how novel the need supported by the invented application is compared to any known or existing need, we can draw a spectrum of novelty. Figure4 depicts some novel activities invented in approximately the last decade and located on the novelty spectrum by the degree of novelty at the time of invention (for they have now become mainstream activities and are thus not considered novel anymore).Figure 4
Novelty spectrum with example novel activities.For example, online shopping is a less novel activity because the need for shopping had been well known before any online shopping service was invented. Mobile texting is more novel than blogging because the need for instant messaging had not been explicitly known beforehand, while the need for diary keeping had been known.Also plotted in this spectrum are the three example applications from the previous section. In the case of My Friends’ Faces, though the overall activity of online photo browsing and sharing is not novel today, the way the application invites the user to browse the list of faces (instead of a group of photos) as the starting point of interaction is the novel aspect of this application afforded by the fully automatic face detection and recognition techniques. More novel than this is a museum visitor being able to browse his/her photos to gain detailed information about the artefacts. While people do visit museums and take photos of artefacts today, the need for uploading them to a system to organise them by unique artefacts and learn more details was unknown and most likely nonexistent. My Visual Diary, for visual lifelog users, is much more novel, as the need for lifelogging and reviewing one’s day is not a familiar concept as a “need” by the general public today.In some cases, especially those on the left side of the spectrum in Figure4, we may be able to create novel activities by observing people’s existing activities and identifying the inherent needs of those activities before trying to link those needs with recent or emerging technical possibilities. Here, proxy activities and proxy users could be used to identify useful information as the basis for inventing more novel activities and tools to support them (e.g., studying active photo bloggers to inform the design of a novel photoware system). Conversely, for those activities that are more novel (those on the right side of the spectrum), observation of people’s existing activities is likely to have less benefit because what is to be created can greatly differ from what people need and practise today. In this sense, those activities on the left side of the novelty spectrum tend to have a greater chance of benefiting from user studies (including interviews and observing people’s behaviour) than those activities on the right side, when creating new applications to support them.The tension between creating novel products and conventional ones has certainly been a reality in industry and a topic of interest in academia, witnessed from the early most advanced yet acceptable (MAYA) principle by the well-known industrial designer Raymond Loewy (Raymond Loewy, The Father of Industrial Design,http://www.raymondloewy.com/) to offer a novel solution but acceptable to current consumers, to a more recent view on how to interpret the phenomenon of incremental versus radical innovations [8]. In a series of consumer studies, Hekkert et al. [9] found that consumers preferred novel designs but not so much novel as to hinder the typical use of it, implying a careful balancing act required by a designer between novelty and typicality. Similar findings are reported in a study of chair design where the correlation between the level of novelty and aesthetic preference by people was observed as an inverted-U curve [10], implying a moderate level of novelty to be perceived best. More clearly categorising design activities by the purpose of design also comes into the picture: “design exploration” to find out what is possible  and what would be desirable or ideal, as opposed to “design practice” to build a system that satisfies a specific group of users in a specific context [11]. These different kinds of design activities suggested can be seen as targeting to create applications at different points on the novelty spectrum. The transfer scenarios [12, 13] try to “combine the best of both worlds” of invention (idea generation) and inquiry (study of people) by studying the people with marginal or unusual practices and transferring the main features discovered to a different domain in order to design novel systems or services. Similarly, “matchmaking” method [14] tries to match the already-known, existing, and intended technology to a specific work domain that is not yet clearly defined or established.Because it is difficult to suddenly come up with a novel activity or novel application that supports such an activity, there have been attempts to explore the methodological solutions to somehow balance between technologically oriented invention (right side of the spectrum) and grounded user study (left side of the spectrum). For example, a framework to align “blue-sky” research with real user interests [15], a playful participatory design to inspire new ideas [16], speed dating method to rapidly and cheaply compare design opportunities well before any prototyping effort is undertaken [17], “technology probe” [18] and other application studies that followed, a participatory envisioning and enactment to make the design-evaluation feedback cycle as dynamic and frequent as possible [19], and recent guidelines on when and how to use such a user enactment [20] are the examples in which the aspects of both users and technology were utilised in trying to come up with new applications on varying points on the novelty spectrum. Each of the methods as mentioned here will be better at creating novel applications on a different range or different points on the novelty spectrum. For example, those methods that involve participatory design approach will more likely end up creating novel applications on the left side, rather than right side, of the spectrum.The novelty spectrum illustrated here is far from objective or accurate, but one of the aspects it highlights is that designing a novel interactive system can range from switching around the initial information access point (My Friends’ Faces) to providing a value-added activity in conjunction with an existing activity (My Virtual Museum) to providing a completely novel activity that might currently be considered obsessive or even ridiculous (My Visual Diary). The spectrum can be used to roughly gauge any group of system or application development in terms of their level of novelty and to guide the kind of approach, tools, and overall mindset required to optimally engage in such development. For example, most of the demonstrator systems showcased at the recent Video Browser Showdown (Video Browser Showdown, at the 19th International Conference on Multimedia Modeling, 7–9 January 2013, available online athttp://mmm2013.org/Video_browser_showdown.htm) where all sorts of novel interactive video search systems had an open competition at a conference venue fall into the left side of the spectrum where the end-user need had been more or less clear from the beginning (quickly search for video clips) although the search features in the systems exploiting the computer vision and information retrieval techniques made them novel and innovative. On the other hand, many prototype systems exhibited at the laboratories such as MIT Media Lab and Interaction Research Studio at the University of London Goldsmiths try to experiment with new affordances that emerging technologies allow even when the result is not what people today have experience about (e.g., “History Tablecloth” [21] shows the trail of objects that had been on the table but moved away later. Nobody actually voiced that such a feature would be useful in our daily lives, but by being able to show such a history of where things had been on the table before, they are trying to discover or create a completely new, potentially beneficial use and need). We can locate this group of prototypes on the right side of the spectrum. Locating any of the developing systems by different research groups or communities on the spectrum provides a rough indication of their stance or agenda in exploring novel possibilities and the kind of weight they place between the knowledge on the end-user needs and the use of technology.Another aspect the spectrum highlights is that, in general, the less novel is an application, the more useful it is in today’s context and vice versa. As we strive to explore more novel applications (the right side of the spectrum), they will be obviously viewed as less useful in today’s context because of the unknown nature of the need and activity that the application tries to support. Judging the value of a designed application purely based on its usefulness in today’s context is thus not a particularly desirable way to think about future applications.


## 3.2. Exploiting a Promising but Immature Technology

In the three design examples, some emerging computational techniques served as the seed idea and starting point of the development process. This seems in stark contrast with the way conventional applications are supposed to be designed, but unlike supporting existing needs and known activities, identifying new needs and creating novel activities cannot solely depend on existing practices and known concepts, as implied in the previous section on the novelty spectrum. The design aim of these projects was to exploit a piece of emerging technological advancements as much as possible in the provision of new interaction features, instead of fitting the technology to an identified user need.One difficulty in doing so is that the technological advancement to be exploited is, by definition, an immature and less-understood technology currently being researched in laboratories. A problematic issue with taking a premature piece of technology out of the lab and developing an application based on it is that it does not perform accurately, reliably or robustly. The developed novel application should be showcased and even user-tested to help explore more avenues for usage scenarios and interaction possibilities. However, when the very element that the design of the application tries to exploit does not function as expected, the value of the application in the eyes of test users inevitably drops [22] as was the case with My Friends’ Faces.As the technological tools become more mature and accurate as R&D in these areas continues, there will be fewer such issues arising for that application. However, instead of simply expecting the technology to mature in a few years, a more constructive strategy is to design to support such inaccuracy. More and more computational technology research communities are realising that user interactions with the applications could support simple and easy manual correction mechanisms or other work-around solutions to alleviate this problem. For example, My Friends’ Faces provides a simple manual name correction feature when the system predicts a person’s name incorrectly; in My Virtual Museum, the user interface provides a drag-and-drop feature for a user to manually regroup the photos and a list of candidate photos from which the user can choose, for possible SIFT technique failures, and My Visual Diary has a minislider bar to adjust the number of events presented on the comic-book style panel if the system did not pick an important event initially. These interaction mechanisms have been designed in from the beginning of the development process and thus are seamlessly and gracefully incorporated as part of the overall interaction flow. Leveraging human users’ intentional input by encouraging more user feedback and other types of emerging online crowdsourcing mechanisms such as Amazon Mechanical Turk (Amazon Mechanical Turk, artificial artificial intelligence,https://www.mturk.com/) can thus be understood as a way of compensating for the inaccuracy in fully automatic computations and is now increasingly featured in the demonstrators developed by computational technology communities. A unique situation the media technology community is facing today is that in trying to exploit computational tools initially intended to automate the labour-intensive tasks, they are beginning to bring back human users on the scene (or behind the scene, to be more accurate) in the form of a large number of anonymous human labours afforded by web connectivity, replacing or complementing the automatic methods.


## 3.3. Design Ideas Unfold as Project Progresses: Reflective Conversation with the Situation

There is a great deal of general design knowledge base that we can use to develop interaction features. Design principles, guidelines, and heuristics that have been generated, accumulated, and evolved over the past 20 years or so were adhered to in the design of the above examples and used to provide detailed interaction strategies, visual layout, sequencing, and emphasis which were correct and usable [23]. Emphasising the general design knowledge rather than the specific, contextual knowledge where the system will operate will lead to designing applications more “open to interpretation” in many ways [24]. By designing not to fit specific user needs or contexts that we do not know but instead to make the interactivity affordances clear, we have better chances to explore where the technical possibilities exploited in the application could further lead. This contrasts to how a system or an application is to be designed to support existing or conventional activities that people do today where a specific set of requirements and a specific context of usage should be targeted in order to satisfy a specific user needs.However, using the general design knowledge base as the starting point for novel application design does not guarantee a successful execution of the project. In the three example design cases and indeed most other novel application design projects we have experienced, the absence of a user base and lack of information on the situations and contexts where an application is to operate forced us to take an approach that is strongly design disciplinary rather than scientific or problem solving. Having only vague initial visions on usage with stronger technical intentions (to exploit a particular technological tool), it was not possible to rigorously define and clarify the requirements that would typically drive a design in a mainstream design project. Rather, the earlier part of the design process proceeded mainly by filling in sensible or possible scenario elements by guessing, imagining, and framing; the latter part of the process proceeded by sketching the interaction and continuously revising by going through the sketches while receiving feedback from the members of the project providing the technical possibilities. The sketching activity was strongly helped by the general design knowledge base mentioned above to ensure general usability and clarity in its interaction affordances, but the crucial guidance of the design progress, especially in shaping the overall interaction strategies, occurred through seeing the sketches and critically reflecting on them to make revisions before reflecting again. In this sense, research through design [25] with its propositional, future-oriented and meaning-after-action approach of inquiry, is one of the ways to explain how these novel applications have been designed and experimented. Understanding the mechanisms of designing novel applications may well form an important body of case studies for formalising the Research through Design into a proper methodology as more theoretical and rigorous foundations are being shaped. A move towards “constructive design research” [26] where the construction of product, system or media itself takes the central place in accumulating knowledge will also contribute to shaping the agreed-upon foundation for understanding how novel applications are to be designed.In our examples of novel applications, the design ideas, sketching, and ways to improve them unfolded and developed as the project progressed, becoming more concrete and detailed as the process continued. It can be said that the required knowledge was created bit by bit along the way, rather than available in large quantity at the beginning. It was impossible to collect all necessary information about the interaction at the beginning to set a definite “problem” and then to find the “solution.” In most cases, we faced more elements of solutions readily available to us (technological tools that we wanted to exploit and general design principles and guidelines) than of problems (where the system will be used and what the users want to do in what context).For My Visual Diary, for example, it was not even clear whether our future users would benefit from seeing a list of past events that were similar to a given event that happened today or, for that matter, whether they will want to review their days every evening. Having great uncertainty in the problem space and yet being fixated on finding solutions regardless via continuous reflective conversation with the sketching [27] is a typical characteristic of design practise in domains such as architecture and industrial design [28, 29]. With regard to novel interactive application design, this design practise characteristic seems to be even more prominent and crucial. For example, the typical design characteristic of quickly generating possible solutions then putting efforts into refining them, instead of investing in fully understanding the problem first, renders itself well for novel application design situations where the weak initial information without definite and clear target contexts force the designer to dwell on the solution space rather than the problem space. In particular, design expertise exhibits the ability to work well with problems that do not lend themselves to exhaustive analysis [30], to consider different levels of abstraction at the same time [28, 29], and to innovate across disciplines [31]. Harnessing these design abilities into the methodological framework specifically tailored for designing novel applications allows more effective and successful technology-to-application transformations.One implication of this shaky starting point in a novel application design (due to the lack of understanding of the eventual usage) and the continuous reflective exploration as the project progresses is that it is very difficult to prescribe a general step-by-step procedure which, when rigorously followed, will guarantee a successful creation of a novel application. The activity of designing a novel application requires a series of tentative exploration at the current situation with a limited information and understanding in order to unfold a more enlightened situation; the subsequent step is very much dependent on what has been found at the previous step and how to frame it for the next. However, reflecting on the example design projects as introduced in Section2, a rough formation of the procedural aspect of this type of design project might take such steps as the following:(1)
identify a specific technological possibility afforded by a recently emerging computation or a combination of it;(2)
by discussing with the researcher or the group who developed that specific technology, find out the overall assumption or projection behind that piece of technology, however vague or far fetched;(3)
using the assumption revealed as the initial clue, start a series of sketching of possible usage scenarios scoping as generic and open as possible. The sketching sessions will be continuous feedback loops between the designer and the technology provider, making up the core creative stage of the design project. This stage will ask the technology provider on the characteristics and properties of the computational technology to be exploited and their implications for the end-user features (e.g., can the computation be real time; can it cope with different domain data; can it cope with small amount of input from the end users, and if so, what will be the minimum amount that could achieve an acceptable accuracy; can the computation run on the server or client; can it run on various interaction devices and gadgets available today or soon-to-be available ones; can the accuracy be enhanced during the interaction by the end-user’s manual input, etc.) and will intensively iterate between sketching and feedback. The result is a series of mockups or sketches of the application to demonstrate the concept and the power of the specific computation incorporated in the concept;(4)
implement a working version of the mockup in order to user-test its interactivity and obtain feedback from the end users.Once a working prototype is available, then many conventional usability engineering methods currently practiced can be utilised to observe, monitor, and get the sense of the usage. As can be seen in these steps, the core of the methodology is in the art of exploiting the computation in concern in the suitable and innovative way (i.e., step 3) rather than in diligently adhering to a prescribed steps. Further prescribing a general procedure within step 3 may be less meaningful due to the many unpredictable factors at play in each specific project and the synthesis-oriented nature of the step perhaps more requiring the designer’s inherent quality of trying to frame the situation from his or her experience and to match a portfolio of prior design solutions, thereby increasing the certainty for the next iteration. In this sense, recently proposed practical methodologies to support quick turnaround for innovative product design in business and industry such as O’Reilly’s Lean UX [32] may be some early examples of how this design thinking-driven, iteration-heavy, and technology-inspired approach might eventually take a more procedural form.


## 3.4. Value of Novel Applications: Evaluation Criteria

For all three example applications described in Section2, the motivation for conducting the user evaluation sessions was less about verifying whether the systems satisfied a set of identified requirements or supported the users’ preconceived wishes but was more about witnessing how they perceived these systems to be beneficial to them and how they might see the novel interaction features could be adopted to their other existing tools and services or to their current lifestyle, although many detailed usability issues and widget-level problems on their user interfaces were also identified along the way as a side effect.Greenberg and Buxton [33] warn against blindly following the evaluation doctrine available today, and Olsen Jr. [34] advises us to avoid the trap of only creating what a usability test today can measure. These are the points particularly relevant with regard to developing novel applications. Conventional application designs are driven by initially identified user needs and requirements, and thus their evaluation is also geared towards assessing whether the designed application satisfies user needs and requirements. The goal of designing novel applications, as demonstrated in this paper, is not so much to fit existing user needs and requirements. Therefore, there is a limitation when its evaluation is driven by the prevailing idea of how well the application satisfies the user needs and requirements.People’s initial reactions to a novel feature could be negative simply because it is not something with which they are familiar or had expected. For example, in My Friends’ Faces, our test users’ opinions about using the application were generally low simply because they expected concepts and styles similar to their familiar photo services. It is impossible to remove such a bias in conducting a one-off user evaluation and asking their opinions about novel features.More appropriate criteria for the evaluation of this type of novel applications would include the following.(i)
How well does the application exploit and represent the back-end technique?(ii)
How well does the application exhibit the generic affordances as to how to interact with it?(iii)
How well does the application provoke brainstorming and discussion?In other words, what should be evaluated is the role of the application in helping explore the connection between technical possibilities and novel usage, not the role of the eventual, situated application with its detailed feature provision to fit a specific usage context. Fitting the application to a specific context is a task that can be performed much later, after major technical possibilities and exploitation efforts have been explored. Once we draw a clearer distinction between design practise (context-driven, supporting real-world practise) and design exploration (idealistic, supporting invention of new artefacts) [11] in terms of the design methodology, then we will be able to devise and offer more specialised, more optimal, and more cost-effective design processes tailored for different kinds of design activities. There is no reason why the technology R&D community should try to create a perfect application in one go that exhibits novel functionalities and fulfills specific user needs at the same time.


## 4. Conclusion

This paper examined how some technology-oriented projects were conducted to design novel interactive applications as front-ends of technological systems in ways that differ from what the conventional usability engineering practise prescribes, yet they achieved highly desirable outcomes.The paper highlighted the currently overlooked and undervalued activity of designing demonstrators in many technology laboratories, which are at the mercy of left-over resources at the end of projects and sometimes designed without the necessary design expertise. In particular, the prevailing mismatch between design approach and design purpose in these projects results in either wasted resources or under exploration of technological possibilities.Through the example applications, we demonstrated that the “novelty” in novel applications can range from supporting a simple new access point in user interactivity, to expanding the scope of an existing activity, to a completely new activity which not many people would normally suggest. Also demonstrated in terms of incorporating emerging computational technologies as the core back end of a system was the shift from relying on full automation to “design for failure”—supporting the features for the users or group of users to easily cope with the imperfect system performance. In addition, testing the demonstrator systems featuring this type of error-coping features will help guide the research direction for the underlying computation and back end technology itself. Our novelty spectrum implied that sometimes less novel activities might have greater chance of benefiting from an initial user study or user research.Interaction design has different purposes depending on the reason for designing a system, and from there, it follows that different design approaches and methodologies are required for different design purposes. Designing novel applications inspired by emerging technologies is an activity in which many R&D laboratories engage, and considering its huge potential influence in shaping our future interactivity with technology in the coming years, a more suitable and pragmatic methodological support is clearly needed for such a community today. We hope the discussions in this paper will push the methodological support to be sufficiently differentiated and tailored for the technology laboratories to more fully exploit the technological possibilities that they explore and thus to maximise their research output in the coming years.



---

*Source: 180192-2013-06-15.xml*