COVER FEATURE 21ST CENTURY USER INTERFACES

Programmingwith Examples
to Develop DataIntensive User Interfaces
Jun Kato, National Institute of Advanced Industrial Science and Technology (AIST)
Takeo Igarashi, University of Tokyo
Masataka Goto, National Institute of Advanced Industrial Science and Technology (AIST)The programming-with-examples workflow lets developers
create interactive applications with the help of example
data. It takes a general programming environment and adds
dedicated user interfaces for visualizing and managing thedata. This lets both programmers and users understandapplications and configure them to meet their needs.

G

UI builders let users define the look of userinterfaces with simple “what you see is what
you get” (WYSIWYG) interaction. However, WYSIWYG interaction is insufficient
to develop working systems; some programming is
also necessary. Programming is essential to user interface design because it lets developers design how users
interact with an application. To support the development of conventional interactive systems, integrated
development environments (IDEs) represent standardized numeric values such as those for MOUSE_CLICKED and
KEY_TYPED as text or symbols. This simple representationreflects the characteristics of systems that run on computers with standardized input and output devices suchas a mouse, keyboard, and display. They typically handle
a limited amount of data transferred intermittently.

34

CO M PUTE R P U B LISH ED BY TH E I EEE COMP UTER SOCI E T YIn contrast, modern interactive applications havebeen becoming more visual and data-intensive. In
the age of big data, computers have become faster andsmaller, handling more visual data that cannot be presented intuitively with text or symbols. There are morevariations among applications than before, such as
robot control, gesture recognition, image processing,and animation.
Prior attempts to support the development of dataintensive systems include programming by example(PbE) systems (see Figure 1a). With PbE systems, users
demonstrate desired pairs of input and output data to
the systems—for example, by manipulating robot postures in front of a camera. Then, the systems infer the
program, such as one to control robot postures. Most
PbE systems let both programmers and end users create
0018-9162/16/$33.00 © 2016 IEEE

LIVE PROGRAMMING AND
PROGRAMMING WITH EXAMPLES
programs. However, because thesesystems typically encapsulate the program logic in a black box, they prevent
users from understanding or directly
specifying detailed behaviors. In addition, example data have no representation in such systems and cannot be
managed or edited, making the development process irreproducible. So,
PbE systems without the capability of
explicit programming are not suitablefor user interface design that preciselyreflects a programmer’s intention.
In contrast, our programming with
examples (PwE) workflow (see Figure1b) enables explicit programming andlets users manage and edit example
data. Programmers collect example
data to aid building program components. For instance, to create a gesturebased application, the programmer
collects a set of example gesture datafrom a sensor and tests the application
with that data. (Brad Myers used theterm “Programming with Example”
in 1986,1 though it indicated the omission of inferences in PbE systems anddid not discuss data representations.)
Here, we discuss using graphicalrepresentations in IDEs to support PwE.
Toward that end, we developed three
IDEs, each of which explores a different use of graphical representations,
including photos, videos, and theirinteractive editing. Our approach overlaps somewhat with live programming(LP), a general technique that enables
live editing of programs during runtime. (For more on LP, see the “Live
Programming and Programming with
Examples” sidebar.) Our experiencehas provided insights on user interface design, including guidelines onwhen to use a particular kind of graphical representation and the potential ofvisuals as a communication medium
for collaborative application authoring.

In terms of the programming experience, the approach most
relevant to programming with examples (PwE) is live programming (LP), which enables live editing of programs at runtime. LP
originated in research on object-oriented programming and visualprogramming, recently attracting attention for its application to
text-based programming environments.
LP and PwE both provide a fluid programming experiencethrough more informative user interfaces, and some overlaps exist. For example, TextAlive is an LP system. However, their focuses
differ slightly. LP focuses on logic, whereas PwE focuses on data.
LP conceptually aims to make the entire program editable duringruntime; PwE splits the program into the editable part (code) andthe rest (data). This separation not only simplifies the implementation but also enables end-user program customization.

Sensor1
session 1

if (Sensor1.isActive) {... // ...
}PbE system

Sensor1
session 2

Outputlive

IDE

Inferred program

Explicitly specified program
Integrated graphical representations

(a)

(b)

Example data

FIGURE 1. Two workflows for developing data-intensive interactive systems. (a) Programming by example (PbE). (b) Programming with examples (PwE). Unlike PbE systems,
PwE systems allow the representation of example data and enable explicit programming.
IDE: integrated development environment.

PROGRAMMING
WITH EXAMPLES

Modern interactive systems often use
a variety of physical user interfaces
consisting of sensors and actuatorsthat receive and send large amountsof raw data as input and output. This
includes visual data, such as photos
and videos; object properties, such ascolor, shape, and location; as well as
more structured information, such as

human or robot posture and parameters for animating graphical objects.
Such data serve as examples to tellcomputers about the problem we wantto solve.
As we mentioned before, PwE uses
this example data to build and test programs. For instance, building a program could involve training models
for machine-learning algorithms andspecifying robot postures. Testing theJ U LY 2 0 1 6 

35

21ST CENTURY USER INTERFACESRELATED WORK IN HANDLING POSTURE DATA

T

opobo is a programming-by-example system
in which users specify how robots should
move by grabbing and moving their joints.1
Choreonoid is a GUI tool with physical simulation
for creating robot motions.2 In both cases, theuser needs no programming knowledge but has
little control over how the robot should respondto user input. To develop interactive systems that
handle posture data, we still need integrated
development environments (IDEs).
Current IDEs usually come with text-based andsymbolic representations, which cannot represent
complex data such as posture information. Several studies have involved integrated still images.
Sikuli shows image data as an inline image in thecode editor,3 and heterogeneous visual programming languages visualize simple tree structures
next to the code editor.4 In these systems, still
images are either the represented data or its

program would involve executing itwith recorded input from sensors anddebugging it by checking output fromtrained models.
In current development environments, example data are usually referenced by textual or symbolic visual
representations such as file names
and icons. In conventional interactive systems, the connection between
the example data, text, and symbols
is usually obvious. In modern interactive applications, that is not the case;programmers often get confused by
the vague connection. In addition, programmers are responsible for managing the data. To examine and edit thedata, they typically must launch external tools outside the development
environments, which is tedious anderror-prone. The IDEs we present here
directly support PwE to address these
problems and make the process more
efficient and productive.
There have been several efforts
to support PwE. Most PbE systems
36

COMPUTER 

visualization. Our Picode IDE uses photos not as
just mere images but as representations of underlying posture data.
References
1. H.S. Raffle et al., “Topobo: A Constructive Assembly Systemwith Kinetic Memory,” Proc. 2004 SIGCHI Conf. Human Factors in Computing Systems (CHI 04), 2004, pp. 647–654.
2. S. Nakaoka et al., “Intuitive and Flexible User Interface forCreating Whole Body Motions of Biped Humanoid Robots,”
Proc. 2010 IEEE/RSJ Int’l Conf. Intelligent Robots and Systems (IROS 10), 2010, pp. 1675–1682.
3. T. Yeh et al., “Sikuli: Using GUI Screenshots for Searchand Automation,” Proc. 22nd Ann. ACM Symp. User Interface Software and Technology (UIST 09), 2009,
pp. 183–192.
4. M. Erwig et al., “Heterogeneous Visual Languages Integrating Visual and Textual Programming,” Proc. 11th Int’l IEEE
Symp. Visual Languages (VL 95), 1995, pp. 318–325.

require no programming knowledge,
but ex­ceptions exist. For instance,
Pygmalion lets programmers build
a program by giving concrete input
data instead of writing abstract program code.2 The Subtext IDE lets programmers specify test cases by writing
example input data next to programcode in a code editor.3 Our IDEs employ
the same PwE workflow as these systems but use graphical instead of textual representations. This difference
reflects the example data becoming
more visual and complex in modern
interactive applications.
Recent IDEs have successfullyin­corporated various forms of examples. The most relevant one is Gestalt,
which supports the implementationof machine-learning algorithms forimage recognition.4 It provides dedicated GUIs to manage and edit example data. Barista lets programmers
paste graphics directly into the code
editor.5 Standing atop these prior
IDEs with PwE support, we exploreda broader design space that employs
graphical representations in the IDEs.
Blueprint lets programmers searchfor example source code in online
repositories and paste the result
through a code completion interface.6 Whereas examples in Blueprint
denote the use of APIs—the logicof the programs—examples in PwE
denote data used in the programs.

PHOTOS REPRESENTINGSTATIC DATA

Here we explore using photos to represent static example data, which does
not change over time. Our Picode IDE
supports the development of programs
that sense human and robot posturesand control robot postures.7 (For moreon handling posture data, see the“Related Work in Handling PostureData” sidebar.)

PicodePicode comprises a code editor, pose
library, and preview window (seeW W W.CO M P U T E R .O R G /CO M P U T E R

Preview window

Figure 2). Programmers first take aphoto of a human or robot in the pre­view window; Picode captures the cor­
responding posture data from a Kinect
sensor or the robot’s servos. Picode
stores a collection of these paired datain the pose library as a single entry.
Then, programmers drag a photo fromthe library and drop it into the code
editor. Picode embeds the photo incode, which the programmers can exe­cute with a single click.

User study

A programmer and nonprogrammer
performed pair programming with
Picode for three hours. The program­mer benefited from the PwE workflow,
and even the nonprogrammer could
infer and comprehend the source codesurrounding inline photos.
Given the encouraging results, we
hosted a workshop for nonprogram­mers to further investigate the roleof photos in the code. We found thatphotos contained three types of infor­
mation that was hardly represented by
text, symbols, or 3D computer graphics
visualizing posture data.
First, photos present the environ­ment surrounding humans and robots.
This helps programmers quickly un­der­stand a program’s operating environ­ment and objectives. For example,
humans employ the same posture
whether they are pushing a cart on aslope or performing calisthenics. A robot
employs similar postures when holding
a small or large ball, with the only dif­ference being how wide the hand opens.
Although numerical distinction is possi­
ble, this is not informative for humans.
Second, photos sometimes indicate
a particular part of a posture. In a textbased IDE, a programmer could indicate
interest in the second joint of a robot
by inserting a textual comment in the

Code editor

Pose library

FIGURE 2. Picode shows inline photos in the code editor to represent example posturedata. Programmers drag a photo from the pose library and drop it into the code editor.
Picode embeds the photo in code, which the programmers can execute with a single click.

source code. However, the phrase “thesecond joint” will not instantly informthe reader which joint it is; a photo canbe a more direct representation in which
the programmer points to the joint.
Finally, photos containing humansubjects can express emotion. This
characteristic has been significant
throughout the medium’s long his­
tory and is not found in conventionalsource code. By our study’s end, thepose library included photos showing
participant enjoyment and creative
shots of the robots. The source codewith inline photos was extremely indi­vidual, showing a variety of clothing,poses, and expressions. The photos
motivated the nonprogrammers to cus­
tomize the program and further learn
general programming.

to real-time video preview in video-​
editing applications. Users can moni­
tor any number of variables, including
input and output, continuously duringruntime in an arbitrary layout. They
can also review a frame of interest in apast program session.
When programmers select a vari­able in the code editor and drag it onto
the canvas, DejaVu represents it as arectangular box with visuals (images
and skeletal data, in the case of humansubjects in the video) or text (numericalor Boolean data). Along with monitor­
ing variable values, programmers candraw sketches and notes on the can­vas to aid visual data management. By
combining sketches with variable val­ues, programmers can turn the canvas
into a “dynamic sketchbook” in whichsketches come alive with dynamic data.
Similarly to interfaces in videoVIDEOS REPRESENTING
editing applications, the timeline inter­
DYNAMIC BEHAVIOR
Here, we explore using videos to rep­ face represents changes in the example
resent time-coded example data. Our data over time. The timeline might con­
DejaVu IDE (see Figure 3) supports the sist of multiple data streams, each cor­development of interactive camera-­ responding to a variable in the canvas.
based programs.8 It incorporates a A stream of visual data is represented
video-­player metaphor into a standard as a strip of frame thumbnails, and atext-based IDE. (For more on handling stream of numerical or Boolean data is
time-coded data, see the “Related Work represented as a time graph. The time­in Handling Time-Coded Data” sidebar.) line enables not only passive review ofexample data but also revising the pro­gram and refreshing it by reexecuting it
DejaVuDejaVu employs two interlinked in­ter­ with recorded input data, which assists
faces. The canvas interface corresponds iterative development.
J U LY 2 0 1 6 

37

21ST CENTURY USER INTERFACESRELATED WORK IN HANDLING TIME-CODED DATAS

ome IDEs visualize time-coded information ofprogram executions. ZStep records all stack
traces and provides a navigation interface to go
back and forth over the trace to see which lineof code executed at a particular point.1 Whyline
records the stack traces and window output.2 It
also provides a “Why did this happen (or not happen)?” interface, which navigates to the cause ofa phenomenon such as the color of a pixel. These
integrations work well for discrete events withsimple data structures but cannot handle continuous data streams such as input from cameras
and output to servo motors.
Some developer tools have timeline interfaces
to help users understand continuous time-coded
data. Exemplar is a standalone tool for authoring
sensor-based interaction by demonstration.3
D.tools records user interactions with physicalcomputing devices and visualizes the events
along with recorded videos.4 Timelapse has
similar features for event logging and webpage
debugging.5 These tools are either a standalone tool outside the IDE or an IDE without atext-based code editor. In contrast, the DejaVu
IDE tightly integrates the code editor and timeline
interface.
References
1. H. Lieberman et al., “Bridging the Gulf between Code andBehavior in Programming,” Proc. 1995 SIGCHI Conf. Human Factors in Computing Systems (CHI 95), 1995,
pp. 480–486.
2. A.J. Ko et al., “Finding Causes of Program Output with theJava Whyline,” Proc. 2006 SIGCHI Conf. Human Factors in
Computing Systems (CHI 06), 2006, pp. 387–396.
3. B. Hartmann et al., “Authoring Sensor-Based Interactions
by Demonstration with Direct Manipulation and Pattern
Recognition,” Proc. 2007 SIGCHI Conf. Human Factors in
Computing Systems (CHI 07), 2007, pp. 145–154.
4. B. Hartmann et al., “Reflective Physical Prototyping throughIntegrated Design, Test, and Analysis,” Proc. 19th Ann. ACM
Symp. User Interface Software and Technology (UIST 06),
2006, pp. 299–308.
5. B. Burg et al., “Interactive Record/Replay for Web Application Debugging,” Proc. 26th Ann. ACM Symp. User Interface
Software and Technology (UIST 13), 2013, pp. 473–484.

User feedback
Code editor

CanvasTimeline

FIGURE 3. DejaVu shows time-coded example data in the canvas and timeline interfaces. It incorporates a video-player metaphor into a standard text-based IDE.
38

COMPUTER 

Three professional developers tried
out DejaVu. They had significant
experience in developing interactive
Kinect-based programs with Visual
Studio, a standard IDE. Afterward,
they agreed that DejaVu correlated
well with their current PwE workflow.
We learned three important lessons
from their feedback.
First, the tight integration in the
IDE enables the synchronous connection between the canvas, timeline,
and code editor. The participants previously had desired DejaVu’s featuresand sometimes even made their own
tools. However, the separate tools
were not as powerful and flexible as
DejaVu’s visual, integrated, and interactive support.
Second, it is not always easy
to collect example data from the
W W W.CO M P U T E R .O R G /CO M P U T E R

Parameter editor

programmer’s surroundings thatsatisfy quality or quantity requirements for program testing. The programmers wanted the ability toimport and manipulate the example
data from external sources.
Third, the canvas went beyond
individual data displays and aroused
the need for customizable visualization. The visualization could range
from simple graphic combinations
such as overlaying a skeleton on acolor image to more semantic compositions such as masking certain
image regions.

Code editor
Timeline

FIGURE 4. TextAlive allows live customization of programs with interactive user interfaces. Programmers and nonprogrammers can collaborate to create programs.

just like familiar video-authoring
tools. When users load an audio fileand its transcription, TextAlive automatically creates a video. Other tools
require users to create videos fromscratch and spend a huge amount of
INTERACTIVE EDITING
time manually synchronizing audio
OF EXAMPLE DATA
Whereas the previous two sections and text. TextAlive requires signifiexplored visualization of example cantly less effort.
For each text sample and graphic indata, this section focuses on usinggraphical operations to edit exam- the video, users can select an animaple data. Our TextAlive IDE (see Fig- tion template and customize its visual
ure 4) supports live programming of effects with the GUI widgets in thedomain-specific applications (video- parameter editor. They can also use
rendering programs) and offers inter- the widgets to debug templates. Synactive user interfaces for tuning their chronization errors can be correctedparameters.9 (For more on editing in the timeline, which visualizes thedata, see the “Related Work in Data text’s timing information.
Although each animation temManipulation” sidebar.)
TextAlive is similar to our Vision- plate supports customization, it still
Sketch, which supports the example-­ somewhat limits the resulting animacentric development of image- tion. To address this issue, TextAlive
processing algorithms.10 Although enables live programming of the temVisionSketch is primarily for program- plates. Many tools for creating animamers, it led us to develop TextAlive, tions include scripting engines, butwhich investigates the potential of such scripting is typically tied to thecollaborative application authoring. specific data and not easily generalWith TextAlive’s clearer separation of izable for later reuse. These tools alsouser interfaces for programming and do not provide much graphical feedparameter tuning, programmers and back during coding. TextAlive enables
nonprogrammers can collaborate to more reuse through an abstraction
mechanism and provides continuous
create applications.
graphical feedback. Programmers canopen a code editor to edit the impleTextAlive
TextAlive lets users create a synchron­ mentation and update the resulting
ized kinetic-typography video; it works video with just one click. There is no

notion of compilation or execution.
The program that creates the animations continues running virtually.
The update process not only
updates the video but also populates
or removes the GUI widgets. To populate a widget, programmers declare avariable in the template definition andwrite a comment block right before
the declaration.

User feedback

Four nonprogrammers and three programmers created videos with Text­
Alive. They had varying expertise invideo authoring and programming.
All of them welcomed the features
for authoring kinetic-typography videos, especially the timeline. Whereas
the nonprogrammers requested morevariations of templates, the programmers could implement new templates
and appreciated live programmingand easy widget creation.
To observe collaboration amongusers, we deployed TextAlive as a Web
service. From September 2015 to January 2016, the users created morethan 300 videos and 50 templates. The
nonprogrammers reported enjoying
authoring videos, whereas the programmers reported enjoying developing templates for extending expressivity with the help of the edited
example data. The programmers sometimes created videos to showcase their
J U LY 2 0 1 6 

39

21ST CENTURY USER INTERFACESRELATED WORKIN DATA MANIPULATION

V

isual programming languages (VPLs) let programmers manipulate visual components to build programs. Most VPLs do notsupport graphical operations other than repositioning boxes andconnecting boxes with lines. However, some go beyond the box-andline notation and allow editing several data types such as bitmaps.
Some text-based IDEs also provide GUIs for inputting values.
Active code completion provides type-specific GUI widgets for
specifying concrete values.1 For instance, when a programmer
instantiates a Color object, a color palette interface is populated
instead of text-based completion candidates. Unity for authoring
games (http://unity3d.com) and Apparatus for interactive graphs(http://aprt.us) enable live customization of program outputs
through graphical operations.
Although these IDEs allow interactive editing of certain data
without external tools, they assume that the data can be constructed from scratch, which is not feasible for creating modern
applications. In contrast, our TextAlive IDE allows interactive editingof example data provided from outside the IDE. It has separate­interfaces for programmers and nonprogrammers, connecting
them through integrated graphical representations.
Similarly, Gneiss provides a spreadsheet interface with a simple
domain-specific language to retrieve and organize real-time example data from the Web.2 Unlike TextAlive, Gneiss provides a single
user interface that both novices and nonprogrammers can use.
References
1. C. Omar et al., “Active Code Completion,” Proc. 34th Int’l Conf. Software Eng.
(ICSE 12), 2012, pp. 859–869.
2. K. Chang et al., “A Spreadsheet Model for Handling Streaming Data,” Proc.
2015 SIGCHI Conf. Human Factors in Computing Systems (CHI 15), 2015,
pp. 3399–3402.

templates. Graphical representationsof the programs (videos) and theirparameters (GUI widgets) served as abridge between the programmers andnonprogrammers.
A limitation of our current implementation is that collaboration is unidirectional; nonprogrammers have no
direct way to request new templateswith specific visual effects. The system ideally should support bidirectional collaboration, which we plan towork on.
40

COMPUTER LESSONS LEARNED

We now present lessons learned fromour experiences with our three IDEs
and describe our outlook on thistechnology.

Integrated graphicalrepresentations for PwE

Our experiments confirmed that integrating graphical representations lets
programmers understand, manage,
and use the example data to build andtest programs, effectively supporting

the PwE workflow. Combining concrete
graphical representations with abstract
textual representations improved programmer productivity. Compared totypical PbE systems that infer the interaction logic, PwE leaves it to the programmers, enabling precise, detailed
interaction design through iterative
testing and edits.
Furthermore, example data withgraphical representations make theprogram specification more accessible to nonprogrammers such as
visual designers and end users. They
can easily understand the program
behavior from visual examples (aswith Picode) and even modify the
behavior by interacting with the visuals (as with TextAlive).

Intuitive representationof example data

Integrated graphical representations
can be realistic (for example, photos
and videos) or symbolic (for example, askeleton representing a human body),
depending on their use cases. Although
it seems obvious that graphics aid programming, which kind of representation to use is not always obvious. Photos
and videos are particularly useful for
capturing the runtime environment,
including the real-world circumstances(as with Picode). They might contain
explicit information, such as interactions the user performed, and implicit
information, such as the user’s emotions and environment. On the other
hand, illustrations and symbolic figures
can eliminate unnecessary information
and are often good for understanding
abstract intentions. Combinations ofrealistic and symbolic representations
also work well. Just as map applications
support layering of satellite views andsymbolic maps, these two representations can be overlaid.
W W W.CO M P U T E R .O R G /CO M P U T E R

Video-authoring interfaces (suchas DejaVu and TextAlive) can intuitively present time-coded structured data. Researchers have proposed many intuitive interfaces for
manipulating data through GUIs;
we can learn from such interfaces tointegrate data manipulation components into IDEs. Although this article
focuses on inherently visual example
data, example data could be sounds,
haptic sensations, tastes, and smells.
Graphical representations could also
be useful in these cases. For instance,
photos of flowers could represent
their scent. At the same time, we foresee that future development environments could ex­­­­­­­ploit human sensory
organs other than the eyes, including
the ears, skin, tongue, and nose.

Collaborative application
authoring in the age of big data

Collaboration among people withdiverse backgrounds is essential fordeveloping successful user interfaces for data-intensive applicationsin the age of big data. The amountand variations of data far surpass
what programmers alone can manage and edit.
Current development environments enable such collaborations in
text-based communication, as seen
in bug reports. Some services, including GitHub, allow embedding graphics, but we argue that such graphics
should be more directly bound to theprogram specifications, as our experiments demonstrated. This will let
programmers and nonprogrammers
create applications that satisfy theirneeds. To achieve this, we need IDEs
with two faces: one for programmers and one for nonprogrammers.
The graphical representation is notjust a tool for programmers to build

and test programs or a tool for nonprogrammers to customize program
behavior. It is a common language
for both to discuss how the program
should work.

Eliminating borders between
the development andruntime environments

After IDEs integrate more graphics
and enable people with diverse backgrounds to comprehend them, we will
eventually reach the point at which noclear distinction exists between thedevelopment and runtime environments. Programmers will ship programs together with the development
environments so that nonprogrammers can customize them by updating
the example data. As TextAlive shows,such development (and runtime) environments should be connected to theInternet for sharing applications. Then,others can run the application as is but
also edit it seamlessly in the online
development environment to improve
it iteratively.
When no distinction exists be­t ween
the development and runtime environments, applications will always remain
editable. However, if we increase
the level of freedom and allow complete rewriting as implemented in
object-oriented environments such asMorphic,11 users might become overwhelmed. They might unintentionally
break the core part of the applications.
Whereas most live-programming en­
vironments aim to make everything
editable anytime, we used graphical representations as the boundary
between what can be easily customized and what cannot. We believe thatuser interface design will become aprocess to create applications withsuch a boundary that defines theappropriate flexibility.

As the number of data-driven
applications increases in the
coming decades, their development will involve much more data
manipulation that cannot (and shouldnot) be handled solely by programmers. PwE will be essential in suchdevelopment, inviting nonprogrammers into the iterative development
cycle and letting them customize
applications. There will be no static
distribution of programs, but they
will always remain somewhat customizable. Designing such flexibility will be the important task of userinterface design, and intuitive representations of example data will beessential to effective design. We hopethat the integrated graphical representations we introduced in this article serve as a good starting point forthe journey toward successful userinterface design in the age of dataintensive applications.

ACKNOWLEDGMENTS
The projects introduced in this article have been done in collaboration
with Xiang Cao, Sean McDirmid, andTomoyasu Nakano. The projects are supported partly by the Japan Science andTechnology Agency ERATO (Exploratory
Research for Advanced Technology)
program and CREST (Core Research for
Evolutionary Science and Technology).
More details are at http://junkato.jp/programming-with-examples.

REFERENCES
1. B.A. Myers, “Visual Programming,
Programming by Example, andProgram Visualization; A Taxonomy,” Proc. 1986 SIGCHI Conf. Human
Factors in Computing Systems (CHI 86),
1986, pp. 59–66.
2. D.C. Smith, “Pygmalion: An Executable Electronic Blackboard,”
J U LY 2 0 1 6 

41

21ST CENTURY USER INTERFACESABOUT THE AUTHORSJUN KATO is a researcher at Japan’s National Institute of Advanced Industrial
Science and Technology (AIST). His research interests include human–computerinteraction, particularly user interfaces for authoring interactive content. Kato
received a PhD in computer science from the University of Tokyo. Contact him
at jun.kato@aist.go.jp or via http://junkato.jp.

gotflaws?

TAKEO IGARASHI is a professor in the University of Tokyo’s Graduate Schoolof Information Science and Technology. His research interests include userinterfaces and computer graphics. Igarashi received a PhD in information engineering from the University of Tokyo. Contact him at takeo@acm.org.
MASATAKA GOTO is a prime senior researcher at AIST. His research interests
include music information research based on signal processing. Goto received
a PhD in engineering from Waseda University. Contact him at m.goto@aist.go.jp.

3.

4.

Find out more
and get involved:
cybersecurity.ieee.org

5.

6.

7.

42

COMPUTER 

Watch What I Do, MIT Press, 1993,
pp. 19–48.
J. Edwards, “Subtext: Uncovering
the Simplicity of Programming,”
Proc. 20th Ann. ACM SIGPLAN Conf.
Object-Oriented Programming, Systems, Languages, and Applications(OOPSLA 05), 2005, pp. 505–518.
K. Patel et al., “Gestalt: Integrated
Support for Implementation andAnalysis in Machine Learning,” Proc.
23rd Ann. ACM Symp. User Interface
Software and Technology (UIST 10),
2010, pp. 37–46.
A.J. Ko and B.A. Myers, “Barista:
An Implementation Framework forEnabling New Tools, Interaction
Techniques and Views in Code Editors,” Proc. 2006 SIGCHI Conf. Human
Factors in Computing Systems (CHI06), 2006, pp. 387–396.
J. Brandt et al., “Example-Centric
Programming: Integrating Web
Search into the Development Environment,” Proc. 2010 SIGCHI Conf.
Human Factors in Computing Systems(CHI 10), 2010, pp. 513–522.
J. Kato, D. Sakamoto, and T. Igarashi,“Picode: Inline Photos Representing

8.

9.

10.

11.

Posture Data in Source Code,” Proc.
2013 SIGCHI Conf. Human Factors in
Computing Systems (CHI 13), 2013,
pp. 3097–3100.
J. Kato, S. McDirmid, and X. Cao,
“DejaVu: Integrated Support forDeveloping Interactive Camera-­
Based Programs,” Proc. 25th Ann.
ACM Symp. User Interface Software
and Technology (UIST 12), 2012,
pp. 189–196.
J. Kato, T. Nakano, and M. Goto,“TextAlive: Integrated Design
Environment for Kinetic Typography,” Proc. 2015 SIGCHI Conf. Human
Factors in Computing Systems (CHI 15),
2015, pp. 3403–3412.
J. Kato and T. Igarashi, “VisionSketch: Integrated Support for
Example-­Centric Programming of
Image Processing Applications,”
Proc. Graphics Interface (GI 14), 2014,
pp. 115–122.
J.H. Maloney and R.B. Smith,“Directness and Liveness in theMorphic User Interface Construction
Environment,” Proc. 8th Ann. ACM
Symp. User Interface Software andTechnology (UIST 05), 1995, pp. 21–28.
W W W.CO M P U T E R .O R G /CO M P U T E R