User Interfaces
for Live Programming
Jun Kato
https://junkato.jp
Researcher,LIVE 2017 Keynote, 10/24/2017

Jun Katojunkato

https://junkato.jp

Research Topic

Computer Science (Human-Computer Interaction, Programming Language)
Phybots

DejaVuPicodeIt’s Alive!

VisionSketch

TextAlivef3.js

ACM DIS’12

ACM UIST’12

ACM CHI’13

ACM PLDI’13

GI’14

ACM CHI’15

ACM DIS’17• Created Tools and Environments for Creativity/Productivity Support• Application Domains: Prototyping, Physical Computing, Computer
Vision, Robots, Internet of Things, Animation Authoring, ...
• Founded SIGPX (SIG on Programming Experience) https://sigpx.org/en
User Interfaces for Live Programming

SIGPX

https://sigpx.org/en

A group of researchers/engineers/teachers in Japan, studying ...

Programming Experience in the intersection of HCI/PL/SE

Meet & Discuss

Collect
https://scrapbox.io/ProgrammingExperience/

PublishPX Special Issue in IPSJ Journal (Nov 2017)
1st meetup, 2/27/2016
User Interfaces for Live Programming

Emerging Research on Programming Experience:
From Language Design to Industrial Applications

Today, I’m going to talk about ...
• What is Live Programming?
• UIs for Live Programming with end-users• UIs for Live Programming of this material world• UIs for Live Programming with time travel• Live Programming as User Interface research
It is about ...

It is not about ...

• Showcase of user interfaces for programming• No λ or greek symbols in slides• Not only my work but also others’ notable work

• Not a consensus in the field (it’s ongoing!)• Discussion on live programming system design• No peer review involved (my personal view)

User Interfaces for Live ProgrammingToday, I’m going to talk about ...
• What is Live Programming?
• UIs for Live Programming with end-users• UIs for Live Programming of this material world• UIs for Live Programming with time travel• Live Programming as User Interface research

User Interfaces for Live Programming

What is Live Programming?
• Programming experience• Continuous feedback• Concrete information• Early examples in VPL and OOP• Attracting much attention these days

Not new but hot!
User Interfaces for Live ProgrammingText-based Programming
Dartmouth BASIC [1964]

Visual Studio Code [as of today]• Text-based editor• Text-based debugger• Text-based ...

IDEs haven’t changed much
User Interfaces for Live Programming

With some exceptions ...
[since 1986]
Interface Builder• A tool for NeXT UIdevelopment• Later integrated
into Xcode• Many IDEs have
similar built-intools

User Interfaces for Live Programming

Visual Programming
Scratch
MITConManHaberli[SIGGRAPH 1988]• Mostly dealing with symbolic representations of programs• Often considered as tools for novices and good for education• Dataflow languages: early examples of live programming
User Interfaces for Live Programming

Character-based UIs or Graphical UIs?

orModernLive Programming as a hybrid approach
User Interfaces for Live Programming

User Interfaces with Text and Graphics

TouchDevelop[PLDI 2014]

https://touchdevelop.com
User Interfaces for Live Programming

User Interfaces with Text and Graphics

f3.js[DIS 2017]

http://f3js.org
User Interfaces for Live Programming

Character-based UIs
and Graphical UIs• It’s like text and figures inresearch papers• Text is good at abstraction• Graphics are good atpresenting concrete information
Integrated Graphical Representations[2014, dissertation] [2016]

They complement each other
User Interfaces for Live Programming

In Live Programming systems, we ...
• first have vague ideas• then explore the ideas with concrete examples• gradually start turning the ideas into programs

Live Programming requires
decent UIs for exploring the problem space
User Interfaces for Live Programming

UIs for Live Programming should ...
• avoid sudden changes in the program behavior• keep the program and its output relevant• allow continuously exploring the problem space

Appropriate user interface design differsfrom application to application
User Interfaces for Live Programming

Take-home message
When designing live programming systems ...

Don’t be afraid to be domain-specific• Good UI is always specifically designed for the target domain• It might be like replaying the history of end-user computing
in the domain of programming• We might need PX workbench (cf. language workbench)

Cf. Programming eXperience Toolkit (PXT)
https://github.com/microsoft/pxt
User Interfaces for Live ProgrammingToday, I’m going to talk about ...
• What is Live Programming?
• UIs for Live Programming with end-users• UIs for Live Programming of this material world• UIs for Live Programming with time travel• Live Programming as User Interface research

User Interfaces for Live Programming

UIs for Live Programming
Good mixture of text-based and graphical user interfaces

User Interfaces for Live Programming

UIs for Live Programming
Why not expose GUI to users so that they can edit programs?

Live Tuning [LIVE 2016]
User Interfaces for Live ProgrammingMode Switch between “Use” and “Build”

Halo in Morphic UI

Pause button in TouchDevelop

What if we add another layer for users?
Promoting universal usability with multi-layer interface design
Ben Shneiderman [2002]
User Interfaces for Live ProgrammingTextAlive[CHI 2015] http://textalive.jp
User Interfaces for Live ProgrammingTextAlive[CHI 2015] http://textalive.jpUser Interfaces for Live ProgrammingTextAlive[CHI 2015] http://textalive.jp
User Interfaces for Live Programming

Co-hosting UIs for programmers and users

Literate Programmingin Jupyter (Ipython) Notebook

Inline Photos
in Picode [CHI 2013]

User Interfaces for Live ProgrammingMerging UIs for programmers and users(direct manipulation on program output)

Sketch-n-Sketch, Hempel et al.
[UIST 2016 etc.]

Para, Jacobs et al.
[CHI 2017]

User Interfaces for Live Programming

Take-home message
When designing live programming systems ...

How about making the ladder of expertise?
• From live programming for programmers• To live programming for the community of people

UseCustomize

Write code

Multi-layer

Co-hostUser Interfaces for Live Programming

Merge

Today, I’m going to talk about ...
• What is Live Programming?
• UIs for Live Programming with end-users• UIs for Live Programming of this material world• UIs for Live Programming with time travel• Live Programming as User Interface research

User Interfaces for Live Programming

What is “live” and what not?
• System response time:• Computation• Network• Touch display response
How Live are Live Programming
Systems?
Rein et al. [PX 2016]• Reflex time:• Visual 0.25s• Audio 0.17s• Touch 0.15s• and more ...
• 3D printers and laser cutters• Shape changing devices and robots• Taste/smell interfaces

Various kinds of “latencies”
User Interfaces for Live Programming

Printing physical computing devices
f3.js [DIS 2017]

Slow “framerates” prevent live feedback
User Interfaces for Live Programming

Slow display
Daniel Saakes et al. [SIGGRAPH Etech 2010]

Slow “framerates” can be useful, tooUser Interfaces for Live Programming

f3.js: IoT apps with enclosures from single code base
[DIS 2017] http://f3js.org
User Interfaces for Live Programming

MakeCode for BBC micro:bit, Microsoft Research [2017]
http://makecode.microbit.org
User Interfaces for Live ProgrammingPicode: inline photos representing posture data in source code
[CHI 2013]

User Interfaces for Live Programming

Take-home message
When designing live programming systems ...

Deceiving users’ perception is a good thing• As long as the lie is reasonable• The actual “framerate” can be very slow
• Emulating, or sometimes even pretending, is needed toprovide the continuous feedback• Make full use of five senses in programming environments

User Interfaces for Live ProgrammingToday, I’m going to talk about ...
• What is Live Programming?
• UIs for Live Programming with end-users• UIs for Live Programming of this material world• UIs for Live Programming with time travel• Live Programming as User Interface research

User Interfaces for Live Programming

“Live” is not always about “now”• UIs for exploring and modifying the past
• UIs for predicting and choosing the future• Absolute vs semantic timeline

User Interfaces for Live Programming

DejaVu[UIST 2012]
User Interfaces for Live Programming

Replay & Refresh• DejaVu

DejaVu[UIST 2012]
User Interfaces for Live Programming

VisionSketch[GI 2012]
User Interfaces for Live Programming

“Live” is not always about “now”• UIs for exploring and modifying the past
• UIs for predicting and choosing the future• Absolute vs semantic timeline

User Interfaces for Live Programming

Light Table – a new IDE concept, Chris Granger [2012]
http://www.chris-granger.com/2012/04/12/light-table-a-new-ide-concept/

User Interfaces for Live Programming

Many-Worlds Browsing for Control of Multibody Dynamics
Twigg et al. [SIGGRAPH 2007]
User Interfaces for Live Programming

Rktcr, McCann (TCHOW)[2013]
User Interfaces for Live Programming

“Live” is not always about “now”• UIs for exploring and modifying the past
• UIs for predicting and choosing the future• Absolute vs semantic timeline

User Interfaces for Live Programming

Absolute time vs semantic time

TextAlive [CHI 2015]

DejaVu [UIST 2012]
User Interfaces for Live Programming

Take-home message
When designing live programming systems ...

Try providing good sense of time
• Enable time travel to find critical timings in the history• Allow editing the code and program input to explore futures

Replay & Refresh“Many worlds”

Timeline

Superspeed & slowmo

Stroboscopic visualization

for absolute/semantic time

User Interfaces for Live ProgrammingToday, I’m going to talk about ...
• What is Live Programming?
• UIs for Live Programming with end-users• UIs for Live Programming of this material world• UIs for Live Programming with time travel• Live Programming as User Interface research

User Interfaces for Live Programming

Take-home message

Live Programming research as
User Interface research• Don’t be afraid to be domain-specific• How about making the ladder of expertise?
• Deceiving users’ perception is a good thing• Try providing good sense of timeIt’s not only about language design,
a single user, a single UI,
but about designing the whole experience
User Interfaces for Live Programming

User Interfaces
for Live Programming
Jun Kato
https://junkato.jp
Researcher,LIVE 2017 Keynote, 10/24/2017

References
1. Jun Kato, Masataka Goto, “f3.js: A Parametric Design Tool for Physical Computing Devices for Both Interaction
Designers and End-users", In Proceedings of the 2017 Conference on Designing Interactive Systems. pp.1099-1110, 2017.
2. Jun Kato, Masataka Goto, "Live Tuning: Expanding Live Programming Benefits to Non-Programmers", In Proceedings ofthe Second Workshop on Live Programming Systems. 2016.
3. Jun Kato, Takeo Igarashi, Masataka Goto, "Programming with Examples to Develop Data-Intensive User Interfaces",
In Computer 49(7). pp.34-42, Jul. 2016. Special Issue on 21st User Interfaces.
4. Jun Kato, Tomoyasu Nakano, Masataka Goto, "TextAlive: Integrated Design Environment for Kinetic Typography",
In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. pp.3403-3412, 2015. ACM CHI2015 Best Paper Honorable Mention Award.
5. Jun Kato, Takeo Igarashi, "VisionSketch: Integrated Support for Example-centric Programming of Image Processing
Applications", In Proceedings of the 2014 Graphics Interface Conference. pp.115-122, 2014.
6. Sebastian Burckhardt, Manuel Fahndrich, Peli Halleux, Sean McDirmid, Michal Moskal, Nikolai Tillmann, Jun Kato, "It's Alive!
Continuous Feedback in UI Programming", In PLDI '13: Proceedings of the 34th ACM SIGPLAN Conference on
Programming Language Design and Implementation. pp.95-104, 2013.
7. Jun Kato, Daisuke Sakamoto, Takeo Igarashi, "Picode: Inline Photos Representing Posture Data in Source Code", In CHI'13: Proceedings of the SIGCHI conference on Human Factors in Computing Systems. pp.3097-3100, Apr. 2013. ACM CHI 2013
Best Paper Honorable Mention Award.
8. Jun Kato, Sean McDirmid, Xiang Cao, "DejaVu: Integrated Support for Developing Interactive Camera-Based Programs",
In UIST '12: Proceedings of the 25th annual ACM symposium on User Interface Software and Technology. pp.189-196, Oct.
2012.
User Interfaces for Live Programming