Reactile: Programming Swarm User Interfaces
through Direct Physical Manipulation

Ryo Suzuki, Jun Kato, Mark Gross, Tom Yeh

Swarm User Interfaces

RobotDynamic MediaApplication
Paradigm

Swarm Robots

Swarm UIProgramming
Paradigm

RobotProgramming?

Programming = Authoring Dynamic Materials

Sketches from “Human Representation of Thoughts” by Bret Victor [UIST 2014 Keynote]Authoring Dynamic Behaviors in GUI

Authoring Dynamic Behaviors in GUI

Robot Programming

Swarm UI Programming

computer screen

direct physical manipulation

low-level controls

high-level interface design

Swarm UI Programming

1. Summary2. Related Work
3. Design and Workflow
4. Implementation
5. Limitations and Future Work

Direct Manipulation in Physical SpaceInteracting with the Pre-programmed Behaviors

DigitalDesk[Wellner 1993]

I/O Bulb[Underkoﬄer and Ishii1998]

Mockup Builder[Bruno et al. 2012]

Toolkits for Robot Programming

Gulf of Execution (Computer

LEGO Mindstorms

Phybots[Kato et al. 2012]

Physical World)

GUI Robots[Guiness et al. 2017]

Tangible Programming Languages

Program Structure

AlgoBlock[Suzuki and Kato 1995]

Program States

Quetzal[Horn and Jacob 2007]

Osmo (Strawbies)[Hu et al. 2015]

Programming by Demonstration

Simple Repetive Operations for a Single RobotTopobo[Raﬄe et al. 2004]

Robot Programming by Demonstration[Billard et al. 2007]

1. Summary
2. Related Work

3. Design and Workflow
4. Implementation
5. Limitations and Future WorkStep 1Create Elements

Step 2
Abstract Attributes

Step 3Specify Behaviors

Step 4
Propagate Changes

<circle id=“Sun”/><circle id=“Earth”/>

<circle id=“Sun”
radius = 5/><circle id=“Earth”
radius = 1
distance = 10/>10

<circle id=“Sun”
radius = 5/><circle id=“Earth”
radius = 1distance = 10
angle = time/>angle

<circle id=“Sun”
radius = mass/><circle id=“Earth”
radius = 1
distance = 1/mass
angle = time/>

Step 1
Create Elements

Step 2
Abstract Attributes

Step 3Specify Behaviors

Step 4
Propagate Changes

Step 1Create Elements

Step 2
Abstract Attributes

Step 3Specify Behaviors

Step 4
Propagate Changes

Step 1Create Elements

Step 2
Abstract Attributes<triangle x="0" y="10"/><triangle x="0" y="10"/><rectangle x="5" y="15"/><arrow x="0" y="10"/>

Step 3Specify Behaviors

Step 4
Propagate Changes

triangle

rectangle

circleStep 1Create Elements

Step 2
Abstract Attributes

var angle = 30
<arrow angle={ angle } />var scale = 2<rectangle scale={ scale } />var length = 10
<line length={ length } />Step 3Specify Behaviors

Step 4
Propagate Changes

Step 1Create Elements

Step 2
Abstract Attributes

var angle = 30
<arrow angle={ angle } />var scale = 2<rectangle scale={ scale } />var length = 10
<line length={ length } />Step 3Specify Behaviors

Step 4
Propagate Changes

circle: diameterrectangle: width → scale → heightpoint: x → angle → yStep 1
Create Elements

Step 2
Abstract Attributes<slider value={ value } /><arrow angle={ angle } />bind(angle, value / 10)<slider value={ value } /><arrow angle={ angle } />Step 3Specify Behaviors

Step 4
Propagate Changes

Step 1Create Elements

Step 2
Abstract Attributes

Step 3Specify Behaviors

Step 4
Propagate Changes

bind(rectangle.width, point.x - 5)

1. Summary
2. Related Work
3. Design and Workflow

4. Implementation
5. Limitations and Future Work1st Layer

2nd Layer

3rd Layer

4th Layer

Side View

Top View

Side View

Top View

Side View

Top View

Side View

Top View

Side View

Top View1. Summary
2. Related Work
3. Interaction Design
4. Implementation

5. Limitations and Future Work

Limitations
Slow refresh rate

Minimum distanceLimitations
Slow refresh rate

Minimum distanceFuture WorkReactile: Programming Swarm User Interfaces
through Direct Physical Manipulation

Ryo Suzuki, Jun Kato, Mark Gross, Tom Yeh
https://github.com/ryosuzuki/reactile