Roboko: Integrating Visual
Representations into a Text-based
Programming Environment
Jun Kato
The University of Tokyo
i@junkato.jp
http://junkato.jp/roboko

1. Project Goal
Current programming environments use textual or symbolic(e.g. flow chart) representations. This paradigm has worked
well for programming abstract behaviors of software, but isproblematic for programming robots that work in the realenvironment. Robots have a physical embodiment and workin the real world. Textual or symbolic representation does
not help people to understand how the robot works in runtime. For example, if the programmer wants to program ahigh degree-of-freedom (high-DOF) robot to reach an arm,each joint angle must be specified in the code. However, it
is difficult to understand that the code is for arm reachingby reading the code. Our goal is to address this problem
by integrating photograph-based visual representations
into a text-based development environment. The integrated
development environment (IDE) allows programming-byexample style instructions for robots.

Then, the IDE retrieves corresponding pose data from therobot. Finally, the captured pose appears as the photographwith an arbitrary name (e.g. “New pose (1)”) in the pose
library and the IDE leaves “capture mode.” Within the pose
library, the programmer can also rename existing robot pose(e.g. “Initial pose,” “Right hand up,” etc.,) load existing
robot pose to see the robot in action, view raw pose values,
or delete existing poses.

2. Technical breakthroughOur prototype implementation of the IDE consists of
two main components: a code editor and an pose library(Figure 1). Each photograph-based visual representationis shown inline in the editor and is bound to a set of raw
pose data. In this section, we will describe the workflow
of the programmer with our development environment andbriefly explain its implementation.
First, the programmer captures a photograph and a setof raw pose data. He clicks “Capture” button in the pose
library and the IDE enters “capture mode.” During thismode, he can move every joint of the robot, whose actuators
are equipped with rotary encoders, to make a desired
pose. When he is satisfied with the current pose, he takesa photograph of the robot with a web camera (Figure 2.)7

Microsoft Research CORE7 Project Summary Booklet

Second, the programmer starts coding. While he can write
code just as with a standard text-based programming
environment, he can also drag and drop a photograph fromthe pose library to the code editor and the photograph isshown inline in the code editor. Behind the scene, theIDE inserts textual representation of the photograph (e.g.
Roboko.pose(“Right hand up”)) to the dropped positionin the source code. Then, it parses the source code to getan abstract syntax tree (AST.) Next, the view is updated
according to the AST, showing the corresponding
photograph where its textual representation is found. Whenthe parsing is failed or the corresponding pose data is notfound, the view is not updated.
During the coding phase, the programmer can benefit froma software library tightly coupled with the IDE. The libraryprovides a set of API that enables easy control of robots.
The main functions are shown below:
Pose Robot.getPose() – returns the current pose data.
float Pose.distance(Pose pose) – calculates the distancebetween the two poses. [0.0-1.0]
boolean Pose.eq (Pose pose, float threshold) – returns
whether the two poses can be thought of as identical or not.
(returns whether the distance between the two poses is lessthan the threshold or not.)boolean Robot.setPose(Pose pose) – set the current pose to
the specified data.
Action Robot.action() – start building an action for therobot.
Action Action.pose(Pose pose) – add this pose to the endof this action.
Action Action.wait(int ms) – wait for the specified time at
the end of this action.
ActionResult Action.play() – play this action.
When the coding is finished, the programmer runs theprogram by clicking “Run” button. The simple workflow
described in this section supports iterative cycle ofcapturing poses, programming and running applications.
It is expected to enable prototyping process not only for
professional programmers but also for novice programmers
without prior knowledge of robotics.

3. Innovative Applications
While the initial goal of this project focused on programming
robots, developing the prototype of the IDE made us awarethat our method is also applicable to Kinect programming.
We extended the prototype IDE to support Kinect-basedskeletal tracking and allow the programmer to easily makepose-based applications (Figure 3.)
Given that the behavior of robot and Kinect-based
applications is easy to understand because of the physicalembodiment, our IDE might be a good starting point for
programming learners. Robot and Kinect are both very hot
topic among interaction designers, and thus, our IDE canhelp their casual development as well.

4. Academic Achievement
We plan to submit this work to CHI’13: ACM SIGCHI 31th
Conference on Human Factors in Computing Systems, atop-tier conference in the Human-Computer Interaction
research community. [CHI13]

5. Achievement in Talent Fostering
The principal investigator of the project, Jun Kato, is a Ph.Dstudent. He attended an internship at Microsoft Research
Asia for 3 months.

6. Collaboration with Microsoft Research
During his internship, Jun collaborated with Dr. Xiang
Cao to investigate potential of this research project. He
developed Kinect extension introduced at 3. InnovativeApplications. Additionally, he started another project andsubmitted the result to UIST’12: ACM 25th Symposium on
User Interface Software and Technology.

7. Project Development
The project is still ongoing and there are much work to
be done including user study and further investigation.

IJARC CORE7 project summary booklet

8Promising direction includes the extension of the IDE tohelp broader activity related to the hardware development.
Current contribution of the project is about the IDE whose
text editor is capable of showing photographs captured bya camera, each of which represents a pose of a robot or aperson. It frees the programmer from struggling with raw
numerical parameters. It is a kind of "what you see is what
you get" interface and is expected to be used by novice
programmers. Thereupon, using photographs benefits not
only programming robot applications but also remembering
how the robot is constructed. For instance, there is an
example application with a LEGO robot in which the robot
waves its hand. Though the LEGO robot can be constructedin many ways with LEGO bricks, photos bound to raw pose
data tell the others how the robot is constructed.
During the development of a robot application, theprogrammer often changes form factors of the robot. It isa kind of iterative cycle of building a robot (hardware) andwriting its controlling code (software). In such development
process, hardware and software evolve together. Though,while there is much support for tracking changes of codesuch as Subversion and Git, there's no apparent support
for tracking changes of hardware. When the IDE takes
snapshot photos of the hardware (and their corresponding
pose data) during the development process, it can help theiterative cycle more effectively compared to the existingdevelopment environment.
In addition, when the application development is finished,
the IDE can not only build the executable binary but also
output photo-based step-by-step guidance that tells how
the robot can be constructed. The binary and the guidance
can be distributed together so that other people can run theapplication at their home.

8. Publications
Paper publication
1) Roboko: Integrating Visual Representations into a Textbased Programming Environment, to be submitted
to CHI’13: ACM SIGCHI 31th Conference on HumanFactors in Computing Systems

9IJARC CORE7 project summary booklet