A Toolkit for Easy Development of Mobile Robot Applications with Visual Markers and a Ceiling Camera
1Daisuke Sakamoto1&3, Jun Kato1&3, Masahiko Inami2&3, Takeo Igarashi1&3
Graduate School of Information 2Graduate School of Media De- 3JST, ERATO, IGARASHI DesScience and Technology,
sign, Keio University (KMD)
ign UI Project, 1-28-1, KoishiThe University of Tokyo,4-1-1 Hiyoshi, Kohoku-ku, Yokawa, Bunkyo-ku, Tokyo, 1127-3-1 Hongo, Bunkyo-ku, Tokohama 223-8526, JAPAN0002 Japankyo, 113-0033 Japan{sakamoto | kato | inami}@designinterface.jp, takeo@acm.org

ABSTRACTWe created a toolkit for software engineers to rapidly develop various robot applications. We distributed the prototype system to students in a Human Computer Interaction(HCI) course and observed that students without any priorexperience in robot application development successfully
developed interesting applications.
ACM Classification: H5.2 [Information interfaces andpresentation]: User Interfaces. - Prototyping. I.2.9[ARTIFICIAL INTELLIGENCE]: Robotics. - Commercial
robots and applications
General terms: Design, Human Factors.
Keywords: toolkit, prototyping, robot application, mobile
robot, home robot
INTRODUCTION

Small mobile robots are now consumer products. Variouscheap and reliable commercial products such as iRobot
Rooomba [1], NetTansor [2], and Lego Mindstorm Robots[3] are available. Furthermore, most of these robots support
remote controlling APIs. However, the development ofrobot controlling software is difficult. Basic tasks such asprocessing of sensor input, recognition, localization (knowing the location of objects and robots), and navigation(moving the robot to a specific location) require a lot ofwork. What’s worse, the difficulties of these tasks vary
between hardware platforms, target task or goal, physicalenvironment and context, etc.
On the other hand, computers are useless without applications. Various applications are now available for a given
computer platform to satisfy individual user needs. However, this is not yet true for robots. Most robots come with
proprietary software and their development is limited torobotics experts. This situation is unfortunate. We envisionthat various useful and interesting applications will emerge
if a variety of people such as HCI researchers and interaction designers can develop applications by themselves.
To ease this problem, we developed a toolkit to make iteasy to write an application that remotely controls mobilerobots in a given environment. To achieve our goal, weCopyright is held by the owner/author(s).
UIST’09, October 4–7, 2009, Victoria, BC, Canada.

Figure 1 Our toolkit only requires a computer, aweb-camera, visual markers, and robots.

designed a system to be as simple as possible to make it
accessible to novices and distributed it as open source toencourage active participation.
IMPLEMENTATION

The toolkit consists of robots and objects on the floor, aceiling mounted camera, and a control computer. The camera is wire-connected to the control computer and the robots are connected wirelessly via Bluetooth or TCP/IP network. Figure 2 shows a snapshot of the setup.
Mobile Robots and Drivers

Our toolkit currently supports three commercially available
robots: iRobot Roomba, Lego mindstorms NXT, and Bandai NetTansor. These robots are small, easy to buy and use,and have the ability to move around the home environment(Figure 2). Our toolkit includes drivers for all supported
robots, and the programmer can control the various robots
with a unified API. A robot driver is managed as a Javaclass, and each robot is associated with a corresponding
driver in the configuration file. This makes it possible touse various robots at the same time. We also designed thedrivers’ API in a way that makes it easy to add a driver fora new robot by writing the robot specific code (e.g. communication protocol, low-level control command, etc.) ineach deriver class.

Figure 2 Example setup of the toolkit
Vision sensor with visual markers

The toolkit uses and includes NyARToolkit for Java [4](wrapper of ARToolkit [5]) to detect the locations of robots
and objects. It detects the robots’ and objects’ IDs and theirlocations by using visual markers in a given image. The
visual marker is a square image like a 2D barcode (Figure2). When the toolkit detects visual markers, it returns theirlocation (where the marker is in the captured image), orientation, and the ID of the robot and/or objects in the world.
Configuration

The user needs to set the following elements in the configuration XML file:
1. Camera connection
2. Robots / objects and world coupling
The toolkit needs to know which visual marker corresponds
to which robot. For each robot, the user specifies the name(1), driver type (2), connection type (TCP/IP, Bluetooth,etc) (3), and visual marker name (4) in the configuration
file. For the object configuration, just the object’s name andvisual marker name are required.
USER STUDY

The goal of our toolkit is to support the development ofrobot applications for all levels of robot programming. To
see if our goal can be achieved, we performed a user study
in a course. This section describes the procedure and resultsof the study.
Procedure

We provided our toolkit and “robot kits” to computerscience graduate students in a Human Computer Interaction(HCI) course. Fifteen university students enrolled in thecourse and formed eleven groups. We provided our toolkit
through a web site that included an installation guide andprogramming guide. The documents were provided as wiki
pages, and it had the capability of receiving questions fromthe students.
Results

We confirmed that the toolkit helped students to develop
their robot applications. All students completed their projects and successfully developed robot applications. It wasalso possible to develop an application that used multiple

Figure 3 Robot applications that students createdrobots. We were surprised to see that some students developed applications we had never expected. Figure 3 showssome of the robot application that the students created.
CONCLUSION

We proposed a toolkit for mobile robots that uses visionmarkers and a ceiling camera as a vision sensor. It allows
non-robotics experts such as software engineers, interface
designers, and interaction designers to develop their own
robot applications. The key aspect of our system is that thephysical locations of robots and objects are defined in thescreen space and the programmer can handle them seamlessly with GUI events. This significantly lowers the barrier
while still supporting a large variety of applications. We
provided the toolkit to the computer science graduate students, and they successfully developed interesting robot
applications.
REFERENCES

1. Bandaico,ltd.
NetTansor.
http://www.roboken.channel.or.jp/nettansor/.
2. iRobot. Roomba. http://www.irobot.com/consumer/.
3. Lego. Lego Mindstorm. http://mindstorms.lego.com/.
4. NyARToolkit.
http://nyatla.jp/nyartoolkit/wiki/index.php?NyARToolki
t%20for%20Java.en.
5. Kato, H. and Billinghurst, M., Marker tracking andHMD calibration for a video-based augmentedreality
conferencing system. In Proceedings of the 2nd IEEEand ACM International Workshop on Augmented Reality, 1999.(IWAR'99), (1999), 85-94.