Wi-Foto 2: Heterogeneous device controller
using Wi-Fi positioning and template matching
Ismail Arai, Soichiro Horimi, and Nobuhiko Nishio
Ubiquitous Computing and Networking Lab., Ritsumeikan University,
1-1-1, Noji-Higashi, Kusatsu, Shiga, 525-8577 Japan{ismail, sow}@ubi.cs.ritsumei.ac.jp, nishio@cs.ritsumei.ac.jp
Abstract. Improving usability of heterogeneous devices is important inpervasive computing. We have developed Wi-Foto, which enables users
to control such devices by using a tablet PC as a client. Wi-Foto has auser-friendly interface which has device controllers overlaid on a photo
depicting the interior of the room where the user is. In order to findthe most appropriate photo, Wi-Foto automatically locates the user by
Wi-Fi positioning but the localization would often fail because of inadequate accuracy. To solve this problem, we propose Wi-Foto 2, which
combines the Wi-Fi positioning with template matching based on a SIFT
algorithm. We confirm that the proposed method realizes accurate andfast positioning. Wi-Foto 2 also enables users to seamlessly select photos
through hierarchical and panoramic views.
Key words: Smart Computing, Device Control, WiFi positioning, SIFT

1Introduction

As heterogeneous devices in a room gain network connectivity, developments ofnovel device control methods, which are more comfortable than direct manipulation and IR remote controls, become active. In the tabletop computing field,
device controllers are displayed on a table, and a ceiling camera captures users’
gestures. [1] AR (Augmented Reality) systems, which are operated on tablet
PCs or smart phones, are user-friendly device controllers.[2] But, they are still
not practical because of as follows. In the former, users should be near the table
to control devices. The latter approach is unrealistic because markers have to beattached on all devices for the AR systems to recognize the devices. Furthermore,
the additional sensors spoil the interior design of rooms.
We have developed a user-friendly device control system, Wi-Foto[3], which
has device controllers overlaid on a current room photo. Wi-Fi fingerprints areobtained when the photos are taken, and the photos are stored together with thefingerprints, so that the system can provide a photo of the place closest to theuser. Therefore, users can control devices anywhere in the room, and the systemneeds no devices attached.
Wi-Foto requires a highly accurate indoor positioning mechanism to automatically choose a photo at the user’s current location from among many candidates. The most practical methods would be Wi-Fi positioning systems; however,

Adjunct Proceedigs of Pervasive 2010 - Demos. The authors maintain the copyright of the paper.

2I. Arai, S. Horimi and N. Nishio

2*."0*'-*3"1&-45%+'!"#$%&%'()"*+&%&'(&$)*+",$-,&*#.?$#1"$,225$-12#2

G299"3#$%&'(&$)*+",-,&*#.$/*0$.122#$#1"$,225$-12#2

@".-2*."
%:;<

H*-4#$%:;<

2*."0*'0%+&-%)'!"#$%&'(&$)*+",-,&*#.$/*0$.122#$#1"$34,,"*#$,225$-12#2
6/,."$#1"$%:;<$/*0$
0&.-9/=$0">&3"$32*#,299",$&32*.$/*0$#1"$,225$-12#2
#2431$/$0">&3"$$
32*#,299",$&32*

%&'(&$)*+",$-,&*#.?$@225$-12#2

%:;<?$
A#2,"0$,225$-12#2

C886$!D8E6F8

82431$/$0">&3"$32*#,299",!"#$%&%',*-.*-
AH(8$B"/#4,"$>"3#2,$+"*",/#2,

@225$-12#2?$("/#4,"$>"3#2,.?$$%&'(&$)*+",-,&*#.
%:;<

I&*0$#1"$-12#2$/*0$%:;<

A&5&9/,$%&'(&$$)*+",-,&*#.$,225$-12#2?$$%:;<?$$%&'(&$)*+",-,&*#?$
B"/#4,"$>"3#2,$%&'(&$62.&72*&*+
,225$-12#2.
8"5-9/#"$5/#31&*+

B"/#4,"$>"3#2,.$/*."0*'0%+&-%)'1*-."0*
@DA8$.",>&3"
@DA8$.",>&3"

Fig. 1. Design of Wi-Foto 2

their error of over 10 meters is too large for our purpose, especially when theuser is in a large room where several candidate photos are found or when radio
waves penetrate the walls around the user. Alternatively the user could manually select a photo but this would be less convenient for the user. Even though
there are some higher-accuracy indoor positioning systems, such as those using
ultrasonic signals or LED beacons, they have high infrastructure construction
costs. It’s desirable to realize other positioning methods with no modification ofinfrastructures.
To solve these problems, we combine Wi-Fi positioning and template matching which utilizes a SIFT (Scale Invariant Feature Transform) algorithm[4]. SIFT
analyzes a photo and generates a feature vector which has enough quantity of
key points of objects for object recognition. Wi-Foto is in a better condition fortemplate matching method because Wi-Foto needs to have room photos in advance. Furthermore, there is less additional infrastructure cost because the mostpersonal tablet PCs and phones have a camera. Generally, template matching
methods become slow when the number of candidate photos increases, and theaccuracy decreases when there are many similar candidate photos. By utilizing Wi-Fi positioning to roughly narrow down the candidate photos, we have
improved the performance of the template matching.
Furthermore, improved Wi-Foto (Wi-Foto 2) has a new feature: support forhierarchical and panoramic views. These views enable users to seamlessly move
to detailed or neighboring photos.

2

Design of Wi-Foto 2Fig. 1 shows the design of Wi-Foto 2. Wi-Foto 2 consists of the Wi-Foto client
and the Wi-Foto server. As heterogeneous devices are RESTful, a Wi-Foto clientWi-Foto 2!"#$%&''()*$+&,$-''.

!3#$=&,$3,;49,$9')+-'22,-$'>$+&,$?:?

!/#$012'"34)*$+&,$54674$8)*,-1-4)+%$$$$$$$")3$+&,$-''.$1&'+'

!9#$:,;49,$9')+-'22,-$49')%$<4+&$$$$$$$+&,$%+'-,3$-''.$1&'+'!,#$@4,-"-9&49"2$;4,<$!>#$?")'-".49$;4,<$3Fig. 2. Democan get device state and control devices by HTTP GET/PUT methods. Also,
the Wi-Foto client discovers Wi-Foto servers by multicast DNS. The Wi-Fotoclient automatically collects Wi-Fi fingerprints as soon as it is activated.
In the device registration mode, users register device controller with the WiFoto server by sending Wi-Fi fingerprints, the room photo and the WADL (WebApplication Description Language) of the device control services. The WADL
describes how to communicate with a REST service. The Wi-Foto server receives
the Wi-Fi fingerprints and the room photo from Wi-Foto client and calculates
SIFT feature vectors of the photo. The server stores the Wi-Fi fingerprints andthe feature vectors in the DB. Then the server receives the WADL and stores it
into the DB.
In the device controller mode, the Wi-Foto client also automatically collects
Wi-Fi fingerprints. After shooting a current room, the client sends the Wi-Fi
fingerprints and the room photo to the Wi-Foto server. As the size of featurevectors is bigger than a size of a JPEG file, the client sends the server not
the feature vectors but the photo. Wi-Fi positioning module receives the WiFi fingerprints, locates the user by calculating similarity of Wi-Fi fingerprints,
and chooses several candidates. The template matching module chooses a photoamong the candidates by calculating the similarity of their feature vectors basedon SIFT algorithm. As all the feature vectors of candidates are stored in theDB, the template matching module doesn’t need to dynamically generate featurevectors.

3Demo

We implemented a Wi-Foto 2 client for Google Android phones. In this demo, auser experiences the quick retrieval of a correct room photo and the user-friendly

4I. Arai, S. Horimi and N. Nishio

device control interface. The target room photos and the device control services
are set up in advance. Steps for using the client are as follows.
1. Activating the Wi-Foto client.
The client displays camera image (Fig. 2 (a)) and automatically collect Wi-Fi
fingerprints.
2. Shooting the room picture which includes devices.
After shooting the room, the client sends Wi-Fi fingerprints and the room
photo. (Fig. 2 (b))
3. Wi-Fi positioning and template matching.
After finishing the Wi-Fi positioning and the template matching, the server
sends the device controller icons and the room photo back to the client.
4. Displaying the device controller icons and the room picture.
The client parses the device controller information and overlays the device
controller icons on the room photo. (Fig. 2 (c))
5. Controlling devices. (Fig. 2 (d))
After pressing the device controller icon, the device controller appears.
6. Panoramic and hierarchical view (Fig. 2 (e, f))Among new features of the Wi-Foto 2, users can instantly add more photos
and create the panoramic and hierarchical views of shots from the cameraon the Android smart phone.

4Conclusion

Toward a high usability heterogeneous device control system, we have improved
Wi-Foto by combining Wi-Fi positioning and template matching based on aSIFT algorithm. Furthermore, we have implemented the hierarchical view andpanoramic view to enable users to seamlessly reach the target devices. In thefuture, we will evaluate and confirm that Wi-Foto 2 enables users to easily andquickly control heterogeneous devices. And we will introduce this system into
many diﬀerent environments and evaluate the performance of the Wi-Fi positioning and template matching.

References
1. Seifried, T., Rendl, C., Perteneder, F., Leitner, J., Haller, M., Sakamoto, D., Kato,J., Inami, M., Scott, S.D.: CRISTAL: Design and Implementation of a Remote
Control System Based on a Multi-touch Display. the ACM International Conferenceon Interactive Tabletops and Surfaces (2009)
2. Suzuki, G., Aoki, S., Iwamoto, T., Maruyama, D., Koda, T., Kohtake, N., Takashio,K., Tokuda, H.: u-Photo: Interacting with Pervasive Services using Digital Still Images. The 3rd International Conference of Pervasive Computing (Pervasive) (2005)
190–207
3. Arai, I., Atsuhito, I., Shiraishi, K., Horimi, S., Nishio, N.: Wi-Foto: A Universal
Service Controller System for Distributed Smart Environment. Adjunct Proceedingsof Pervasive (May 2009) 197–200
4. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International
Journal of Computer Vision 60(2) (Nov 2004) 91–110