Smart-home control by using half-reflective AR mirror display
ZIQING FENG†

Mana SASAGAWA†

Itiro SIIO†

Yasushi MATOBA�†

Graduate School of Human and Social Sciences, Ochanomizu University

1 Abstract
In this paper, we present a smart-home control
interface system by using a half-reflective AR mirrordisplay. When the user stands in front of thehalf-mirror display, the user can see reflected imagesof target IoT devices such as smart-home lamps. The
user can manipulate these devices by touching themirror surface, selecting a menu that appeared at thetouchpoint. We also have a user correction method
for our system, the user can move the reflection
position anytime to correct the reflection position.
The display has a couple of stereo cameras at bothsides of the frame, and a touch sensor on the display
surface. Because the system is made of an all-in-one
configuration, the calibration and registration efforts
are reducing significantly.

Figure 1: Left: Top view

Right: User’s view

2 Background
With the development of technology, smart hometechnology and augmented reality technology havebecome an indispensable part of our lives. In thefield of smart home, most of what we know is tocontrol smart-home furniture through voice, such as
Siri Home pod1 , Amazon Alexa 2 , Xiao Ai3 , andso on. In the field of augmented reality, many
software developers have developed very successful
AR applications, such as Pokémon Go4 , which is very
well-known in the game field, or Ikea place5 developedby a smart home company Ikea, which is helping
people choose more suitable furniture for their room.
It would be great if AR and smart home systems
could be integrated, so we thought about why can’t
the mirror be made into a smart home touch control
system? Before this research, there were alreadysome reality augmentation researches for mirrors. For
example, the MR-Mirror[1], which allows users tointeract with virtual objects in front of the mirror,
and Mirror Mirror[2], which allows users to see
different clothes designs in front of the mirror throughprojection, etc.

3 Proposed System
Figure 1 shows our proposed system. When theuser stands in front of our system, the user can see
the reflection point of the loT devices behind the user
1 https://www.apple.com/jp/homepod-mini/
2 https://developer.amazon.com/ja-JP/alexa
3 https://xiaoai.mi.com
4 https://pokemongolive.com/ja/
5 https://apps.apple.com/us/app/ikea-place/id1279244498

Figure 2: Screenshots from a user’s view.
like figure 1 right, and user can touch the screen to
manipulate those devices as figure 1 left. Figure 2
shows the actual screenshots from a user’s view. She
can locate the target device by touching the surface,and selecting a menu to control it.
3.1

System ConfigurationFigure 3 shows our proposed system configuration�
Our system is consists of 4 sections, the first section isa display for showing target marks and menus at theirreflection position prepared by a computer, the second
section is the half-mirror for reflection as a mirror but
also can let the user see the display screen, the third
section is a touch frame to get user’s touch position,and the last one is two cameras for getting the user’s
head position.
3.2

Data Processing

The data processing flow of this system is shownin figure 5.
First, two stereo cameras obtainthe user’s image, and then the 2d coordinates of
the user’s head are obtained through openpose 6 ,6 https://github.com/CMU-Perceptual-Computing-Lab/openpose

Figure 5: Data Processing Flow

Figure 3: System Configurationand then 3d coordinates in space are calculated byusing the method of our research (will be explainedin the next chapter).
At this time, the 3D
coordinates of the user’s head will be transmitted
to the processing program through MQTT 1 . In
the processing program, the user’s touch position isobtained through the touch-frame, and then it isjudged whether the user has touched the reflectedpoint. If touched, the menu is displayed, and theuser can control the smart home through the menu.
The processing program issues commands to control
devices such as smart LED lights through Apple theHomeKit library.

Figure 6: Undistortion
figure 6. Therefore, before the calculation, thecamera distortion must be corrected first so that thesubsequent calculations are more accurate. To correct
the distortion of the camera on the left side of thefigure 6 to the right of figure 6, we used 2nd orderdistortion calibration by determining parameters fromthe checker pattern.
Now we get the correct 2d coordinates throughdistortion correction. Then, we can calculate theuser’s head position of space coordinates. Throughmeasurement, the distance between the two cameras
is 65cm, the height is 162cm, and the horizontal
deflection angle of the camera is 25 degrees to 30
degrees (26 degrees is used in this study because 26
degrees has the highest accuracy). From the two
direction vectors to the user from each camera, wehave determined the user’s position by stereoscopic
calculation.

4Figure 4: Move Mode and Menu
Our system also has a function for the user tocorrect the reflection position for the target device
on the mirror, as shown in Fig. 4. A user can enterthe ”move mode” by double-clicking the screen andthen manually moving the reflection point for target
devices. This eliminates efforts for registering device
locations before initial use.
3.3 Calculation of Space Coordinates
Every camera has some distortion. The distortionof the camera used in this paper is shown in the1 https://mqtt.org

Future WorkAs mentioned above, our system provides a ”movemode” to register device positions. We will improve
the function so that the position of target devices
will be gradually precise through everyday use of thesystem. This will also be effective to track devices
that might be moved by a resident of the smart home.

References[1] Sato, H., Kitahara, I., Ohta, Y.: MR-Mirror: A Complexof Real and Virtual Mirrors; Proc. HCI International ’09,
pp.482–491 (2009).
[2] Saakes, D., Yeo, H. S., Noh, S. T., Han, G., Woo, W.: MirrorMirror: An On-Body T-shirt Design System; Proc. CHI ’16,
pp.6058–6063 (2016).