# Dynamir : Optical Manipulations using Dynamic Mirror Brushes

Florent Berthaut, Deepak Ranjan Sahoo, Jess McIntosh, Diptesh Das, Sriram Subramanian

► To cite this version:

Florent Berthaut, Deepak Ranjan Sahoo, Jess McIntosh, Diptesh Das, Sriram Subramanian. Dynamir : Optical Manipulations using Dynamic Mirror Brushes. ACM International Conference on Interactive Tabletops and Surfaces, Nov 2015, Madeira, Portugal. hal-01222512

HAL Id: hal-01222512

https://hal.science/hal-01222512

Submitted on 24 Nov 2015

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Dynamir : Optical Manipulations using Dynamic Mirror Brushes

**Florent Berthaut**

University of Bristol
florent@hitmuri.net

**Deepak Ranjan Sahoo**

University of Bristol
deepakranjan@gmail.com

**Jess McIntosh**

University of Bristol
jm0152@my.bristol.ac.uk

**Diptesh Das**

University of Bristol
diptesh.das09@gmail.com

**Sriram Subramanian**

University of Bristol
sriram@sussex.ac.uk

<figure><img src="image_2.png" /><figcaption>Figure 1. Dynamic mirror brushes allow for optical manipulations by deforming the mirror surface (displayed below). A) Applying a controlled stretching effect using a vertical brush with a 1D curvature. B) Contracting the user's hand for the selection of a small sphere on a stereoscopic screen by changing the curvature and position of a small brush interactively. C) Moving the reflection of physical objects with a finger by changing the orientation of a large brush.</figcaption></figure>

## ABSTRACT

Mirror surfaces are part of our everyday life. Among them, curved mirrors are used to enhance our perception of the physical space, e.g., convex mirrors are used to increase our field of view in the street, and concave mirrors are used to zoom in on parts our face in the bathroom. In this paper, we investigate the opportunities opened when these mirrors are made dynamic, so that their effects can be modulated to adapt to the environment or to a user's actions. We introduce the concept of dynamic mirror brushes that can be moved around a mirror surface. We describe how these brushes can be used for various optical manipulations of the physical space. We also present an implementation using a flexible mirror sheet and three scenarios that demonstrate some of the interaction opportunities.

## Author Keywords

Dynamir; Shape Changing Mirrors; Mixed Reality;

## ACM Classification Keywords

H.5.m. Information Interfaces and Presentation (e.g. HCI): Miscellaneous

## INTRODUCTION

Curved mirrors are extensively used in everyday life, from theme parks to street signs. These surfaces create a reflection of the physical space that can be altered while preserving some of its physical qualities. For example, convex traffic mirrors are often used at road intersections so on-coming vehicles can see around corners. Large field of view convex mirrors are also embedded in side-view mirrors of vehicles to minimise blind spots. It is common to find concave mirrors in hotel bathrooms that provide a magnified image of the face for shaving. Curved mirrors have also been used in the context of imaging and displays. For instance, in [5], the authors increase the field-of-view (FOV) of a head mounted projected display using a hyperbolic half-silvered mirror. Continuous deformable mirrors are commonly used to correct optical aberration in telescopes, microscopes and laser optical systems. Discrete deforming mirror array are used in projectors [7], as well as solar cells, camera and interactive art. Planar mirrors have been used extensively in mixed-reality displays, however these mirrors remain static. For example, a semi-transparent flat-mirror enables the reflection of a virtual screen to be projected in a physical space allowing
augmentation of an interactive surface [3]. Another use is
in [6], where the authors show how the shared space created
by semi-transparent mirrors can be used for bidirectional aug-
mentations, e.g., of physical objects on both sides. However,
the mirrors remain flat and static, and manipulations of the
physical space are done through the virtual augmentations.

Dynamir is a shape changing mirror system that enables a
novel approach to optical manipulations based on dynamic
mirror brushes. These brushes allow for controlling local de-
formation of the surface of a flexible mirror with the purpose
of enabling manipulations of the physical space, resulting in
novel interaction possibilities. Dynamic street mirrors could
adapt to the traffic, giving a wider field of view when more
and/or faster vehicles are detected. Bathroom mirrors could
allow for different levels of zoom when shaving. Individual
mirrors could emerge from a collective one when practicing
sports or arts and provide private views on movements of
users. Users could also be augmented with physical content
in novel ways. For example their reflection could be adjusted
so that it fits specific clothes or accessories in shopping win-
dows, or they could be made smaller with a convex mirror
and integrated in miniature scenes during storytelling perfor-
mances. On the contrary to augmented mirrors based on cam-
eras and displays, which might limit resolution and contrast
of the mirrored scene, Dynamir preserves the optical scene,
including the visual cues. Finally, Dynamir can also be used
for 3D user interfaces similar to the HoloCubtile [1], adding
the possibility of optically manipulating the physical part of
interaction techniques, e.g. hands and props, in addition to
the virtual part.

Our contribution is three-fold: 1) We propose dynamic mirror brushes that allow for optical manipulations with shape-changing mirrors; 2) We propose an implementation of the brushes with a flexible mirror sheet that also allows for augmented reality scenarios; 3) We demonstrate three novel interactive scenarios.

DYNAMIC MIRROR BRUSHES

Mirrors produce specular reflection of light, i.e., the angle of incidence $\theta_i$ at a point P on the mirror equals the angle of reflection $\theta_r$. When an object is placed in front of the mirror, the reflections of the rays coming from the object produce a reflected image of this object. The perception of this reflected image by an observer will depend on the positions of both the observer and the object relative to the mirror. It will also depend on the normals on the mirror surface at all intersections with the light rays coming from the object. Manipulating the reflections therefore amounts to controlling the normals at all points of the surface of the mirror. In this paper, we propose to simplify this manipulation in the context of interactive applications by introducing the concept of dynamic mirror brushes. These brushes are local deformations of the mirror surface which: 1) are aimed at a group of observers, 2) affect a certain region of interest (ROI), and 3) have several attributes that define the effect that they create. The ROI can be the observers themselves or any group of objects in the physical space. As depicted on Figure 2, several brushes can be created and manipulated on a single mirror surface, allowing

for different optical effects for different groups of users.
Next, we describe the attributes of the optical brushes, i.e.,
how various optical effects can be specified using them, and
how they can be translated into the description of the curva-
ture of the mirror.

Size and shape attributes

The first group of attributes, *size and shape*, allow for con-
trolling both the number of observers and the size of the ROI.
Changing the size allows, for example, to control how much
of a group of people an observer can watch. By varying the
shape of the brush, a particular selection of physical objects
can be defined from the observer's point of view with any
contour, as shown on Figure 2.B. Given the extremities of a
ROI, we can calculate the brush size from the lines forming
the frustum which define the view of the observer. Finding the
intersection points between these lines and the mirror plane
give us the coordinates for the corners of the brush.

Position and orientation attributes

The second group of attributes correspond to the *position and
orientation* of the brushes relative to the mirror surface. They
can be seen as allowing for three different effects. The first is
the selection of a region of interest from the observer's point
of view. In this case, a brush with a fixed position is rotated
according to the observer and ROI positions. The second ef-
fect is the manipulation of the position of the reflected image
of the ROI. The brush can be moved and rotated in order to
move the reflection relative to the observer. Finally, these at-
tributes can be used to inspect the ROI from different angles.

Curvature attribute

The *curvature* attribute enables two main effects, contraction
and expansion (i.e., zoom in and zoom out) of the ROI. These
effects can be applied either horizontally or vertically as 1D
effects, or by combining both as 2D effects. A possible use is
the modification of the field of view so that observers can

<figure><img src="image_3.png" /><figcaption>Figure 2. Mockup of three brushes controlled on a large mirror: Left) a group of observers changes the curvature of the Brush A to perceive more of the physical space, Center) the blue character controls the orientation, size and position of the Brush B to select the head of the green character, Right) the yellow character scales his body by controlling the curvature of Brush C.</figcaption></figure>
perceive more of a scene while preserving the size of the brush, as demonstrated in Figure 2 Left. It can also be used as a way of enriching 3D interaction done through the observer's reflection, by manipulation the observer himself. The contraction/expansion and hence, zooming out/in of the ROI is achieved by decreasing/increasing the radius of curvature, making the brush respectively convex or concave.

## IMPLEMENTATION

<figure><img src="image_4.png" /><figcaption>Figure 3. Implementation of dynamic mirror brushes: The shape of a horizontal HIPS mirror sheet (a) is controlled using a LCD monitor (b) displaying a depth-map of the mirror surface and the Shapeclips (c). The reflection image of the mirror is formed as a vertical surface (d) using a half-silvered semi-transparent mirror (e) and is overlapped with a stereoscopic screen (f). A Leap Motion (g) and an Asus Xtion depth camera (h) are used for tracking the positions of the user's head and hands to interact with the mirror.</figcaption></figure>

## Hardware

We used a silvered high impact polystyrene (HIPS) sheet with area $420 \text{ mm} \times 297 \text{ mm}$ (A3 size) and thickness $t = 1 \text{ mm}$ as the shape changing mirror surface in our implementation. The HIPS mirror is highly flexible with flexural modulus $E \approx 2.4 \times 10^5 \text{ psi}$, and highly stress resistant with flexural strength $\sigma_m \approx 4.21 \times 10^3 \text{ psi}$. This corresponds to yield strain $\epsilon_m = \sigma_m/E = 1.75 \times 10^{-2}$, which means that a sharp edge can be approximated with a round edge with minimum radius of curvature $\rho_m \approx t/2\epsilon_m = 2.85 \text{ cm}$. Using the HIPS sheet with $t = 0.1 \text{ mm}$, very sharp edges with $\rho_m = 2.85 \text{ mm}$ can be achieved. Large HIPS mirror with size $2500 \text{ mm} \times 1250 \text{ mm}$ are available off-the-shelf. HIPS is a copolymer of polybutadiene rubber, which has high Poisson ratio $\nu \approx 0.5$. In other words, the HIPS mirror shows good flatness for undistorted reflection, and at the same time good bending, stretching and twisting performance for local shape control.

To change the shape of the mirror, we used a new linear actuator, ShapeClips [4], which has a footprint $W^2$ of $2 \text{ cm} \times 2 \text{ cm}$ and travel range $R = 6 \text{ cm}$. The ShapeClips were mounted vertically to support the weight of the mirror for simplicity; i.e., we kept the actuated mirror surface horizontal. A spring

mechanism was used to attach the HIPS sheet that allowed the desired deformation of the mirror. In our implementation, the maximum magnitude of shapes is 6 cm, and the maximum angle of sharp edge is $\tan^{-1}(R/W) = 71.56^\circ$. Due to the inherent solid mechanics of the HIPS sheet, the shapes generated comprise of hyperbolic functions; the radius of curvature can be calculated from the solid mechanics analysis [9]. The maximum speed of the ShapeClips is 8 cm/s, which dictates the speed of our implementation. We used nine ShapeClips in our implementation. The ShapeClips are compact and modular, and more of them can be incorporated in our implementation seamlessly to create shapes with higher complexity [2], and to alleviate the trade-off between achieving flat surface and sharp edges. For example, more ShapeClips and thinner HIPS mirror are required to effectively demonstrate the application described in Figure 2.B that uses both flat surfaces and sharp edges. Our implementation with nine ShapeClips and a thicker HIPS mirror allows for curved brushes, either two dimensional (bumps), or one dimensional vertical or horizontal curved ridges, as well as flat brushes when rotating the entire mirror sheet. As depicted on Figure 3, we placed a half-silvered mirror at 45° angle above the shape changing mirror. The space in front of this half-silvered mirror is therefore reflected on to the shape changing mirror, and back to the front space towards the user. In addition, a polarized stereoscopic screen is placed behind the half-silvered mirror at the virtual image of the shape changing mirror, making it possible to overlap virtual content with the manipulated/deformed reflection space.

## Software

The software part of our implementation is mostly written in Javascript and runs in a browser. 3D shapes are rendered with Three.js, Leap Motion data is gathered with the leap Javascript SDK and position of the head computed from the depth camera is sent from a C++ application to the browser via a node.js bridge and OpenSoundControl messages. We want to be able to manipulate the optical brushes by simply generating and controlling corresponding virtual brushes in the form of meshes. Because ShapeClips translate the light field below them to a vertical displacement, we render the virtual brushes in a 3D scene below them. The frustum is defined so that near and far clipping planes match the minimum and maximum physical heights, and its dimensions match the dimensions of the screen. We then use a GLSL fragment shader that renders the depth of the meshes instead of their color. In order to compensate for variations in the ShapeClips heights, we also integrate a calibration texture in our fragment shader, which offsets the rendered depth separately for each ShapeClip. The virtual brushes themselves can be created in different ways depending on which attributes need to be controlled. For example, spheres with changes in position and scale can serve as brushes with a controllable 2D curvature, cylinders as brushes with a controllable 1D curvature, simple planes for flat brushes with changes in orientation. A more generic method consists in building the brush mesh from a Non-Uniform Rational B-spline surface, allowing for changes in size, shape and curvature on both axes, following the guidelines on shape-changing displays given in [8].
## SCENARIOS

We propose three interactive scenarios enabled by Dynamir which demonstrate the effectiveness of our implementation.

The first scenario, depicted on Figure 1.A, and demonstrates the use of the position and curvature attributes. It allows an observer to apply a vertical contraction effect to himself, which remain applied to him as he moves, contrary to what would be possible with a static mirror. The magnitude of the contraction, is increased and decreased by moving the hand respectively away from and towards the mirror. With a larger scale implementation, several brushes could be created and applied to different regions of interest of the scene reflected in the mirror. Being defined relative to the user, these brushes would move with him as the single brush does in our scenario, keeping the region of interest of each brush constant. Each of their attributes could be set independently. In effect, this scenario corresponds to an optical version of an image manipulation application.

The second scenario, depicted on Figure 1.B, demonstrates how the curvature attribute can be controlled to expand and contract a region of interest in two dimensions. It draws inspiration from augmented mirrors such as the ones presented in [6], which allow users to manipulate virtual content through their reflection. Using our dynamic mirror brushes, their reflection can be altered to enrich the interaction techniques. In our case, a small sphere and a large cube are displayed on the stereoscopic screen placed behind the half-silvered mirror. By closing / opening their hand, the user controls the curvature of the brush, which in turns makes their hand smaller or larger, allowing them to select and manipulate the object of corresponding size at an appropriate scale. Other 3D manipulation and navigation techniques could be enriched by manipulating the user's image, for example to optically modify the control-display ratio.

With our third scenario, depicted on Figure 1.C, we demonstrate the use of the orientation attribute to manipulate the physical world. The user is able to move the reflection of a pre-defined physical object, here a mug. From the point of view of the user, the mug remains at the tip of his finger. The optical brush, which covers the whole mirror, changes orientation in order to align the virtual image of the object with the finger. With multiple brushes, multiple objects from the physical space could be manipulated at the same time.

## CONCLUSION

In this paper, we proposed a novel approach for optical manipulations using shape changing mirrors. We described dynamic mirror brushes that allow for describing these manipulations with controllable attributes. We described an implementation of a shape-changing mirror and three interactive scenarios demonstrating the use of the brushes. We believe that our approach opens opportunities for research in public displays, augmented reality and 3D interaction. It could also benefit many everyday life applications. Finally, it leads to more interesting research questions. A large scale implementation will for example require studying materials with different stretchability and actuation mechanisms, in order to

enable large changes in position and size attributes and sharp changes of orientation anywhere on the surface.

## ACKNOWLEDGEMENTS

This project was funded by the EC FP7 through the Marie Curie Fellowship (PIEF-GA-2012-330770), ERC Starting Grant (278576) and FET Open Scheme (309191).

## REFERENCES

1. De la Rivière, J. B., Dittlo, N., Orvain, E., Kervégant, C., and Courtois, M. Holocubtile: 3d multitouch brings the virtual world into the user's hands. In *ACM International Conference on Interactive Tabletops and Surfaces*, ITS '10, ACM (New York, NY, USA, 2010), 311–311.

2. Follmer, S., Leithinger, D., Olwal, A., Hogge, A., and Ishii, H. inform: Dynamic physical affordances and constraints through shape and object actuation. In *Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology*, UIST '13, ACM (New York, NY, USA, 2013), 417–426.

3. Hachet, M., Bossavit, B., Cohé, A., and de la Rivière, J.-B. Toucheo: Multitouch and stereo combined in a seamless workspace. In *Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology*, UIST '11, ACM (New York, NY, USA, 2011), 587–592.

4. Hardy, J., Weichel, C., Taher, F., Vidler, J., and Alexander, J. Shapeclip: Towards rapid prototyping with shape-changing displays for designers. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, CHI '15, ACM (New York, NY, USA, 2015), xxx–xxx.

5. Kiyokawa, K. A wide field-of-view head mounted projective display using hyperbolic half-silvered mirrors. In *Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality*, ISMAR '07, IEEE Computer Society (Washington, DC, USA, 2007), 1–4.

6. Martinez Plasencia, D., Berthaut, F., Karnik, A., and Subramanian, S. Through the combining glass. In *Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology*, UIST '14, ACM (New York, NY, USA, 2014), 341–350.

7. Nayar, S. K., Branzoi, V., and Boult, T. E. Programmable imaging: Towards a flexible camera. *Int. J. Comput. Vision* 70, 1 (Oct. 2006), 7–22.

8. Roudaut, A., Karnik, A., Löchtefeld, M., and Subramanian, S. Morphees: Toward high "shape resolution" in self-actuated flexible mobile devices. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, CHI '13, ACM (New York, NY, USA, 2013), 593–602.

9. Ventsel, E., and Krauthammer, T. *Thin Plates and Shells: Theory: Analysis, and Applications*. CRC Press, 2001.