# Virtual Guia Fortress: A 3D-printed eXtended Reality Playset

Iok Fong Chau, Jorge Cardoso, Gerald Estadieu

► To cite this version:

Iok Fong Chau, Jorge Cardoso, Gerald Estadieu. Virtual Guia Fortress: A 3D-printed eXtended Reality Playset. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.370-376, 10.1007/978-3-030-89394-1_28 . hal-04144379

HAL Id: hal-04144379

https://inria.hal.science/hal-04144379

Submitted on 28 Jun 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Virtual Guia Fortress: a 3D-printed eXtended Reality Playset

Iok Fong Chau<fnref n="1" />, Jorge C. S. Cardoso<fnref n="2" />[0000-0002-0196-2821], and Gerald Estadieu<fnref n="1" />

**Abstract.** This paper reports a work-in-progress that explores the use of 3D-printing for marker-based Tangible User Interfaces for use with smartphone-based eXtended Reality experiences. We have explored several ways to 3D-print matrix markers that are re-configurable so that the underlying values can change with user interaction. We have also applied 3D-printed markers to the development of a modular playset for exploring the Guia Fortress cultural heritage site in Macau SAR, China. We report some of the explorations of 3D-printed markers and describe the Virtual Guia Fortress playset.

**Keywords:** 3D-printing · Tangible User Interface · Virtual Reality · Augmented Reality · Mixed Reality.

## 1 Introduction

Tangible User Interfaces (TUIs) are known for enhancing the user experience with an interactive system by providing a more natural way to manipulate information [4]. Various technologies can be used to create tangible objects, from electronically instrumented, active, objects, to passive ones. The use of passive objects is generally cheaper and easier to build. One way to achieve this is through visual marker-based detection. When combined with smartphones, this allows the development of TUIs for smartphone-based, accessible, eXtended Reality (XR) experiences.

While smartphone-based TUI XR experiences are not new (e.g. [2]), this work explores the use of 3D-printed, dynamic (i.e., that change shape), markers for the development of modular toys. Our aim is to achieve an accessible tangible system with low-cost materials and appropriate for smartphone-based XR. We also aim at developing a system that appeals to the maker community by providing 3D-printed TUI solutions that can then be used in a variety of ways. In this paper, we present the design explorations of several 3D-printed marker structures and the application of one 3D-printing solution to the creation of the Virtual Guia Fortress playset – a modular playset for exploring an iconic cultural heritage location in Macau SAR, China.

<footnote n="1">¹ University of Saint Joseph {201900002,gestadieu}@usj.edu.mo</footnote>

<footnote n="2">² University of Coimbra, CISUC, DEI jorgecardoso@dei.uc.pt</footnote>
## 2 Related Work

Visual markers have been used in many different ways to create tangible interfaces with very different interaction capabilities.

Zheng et al. [6] use the particularity of paper to change its shape and create markers that appear when the paper is folded or handled in specific ways. They explore multiple techniques: printing markers on paper edges, completing markers through collapsing paper folds, deconstructing markers through stretchable paper structures, tearing printed paper markers, making markers visible through wetting paper, and applying light behind the paper. Drogemuller et al. [3] also explore printed paper markers, but in this case applying them in everyday objects. They print paper strips with multiple markers that can be fitted into different kinds of daily objects to allow interaction by hand gestures. Lee et al. [5] apply markers in the faces of cubes and explore two-handed interaction. They explore two metaphors for object assembly based on the manipulation of two cubes: screw-driving, and block-assembly.

While the previous examples were mostly applied to Augmented Reality (AR) settings, markers have also been used in immersive Virtual Reality (VR) setting. Cardoso & Ribeiro [2] explored the possibility of using visual markers in card-board objects for interaction inside an immersive virtual environment. They explored several prototypes for interacting with cultural heritage, including a Tangible VR Book where the pages were detected through printed markers.

## 3 Design Explorations

Similarly to Zheng et al. [6], we started by exploring several possibilities for 3D-printed markers. This design exploration phase was purposefully divergent in order to understand the possibilities offered by diverse alternatives. We report here only a subset of the prototypes that have been explored so far.

We use the well-known 2D barcode markers from ARToolkit [1], which are square markers with a black border. We use particularly the 3x3 markers. When counting the border, these can be thought as a 5x5 matrix where the cells are either black or white squares.

### 3.1 Puzzles

We started by decomposing the marker matrix into individual elements, but this decomposition would result in very small pieces and would also be uninteresting as a puzzle. We thus explored rectangular elements (2 square units) and corner elements (3 square units). The elements are only for the black cells, the white parts of the marker are assumed to be the background. Figure 1a show the pieces and some of the markers that can be created. Of the 64 possible markers, using the rectangular and corner elements allows us to create 62 markers.

We took this approach further and created pieces composed of 4, 5 and 6 square units. This reduces further the number of possible combinations that
result in valid markers, but the pieces are bigger and easier to manipulate. By combining these elements with a white box that also includes the white frame for the markers, a set of pieces can be used as a puzzle game (Figure 1b).

We also experimented with a sliding puzzle, combining white and black square pieces, although this further limits the number of valid combinations (Figure 1c). A flipping puzzle was also developed, which allows achieving all possible valid combinations of the matrix markers (Figure 1d).

## 3.2 LED-based markers

We also explored the possibility of creating markers with Light-Emitting Diodes (LEDs) as this would open the possibility of dynamically and automatically changing the value represented by the object.

One of the LED-based markers consisted of a 3D-printed case where seven LED toggle buttons where fitted. This enables users to toggle a matrix cell on or off thus changing the value represented by the marker (Figure 1e).

A more interesting approach is based on magnetically activated LEDs, which consist of an independently powered LED with a magnetic switch (Figure 1f). Hovering a magnet over each cell of the matrix marker turns the LED on or off. Although we haven't further pursued this approach, it should be possible to create 3D-printed objects fitted with magnets that would turn on/off all LEDs simultaneously when in contact with the LED-based marker, thus changing its value.

## 3.3 Moving mechanisms

We also explored moving marker mechanisms to change the underlying value. We developed 3D-printed markers composed of two components: the first is a recessed marker that is in itself identifiable as the marker number zero. The second component is a 3D-printed white plate with a single protruding black cell (Figure 1g). The allows creating a marker that can change its value by moving (translating) the underlying plate within the available white space of marker zero (we chose marker number zero because it is the one that has the highest inner white area). This allows six possible values to be created.

It is also possible to create simple rotation mechanisms. Figure 1h show an example of a marker composed of a circular plate with two black cells. By placing the second component on top (marker number zero), we can rotate the plate to change the associated marker value. This arrangement allows changing between four different values.

# 4 Virtual Guia Fortress Playset

After the diverging design explorations summarised in the previous section, we started converging design ideas to apply some of the 3D-printed marker solutions into mixed-reality modular toys.
<figure><img src="image_3.png" /><figcaption>(a) Puzzle with two different pieces: 1x2 and corner pieces.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(c) Sliding puzzle.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(d) Flip puzzle.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(e) LED-based markers with LED buttons.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(f) LED-based markers with magnetically activated LED.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(g) Single cell moves inside marker.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(h) Rotating mechanism with two black cells.</figcaption></figure>

<caption>Fig. 1: 3D-printed marker explorations.</caption>
We are currently exploring the use of 3D-printed markers on a playset (Figure 2) for the exploration of the Guia Fortress – an iconic location in Macau SAR, China.

<figure><img src="image_10.png" /><figcaption>Fig. 2: Virtual Guia Fortress 3D-printed playset.</figcaption></figure>

The Guia Fortress was built in the 17th century for military use, located at the highest point on the horizon of Macau. It is regarded as one of the most important landmarks in Macau. The lighthouse being now a tourist attraction,
has also played an essential role in the maritime history of Macau. As the first lighthouse in the Far East, the Guia Lighthouse has contributed to the maritime history of Macau since the 17th century.

The playset is a tangible object that allows users to learn about the Guia Fortress and with the lighthouse in AR/VR. The Virtual Guia Fortress Playset is designed as seven interactive, stackable, layers. It includes three maps from the 18th, 19th, and 21 centuries. On top of these, there is also the Guia Fortress's ground layer, church, and lighthouse. There is also a tangible boat that is used in combination with the layers. When all the layers are combined, the church and the lighthouse stay on top. The map's outer frame was designed to look like a brick wall and blend into the foundation outlook of the building.

Interaction with the playset occurs through the manipulation of a boat tangible marker (Figure 2a, bottom left) that can be placed over the protruding black cells in the lower right corner of each map. On each layer, placing the boat will display an animation of a boat (with different visual appearance in accordance to the century represented by the layer). In addition, the combination of the boat and protruding cells functions as a moving mechanism so that moving the boat up/down or left/right (depending on the layer) will alternate between a day/night representation (Figure 2b). On the Guia Fortress ground plate, the boat can be placed on one of four locations in a row on the bottom part. Each location will trigger a different visual rendering of the lighthouse scenario: day view, night view, lighthouse lights on, and cannon firing (Figure 2c).

We are currently evaluating the usability and user experience of this playset.

## 5 Conclusion

Exploring 3D-printed visual markers for object detection in smartphone-based XR experiences can foster the development of accessible tangible systems that anyone can 3D-print, adapt and reuse.

We have started exploring this avenue by 3D-printing several different ways to create modular objects with dynamic markers that can then be used interactively to affect the XR environment. We explored puzzles, active LED-based markers, movable parts, etc. To demonstrate the applicability of the approach, we have also created a 3D-printed playset for the virtual exploration of the Guia Fortress heritage site in Macau SAR, China.

The preliminary results show that users consider the playset “creative” and “entertaining” which gives us confidence on the usefulness of this approach.

## References

1. ARToolkit: Official ARToolkit site. https://artoolkit.org/ (2018), https://artoolkit.org/

2. Cardoso, J.C.S., Ribeiro, J.M.: Tangible VR Book: Exploring the Design Space of Marker-Based Tangible Interfaces for Virtual Reality. Applied Sciences **11**(4), 1367 (2 2021). https://doi.org/10.3390/app11041367, https://www.mdpi.com/2076-3417/11/4/1367
3. Drogemuller, A., Walsh, J., Smith, R.T., Adcock, M., Thomas, B.H.: Turning everyday objects into passive tangible controllers. In: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction. pp. 1–4. Association for Computing Machinery (ACM), New York, NY, USA (2 2021). https://doi.org/10.1145/3430524.3442460, https://dl.acm.org/doi/10.1145/3430524.3442460

4. Hinckley, K., Pausch, R., Goble, J.C., Kassell, N.F.: Passive real-world interface props for neurosurgical visualization. In: Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI '94. pp. 452–458. ACM Press, New York, New York, USA (1994). https://doi.org/10.1145/191666.191821, http://portal.acm.org/citation.cfm?doid=191666.191821

5. Lee, H., Billinghurst, M., Woo, W.: Two-handed tangible interaction techniques for composing augmented blocks. Virtual Reality 15(2-3), 133–146 (6 2011). https://doi.org/10.1007/s10055-010-0163-9, https://link.springer.com/article/10.1007/s10055-010-0163-9

6. Zheng, C., Gyory, P., Do, E.Y.L.: Tangible interfaces with printed paper markers. In: DIS 2020 - Proceedings of the 2020 ACM Designing Interactive Systems Conference. pp. 909–923. Association for Computing Machinery, Inc (7 2020). https://doi.org/10.1145/3357236.3395578, https://doi.org/10.1145/3357236.3395578