# A Scoping Review on Tangible and Spatial Awareness Interaction Technique in Mobile Augmented Reality-Authoring Tool in Kitchen

Authors: Siti Soleha Muhammad Nizam; Meng Chun Lam; Haslina Arshad; Nur Asylah Suwadi
Journal: Advances in Multimedia
 (2018)

---

## Abstract

The interaction paradigm has changed with the emerging technology, mobile augmented reality, and spatial awareness mobile device. The traditional way for designing a kitchen can cause mistake in measurement and user’s imagination is different from kitchen advisor’s sketch design due to limitation of human imagination. Using mobile augmented reality technology, the user can overlay the virtual kitchen outcome that could fit with the actual kitchen environment. Interaction technique is required in order to allow the user to change the characteristic of the virtual kitchen to suite their need. Thus, this paper is intended to propose the tangible and spatial awareness interaction techniques in the kitchen design authoring tool. A scoping review related to the previous research on tangible and spatial awareness interaction is presented in this paper. The proposed techniques were based on the review on the existing interaction technique and the interview section with the Malaysian kitchen designer by understanding the kitchen design elements and flow. The proposed techniques will be further improved by going through the heuristic evaluation with the augmented reality expert.


---

## Body

## 1. Introduction

Augmented Reality (AR) receives a lot of attention globally. There are a lot of efforts in research and ongoing development in this area as it is the newest technology. Augmented Reality can provide attractive and interactive experience in kitchen design. Interior design has gained tons of attention among the users in order to make a convenient and comfortable kitchen environment. Based on the previous finding, AR offers many advantages in digital architectural design and construction fields [1]. It is a new design approach for interior design, where AR environment is able to display virtual furniture and it can be modified in real-time where the user can directly see the virtual outcome of the design.Some of the kitchen design companies are using physical board to show rough design by arranging the physical blocks into the empty kitchen layout as shown in Figure1. However, with the limitation of human imagination, it is challenging to visualize how the cabinet will look like or align in the real kitchen space [2]. Mobile Augmented Reality (MAR) application allows the virtual furniture to overlay into real scene. Thus, the user can see the outcome of the kitchen design whether the design outcome could fit with the actual kitchen environment. The interaction with the augmented reality environment plays an important role in this authoring type application. It could affect the performance and user preference such as to discourage the user if the interaction is too complicated. Thus, this paper is focusing on the interaction technique design in kitchen design. There are two types of potential interaction techniques, which are suitable for the kitchen design: tangible and spatial awareness interaction technique. This work reviews the existing studies on the tangible and spatial awareness technique in order to identify and design a new interaction technique for the kitchen design in MAR environment.Figure 1
Physical kitchen miniature modelling kit [3].This paper consists of six parts: the first part is the introduction of the study. The second part discusses the documents selection process. The third part is the reviews of the studies related to tangible and spatial awareness interaction technique to explore their possibilities in kitchen design MAR authoring tool. Next, the kitchen design process is defined based on the interview section with the Malaysian kitchen design company. The fifth part of the paper discusses the proposed tangible and spatial awareness interaction techniques in mobile augmented reality setting. The conclusion and the future works are in the final part of the paper.


## 2. Documents Selection

This study focused on two types of interaction techniques, which are tangible interaction technique and spatial awareness interaction technique. Tangible interaction technique is an approach of designing the AR interfaces that focus on interactions using related physical object to interact with the virtual object [4, 5]. Tangible interaction technique registers the virtual object to a physical object, and a user interacts with the virtual objects by manipulating the corresponding physical objects [1]. In the designing session, it allows the user to interact with the virtual furniture in order to move, rotate, and copy them as in the real environment using physical object. With the latest development, RGB and depth camera embedded in the smart phone have enabled the spatial awareness interaction technique in a mobile setting [6]. It allows the user to measure the depth of the object by using depth sensor. Spatial awareness interaction technique involves motion tracking, depth sensing, and area learning where the environment properties could be recognized by the AR system. Both types of interaction techniques require further exploration on their effectiveness in helping design process between the kitchen designer and customer.


### 2.1. Documents (Tangible Interaction Technique)

Based on Figure2, the initial result of query search is 1768 from SCOPUS database. Then, the result was filtered by the document type, field, language, and year. The filtration ended up with 1528 documents. After title and abstract filtering, 1528 documents have been reduced to 145. Then, full text reading process excluded 133 documents, resulting in 12 documents as the final document to be analysed.Figure 2
Document selection for tangible interaction technique.


### 2.2. Documents (Spatial Interaction Technique)

Based on Figure3, the initial query shows 3565 raw results in SCOPUS. The field was then specified to computer science leaving 771 results. 717 articles and conference papers were then chosen to be included in this filtering. The documents from year 2000 until 2017 were chosen, reducing the documents into 650 documents. After scanning the title and abstract, only 116 documents were applicable. At the end, 8 documents were chosen after finishing the full text reading.Figure 3
Document selection for spatial interaction technique.


## 2.1. Documents (Tangible Interaction Technique)

Based on Figure2, the initial result of query search is 1768 from SCOPUS database. Then, the result was filtered by the document type, field, language, and year. The filtration ended up with 1528 documents. After title and abstract filtering, 1528 documents have been reduced to 145. Then, full text reading process excluded 133 documents, resulting in 12 documents as the final document to be analysed.Figure 2
Document selection for tangible interaction technique.


## 2.2. Documents (Spatial Interaction Technique)

Based on Figure3, the initial query shows 3565 raw results in SCOPUS. The field was then specified to computer science leaving 771 results. 717 articles and conference papers were then chosen to be included in this filtering. The documents from year 2000 until 2017 were chosen, reducing the documents into 650 documents. After scanning the title and abstract, only 116 documents were applicable. At the end, 8 documents were chosen after finishing the full text reading.Figure 3
Document selection for spatial interaction technique.


## 3. Result and Statistical Information of Documents

In summary, 20 articles have been selected from the SCOPUS database and reviewed in which 60% are in tangible interaction technique and 40% are related to spatial awareness interaction technique. All the papers were published in proceedings and journals from 2002 until 2017. Study on spatial awareness interaction technique is a new topic and the reviewed papers are from 2013 to 2017. 50% of the reviewed papers for tangible interaction technique are proceeding papers and another 50% are from journal articles. Meanwhile, all the reviewed papers for spatial awareness interaction technique are proceeding papers. The selection process will be discussed in Section2. There are 6 scenarios that used tangible interaction technique in their study as shown in Table 1.Table 1
Scenarios.
Interaction Scenario Percentage Tangible Product design 8% Rapid Prototype 8% Game 18% Music 8% Interior design 25% Education 33%  Spatial awareness Architecture 37% Information Sharing 37% Game 13% Medical 13%Based on the review there are four types of AR, which are projection based, see-through based, PC based, and mobile based which used tangible and spatial awareness interaction techniques. The statistics of the types of AR used in the reviewed papers are shown in Table2.Table 2
Type of AR.
Type of AR Type of interaction Tangible Spatial awareness Projection 17% 50% See-through 58% 12.5% PC 17% 12.5% Mobile 8% 25%


### 3.1. Studies Related to Tangible Interaction Technique

Table3 shows a review on existing tangible interaction technique. There are 12 researches reviewed; most of the researches (50%) used card with marker as interaction medium to interact with virtual object. 18% used cube to manipulate the virtual object. Another 32% used specific physical object such as LP records, paddle, pen, and cup as medium of interaction. For the physical setup statistic, 58% of the researchers used HMD as a device to display the virtual object. 17% of the studies used projector and 17% used PC to display the virtual environment. Only one study used mobile phone as a display device.Table 3
Scoping review on existing tangible interaction technique.
Name of invention Interaction technique: medium Interaction technique Scenario Type of AR Physical setup MagicMeeting [7] Tangible: cake platter, clipping plane, card MagicMeeting is a collaborative AR system designed to support a scenario where a group of experts meet to discuss the design of a product.(i) Rotate cake platter: rotate view(ii) Transfer the virtual model: put PDA near cake platter:(iii) Operate virtual model: use clipping plane(iv) Change light source: control light prop device(v) Change virtual model colour: place the marker near the virtual model Product design Projection based (i) Meeting table(ii) HMD(iii) Projection screen(iv) Cake platter(v) PDA(vi) Clipping plane(vii) Three card colour (red, green, yellow)(viii) Light prop device  Shared Space [8] Tangible: card Shared space is an AR collaborative card game.(i) Place the card side by side. If cards contain correct matches, an animation is triggered Game See-through based (i) HMD + camera(ii) Card  ARGroove [8] Tangible: LP records Manipulate the LP records to composite music.(i) Up: pitch(ii) Down: amplitude(iii) Rotate: filter cut-off frequency(iv) Tilt: delay mix Entertainment Projection based (i) Projector(ii) LP records(ii) Projection screen  Tiles [8] Tangible: card Tiles is an AR authoring interface that supported rapid prototyping for aircraft instrument panels.(i) Data tiles: show virtual object from menu book(ii) Operator tiles: delete, copy, help Rapid prototyping See-through based (i) Whiteboard(ii) Book(iii) Magnetic tiles(iv) HMD  VOMAR [8] Tangible: Paddle An AR multimodal interface for designing house via combination of speech and paddle gestures interaction.(i) Place paddle on the marker of menu book to pick the object.(ii) Tilt the paddle to place the object.(iii) Push to move the object.(iv) Shake to delete the object from paddle.(v) Hit to delete object from virtual room. Interior Design See-through Based (i) Menu Book(ii) Cardboard Paddle(iii) Large Piece of Paper(iv) HMD  Jumanji Singapore [9] Tangible: Two Cubes Jumanji Singapore is a game that allows users to have a magical tour of Singapore by simple manipulations of two physical cubes.(i) Dice: Rolls the dice, the metaphor will move based on the number.(ii) Control cube: (1) Rotate to view different photo and video.(2) Stack the cubes will function as shopping cart.(3) Navigation tool: Turn left, turn right, tilt upwards and tilt downwards. Game See-through Based (i) HMD(ii) Game Board(iii) Two Cube as Dice and Controller  Magic Story Cube [10] Tangible: Foldable Cube A foldable cube for storytelling with 3D graphics, 3D sound, music and speech.(i) Unfold the cube will function as turning pages of traditional book. Education See-through Based (i) HMD + Camera(ii) Foldable Cube  ARTalet [11] Tangible: Pen-type Haptic Device A digilog book that supports novel visual, audio and haptic authoring functions.(i) Place the pen-type on the marker in digilog to copy the virtual object.(ii) Pressing and holding the button on the pen to edit shape of the trajectory object. Education See-through Based (i) Input Images from a Camera(ii) CV Based Tracking Book(iii) Menu Prop(iv) Pen-type Manipulation Prop(v) HMD  MAR authoring tool [12] Tangible: Card Mobile AR authoring tool system that enables user to create AR content.(i) Place the card side by side to augment the new virtual object to be augmented. Education Mobile Based (i) Phone(ii) Card Marker  AR-SEE: Augmented Reality for Passive Solar Energy Education [13] Tangible: Marker Card AR-SEE system that allows user to learn about science of architectural design by interacting with a tangible physical 3D model house.(i) Marker: Manipulate the type of roof and window for the house. Education Mobile Based (i) Phone(ii) Markers  TAR for interior design [1] Tangible: 3 Cards Marker An augmented reality system for designing/educating/presenting interior design projects.(i) Add or hide marker to adjust colour and material of the furniture Interior Design PC Based (i) PC + Webcam(ii) Logitech Quickcam Vision Pro(iii) Markers  MagicCup AR [14] Tangible: Cup-shaped Handheld Compact The MagicCup is a cup-shaped handheld compact AR input device with a tracker that detects six dimensional positions and pose information.(i) Pick: Cover a virtual object with the cup.(ii) Allocate: Place the cup with the virtual object.(iii) Move: Slide the cup.(iv) Rotate: Rotate the cup.(v) Delete: Shake the cup.(vi) Change Mode: Shake the empty cup Interior Design See-through Based (i) Menu Book(ii) Cup-shaped Handheld Compact(iii) HMDA collaborative AR system with tangible interaction that is designed to support a scenario where a group of experts meet to discuss the design of a product has been presented in [7]. The tangible interaction includes a physical round plate for the object rotation manipulation, seeing the internal view of an object using clip plane method controlled by a normal marker to cut through the virtual object, and other manipulation functions. The result showed the interaction technique runs stably and is easy to use. However, based on the user feedback, several aspects of the system could be improved such as reducing the time consumed for tracking and improve quality of image and HMD. Another four applications have been developed using the design principles of tangible interaction technique [8]. All the tangible interaction approaches are using different image patterns on physical card for representing different virtual object such as Shared Space for gaming purpose, ARgroove focus on interactive music, rapid prototyping for aircraft instrument panels in Tiles application, and VOMAR for furniture relocation application. Their finding concludes that the tangible interaction allows seamless spatial interaction with virtual objects in the physical workspace [8].Physical cube is one of the popular mediums of interaction in tangible interaction. For example, a game that has been developed by [9] uses two cubes as a medium of interaction with the virtual object and supports two players to play in turn. This game uses Singapore’s map as the game board. Players were able to view the 3D map of Singapore by using HMD. Cube number one is used as dice and cube number two is used as a controller. When the player rolls cube one, the metaphor will move based on the number. Cube two consists of three functions which are to view photos and videos, as a shopping cart and as a navigation tool. Player can rotate cube two to change the displayed photo and the video. When the player stacks two cubes together, it acts as shopping cart function. It allows the player to add virtual object into an empty virtual cart. Cube two can be turned to left or right and tilted upwards and downwards as a navigation tool to tour around in the virtual world. Another research that used cube as medium interaction is Magic Story Cube [10]. Magic Story Cube is a foldable cube for storytelling with 3D graphics, 3D sound, music, and speech. Users can explore the story content by unfolding the cube and view the content by using HMD. A digilog book that supports novel visual, audio, and haptic authoring functions has been introduced in [11]. This system uses pen-type manipulation prop as medium interaction by placing the pen-type manipulation prop on the marker in digilog book to copy the virtual object. Meanwhile, pressing and holding the button on the pen-type manipulation prop will edit the shape of the trajectory object.A system that uses card marker as a medium interaction has been introduced in [12]. This system allows the user to create virtual AR content by selection menu technique. Then, the user can manipulate the virtual object by using the card marker. For example, when the user puts two markers side by side, a new virtual content will be shown. Another research on tangible augmented reality is called AR-SEE [13]. Passive Solar Energy Education has presented AR-SEE as an augmented reality application for mobile phones. It is a combination of mobile phone-based AR with the physical model of a house compartment such as roof styles, windows, and other building materials. Those design changes will then affect the internal temperature inside the house and the temperature displayed through particle visualization. Then, the users can realize how passive solar energy affects the temperature inside the house and the energy usage efficiency. The users can interact with the AR-SEE via phone by looking through it and pointing at the big green marker. Then, three buttons are allocated on the phone with one of them for the material selection for the roof of the house, another button for the material for the base of the house, and the last button for X-ray vision function which makes the house partially transparent to allow the user to see the visualization inside the house. Other than that, it uses virtual human with different actions to represent temperature condition. For example, the virtual human is sitting if the temperature is less than 81 degrees Fahrenheit and falling forward if temperature rises above 95 degrees Fahrenheit. Their finding shows that the virtual human visualization enables significantly lower temperature estimation error than the particle visualization.The previous researches which are almost similar to this study are VOMAR [8] and TAR for interior design [1]. A study on tangible AR for interior design has been conducted in [1]. This study introduced an augmented reality system for designing or educating in interior design projects by overlaying virtual furniture in a physical environment on a regular PC system. The system uses marker card as a medium to interact with the virtual furniture. The colour and material of the virtual furniture can be adjusted by adding or hiding the card marker. Another similar study was a cup-shaped handheld compact that has been labelled with marker as the medium interaction between user and virtual furniture [14]. This system includes six main functions, which are picking, allocating, moving, rotating, deleting, and changing mode. The user can pick the virtual object by covering the virtual object with the cup. To allocate the virtual object, user can place the cup that contains the virtual object on the table. User can move the virtual object by sliding the cup. Furthermore, rotating the cup will rotate the virtual object and shaking the cup will delete the object. In interior design scenario, basic manipulation ability of the interaction technique is picking, allocating, moving, rotating, and deleting the virtual furniture. All the virtual furniture manipulations in the previous studies were limited where user cannot change the feature of the virtual furniture.


### 3.2. Studies Related to Spatial Awareness Interaction Technique

Depth sensing and motion tracking are useful features in spatial awareness interaction. It allows user to move in 3D space (x, y, and z position) and to track its orientation in yaw, pitch, and roll [23]. This 6 degrees of freedom of sensing capabilities allows the phone to respond to the physical movements of the user. Table 4 shows a review on existing spatial awareness interaction technique. There are 8 researches that have been reviewed.Table 4
Scoping review on existing spatial awareness interaction technique.
Name of invention Interaction technique: medium Interaction technique Scenario Type of AR Physical setup iSarProjection [15] Spatial (i) Involves only projection visualization technique Medical Projection based (i) Handheld projector(ii) RGBD camera (iii) Physical model  SAR/VR HYBRID SPACE [16] Spatial (i) Wand controller: points at a certain location and presses the controller’s trigger to start the drawing Architecture Projection based (i) Wand controller(ii) Projector  Garden [17] Spatial (i) Durovis Dive headset: user can see the world in VR via tablet that has been mounted on it(ii) Hands and Legs: create voxels to fill in the wireframe and draw objects Game Mobile based (i) Tango tablet(ii) Durovis Dive headset  Tabletop SAR [18] Spatial and tangible:rectangular object, scroll wheel Tangible(i) Rectangular object: specification of zoom region in the virtual volume with desired location and orientation(ii) Push button: locks zoom position in place(iii) Scroll wheel: increases or decreases the zoomed volume.Spatial(i) Handheld input device (push button): selection (ii) Handheld input device (scroll wheel): 2D zooming (iii) 6DOF pointing device: drills down on 3D projected onto the physical objects(iv) 4K monitor: displays detailed views of 3D visualization Information sharing Projection based (i) Handheld projecting device(ii) Physical object (iii) 6DoF Tracking (iv) Projector  Tag It! [19] Spatial (i) Tango: extends Google Glass’ sensing ability by sharing the location and pose of the user as well as tag the virtual information in real world(ii) Glass: overlays virtual tags based on the spatial information received from the Tango(iii) Glass touchpad: user can initiate the wireless connection by using a three-finger touch gesture on it Information sharing See-through based (i) Glass(ii) Glass touchpad(iii) Tango (iv) Target  Social spatial mashup (SSM) [20] Spatial (i) Tango: registers the virtual information on the face of the 3D object after social spatial mashup via tapping the virtual button on the device Information sharing Mobile based (i) Google Project Tango  AR interior designer [21] Spatial A depth camera is used to measure the size of available space in terms of supporting surfaces for new furniture(i) User uses mouse to interact and manipulate virtual furniture Architecture PC based (i) PC + Mouse(ii) Microsoft Kinect  PimpMyKitchen [22] Spatial A system that supports the early design process of kitchens and helps to ease designer and client in meeting session(i) Blank (white cabinet): moveable cabinet placed in an environment of the shape and size of the target kitchen(ii) Identical card: resizing projected components. Initially, the components automatically fit the available space(iii) SAR Swatches: choose colour, loading different design and save design (iv) Stylus: taping the cabinet surface to change materials Architecture Projection based (i) Projector(ii) Blanks (white cabinet)(iii) Card(iv) SAR Swatch(v) StylusProjection-based augmented reality is a normal setup for the spatial augmented reality. General technique of dynamically augmenting physical models was introduced back then. iSarProjection is using a handheld projector attached with an RGBD camera to project the texture on a physical model [15]. The physical model is mapped with corresponding textures, and it develops a real-time dynamic calibration, registration, tracking, and rendering. However, this study only focuses on the visualization in the spatial augmented reality environment.A hybrid space combining spatial augmented reality and virtual reality has been proposed and has obtained a positive feedback from interviewees [16]. The hybrid spatial augmented reality (SAR)/Virtual Reality (VR) hybrid space is a projection-based AR that enables multiusers manipulation with an augmented object, and immersive experiences. Virtual objects can be manipulated using the controller’s grab button. To start drawing, the user can directly point at the target location and press the controller’s trigger. Garden by Sing [17] is a study which used the game scenario to explore the mixed reality (MR) experience that combines both VR with AR. It allows users to transform their physical environment into a virtual garden they can play in. The exploration of mixed reality with motion tracking and depth sensing is going very well here. An immersive experience allows players to move through their surroundings untethered while exploring and interacting with the virtual world by using Durovis Dive headset where users can see the world in VR via tablet mounted on it. It consists of a simple hand and leg interaction where user draws a virtual object via voxel that is created by their hand and leg. Another projection-based AR is the Tabletop SAR [18] where it uses physical object such as cube and cone to interact with the 3D data. It is a combination of spatial augmented reality with the tangible interaction technique. The physical object in their study comes with a push button for selection and scroll wheel for 2D zooming. Users manipulate the physical object and push the button to lock the zooming position in the 3D data visualization area and then increase or decrease the zoom level using the scroll wheel.Besides the projection-based AR, researchers have also studied wearable augmented reality and spatial awareness mobile device such as Google Project Tango. Nassani et al. [19] invented a wearable system that allows people to place and interact with 3D virtual tags placed around them and it is named Tag It!. A head-worn wearable computer (Google Glass) and a chest-worn depth sensor (Tango device) were used as wearable technology. The users can create virtual tags with text content generated by voice input and place it on where they are looking at by wearing the system. Tango is utilized to extend Google Glass’ sensing ability by sharing the location and pose of the user as well as the tagged target position in the real world. The Google Glass then overlays virtual tags based on the spatial information received from the Tango, and the background of the glass display is set to black to act as an optical see-through display. Via the Google Glass touchpad, the user can initiate the wireless connection by using a three-finger touch gesture on it. Besides the Tag It! invention that benefits the user, the social spatial mashup (SSM) [20] for information sharing in public space also improves the interaction among human with overlaying the virtual information surrounding them. The proposed SSM is based on RGB-D SLAM to create a 3D feature map and allows users to locate information and contents in 3D space. Information placing is through tapping button in the Tango. The virtual information is presented in an image (a small logo) form, which is located on the face of the 3D object after social spatial mash up. The virtual information is well registered even with the movement of the Tango device.For the interior design area such as in the study of Tang et al. [21], Kinect was used to detect the spatial information of the physical environment where the furniture reseller can create virtual 3D furniture. The virtual furniture is rendered in the real scene of the home space via AR technology. The study performed an evaluation on the system usability, flexibility, and accuracy where overall result is positive, and the system is helpful, usable, efficient, and effective for home furnishing and furniture purchase. It is related to this study; however, it does not have the mobile ability where user can move and see the furniture in different perspective. Blanks (white cabinet) is a metaphor of the cabinets that are placed in an environment of the kitchen with actual shape and size [22]. It supports the early design process of kitchen. It will ease the discussion session between the designer and the customer. The designer can move the cabinet blanks to the desired location. Then, the designer and the customer can decide on the cupboards, the drawers, and other components. The system automatically arranges the components to fit the available space. The component can be manipulated by using the manipulation tools, which are two identical cards to resize the component and to use SAR Swatches for choosing the colour and loading and saving the kitchen designs. It also allows user to compare different designs simultaneously by quickly flicking through several different designs in a more natural way than using a menu. Final design phase allows the designer and the customer to set the material of the cabinet by tapping the cabinet surface with the Stylus.In conclusion, the previous studies have shown the possibility of the spatial awareness interaction between user and the systems such as the SAR/VR Hybrid Space, Tabletop SAR, Tag It!, Social Spatial Mashup, and Garden.


## 3.1. Studies Related to Tangible Interaction Technique

Table3 shows a review on existing tangible interaction technique. There are 12 researches reviewed; most of the researches (50%) used card with marker as interaction medium to interact with virtual object. 18% used cube to manipulate the virtual object. Another 32% used specific physical object such as LP records, paddle, pen, and cup as medium of interaction. For the physical setup statistic, 58% of the researchers used HMD as a device to display the virtual object. 17% of the studies used projector and 17% used PC to display the virtual environment. Only one study used mobile phone as a display device.Table 3
Scoping review on existing tangible interaction technique.
Name of invention Interaction technique: medium Interaction technique Scenario Type of AR Physical setup MagicMeeting [7] Tangible: cake platter, clipping plane, card MagicMeeting is a collaborative AR system designed to support a scenario where a group of experts meet to discuss the design of a product.(i) Rotate cake platter: rotate view(ii) Transfer the virtual model: put PDA near cake platter:(iii) Operate virtual model: use clipping plane(iv) Change light source: control light prop device(v) Change virtual model colour: place the marker near the virtual model Product design Projection based (i) Meeting table(ii) HMD(iii) Projection screen(iv) Cake platter(v) PDA(vi) Clipping plane(vii) Three card colour (red, green, yellow)(viii) Light prop device  Shared Space [8] Tangible: card Shared space is an AR collaborative card game.(i) Place the card side by side. If cards contain correct matches, an animation is triggered Game See-through based (i) HMD + camera(ii) Card  ARGroove [8] Tangible: LP records Manipulate the LP records to composite music.(i) Up: pitch(ii) Down: amplitude(iii) Rotate: filter cut-off frequency(iv) Tilt: delay mix Entertainment Projection based (i) Projector(ii) LP records(ii) Projection screen  Tiles [8] Tangible: card Tiles is an AR authoring interface that supported rapid prototyping for aircraft instrument panels.(i) Data tiles: show virtual object from menu book(ii) Operator tiles: delete, copy, help Rapid prototyping See-through based (i) Whiteboard(ii) Book(iii) Magnetic tiles(iv) HMD  VOMAR [8] Tangible: Paddle An AR multimodal interface for designing house via combination of speech and paddle gestures interaction.(i) Place paddle on the marker of menu book to pick the object.(ii) Tilt the paddle to place the object.(iii) Push to move the object.(iv) Shake to delete the object from paddle.(v) Hit to delete object from virtual room. Interior Design See-through Based (i) Menu Book(ii) Cardboard Paddle(iii) Large Piece of Paper(iv) HMD  Jumanji Singapore [9] Tangible: Two Cubes Jumanji Singapore is a game that allows users to have a magical tour of Singapore by simple manipulations of two physical cubes.(i) Dice: Rolls the dice, the metaphor will move based on the number.(ii) Control cube: (1) Rotate to view different photo and video.(2) Stack the cubes will function as shopping cart.(3) Navigation tool: Turn left, turn right, tilt upwards and tilt downwards. Game See-through Based (i) HMD(ii) Game Board(iii) Two Cube as Dice and Controller  Magic Story Cube [10] Tangible: Foldable Cube A foldable cube for storytelling with 3D graphics, 3D sound, music and speech.(i) Unfold the cube will function as turning pages of traditional book. Education See-through Based (i) HMD + Camera(ii) Foldable Cube  ARTalet [11] Tangible: Pen-type Haptic Device A digilog book that supports novel visual, audio and haptic authoring functions.(i) Place the pen-type on the marker in digilog to copy the virtual object.(ii) Pressing and holding the button on the pen to edit shape of the trajectory object. Education See-through Based (i) Input Images from a Camera(ii) CV Based Tracking Book(iii) Menu Prop(iv) Pen-type Manipulation Prop(v) HMD  MAR authoring tool [12] Tangible: Card Mobile AR authoring tool system that enables user to create AR content.(i) Place the card side by side to augment the new virtual object to be augmented. Education Mobile Based (i) Phone(ii) Card Marker  AR-SEE: Augmented Reality for Passive Solar Energy Education [13] Tangible: Marker Card AR-SEE system that allows user to learn about science of architectural design by interacting with a tangible physical 3D model house.(i) Marker: Manipulate the type of roof and window for the house. Education Mobile Based (i) Phone(ii) Markers  TAR for interior design [1] Tangible: 3 Cards Marker An augmented reality system for designing/educating/presenting interior design projects.(i) Add or hide marker to adjust colour and material of the furniture Interior Design PC Based (i) PC + Webcam(ii) Logitech Quickcam Vision Pro(iii) Markers  MagicCup AR [14] Tangible: Cup-shaped Handheld Compact The MagicCup is a cup-shaped handheld compact AR input device with a tracker that detects six dimensional positions and pose information.(i) Pick: Cover a virtual object with the cup.(ii) Allocate: Place the cup with the virtual object.(iii) Move: Slide the cup.(iv) Rotate: Rotate the cup.(v) Delete: Shake the cup.(vi) Change Mode: Shake the empty cup Interior Design See-through Based (i) Menu Book(ii) Cup-shaped Handheld Compact(iii) HMDA collaborative AR system with tangible interaction that is designed to support a scenario where a group of experts meet to discuss the design of a product has been presented in [7]. The tangible interaction includes a physical round plate for the object rotation manipulation, seeing the internal view of an object using clip plane method controlled by a normal marker to cut through the virtual object, and other manipulation functions. The result showed the interaction technique runs stably and is easy to use. However, based on the user feedback, several aspects of the system could be improved such as reducing the time consumed for tracking and improve quality of image and HMD. Another four applications have been developed using the design principles of tangible interaction technique [8]. All the tangible interaction approaches are using different image patterns on physical card for representing different virtual object such as Shared Space for gaming purpose, ARgroove focus on interactive music, rapid prototyping for aircraft instrument panels in Tiles application, and VOMAR for furniture relocation application. Their finding concludes that the tangible interaction allows seamless spatial interaction with virtual objects in the physical workspace [8].Physical cube is one of the popular mediums of interaction in tangible interaction. For example, a game that has been developed by [9] uses two cubes as a medium of interaction with the virtual object and supports two players to play in turn. This game uses Singapore’s map as the game board. Players were able to view the 3D map of Singapore by using HMD. Cube number one is used as dice and cube number two is used as a controller. When the player rolls cube one, the metaphor will move based on the number. Cube two consists of three functions which are to view photos and videos, as a shopping cart and as a navigation tool. Player can rotate cube two to change the displayed photo and the video. When the player stacks two cubes together, it acts as shopping cart function. It allows the player to add virtual object into an empty virtual cart. Cube two can be turned to left or right and tilted upwards and downwards as a navigation tool to tour around in the virtual world. Another research that used cube as medium interaction is Magic Story Cube [10]. Magic Story Cube is a foldable cube for storytelling with 3D graphics, 3D sound, music, and speech. Users can explore the story content by unfolding the cube and view the content by using HMD. A digilog book that supports novel visual, audio, and haptic authoring functions has been introduced in [11]. This system uses pen-type manipulation prop as medium interaction by placing the pen-type manipulation prop on the marker in digilog book to copy the virtual object. Meanwhile, pressing and holding the button on the pen-type manipulation prop will edit the shape of the trajectory object.A system that uses card marker as a medium interaction has been introduced in [12]. This system allows the user to create virtual AR content by selection menu technique. Then, the user can manipulate the virtual object by using the card marker. For example, when the user puts two markers side by side, a new virtual content will be shown. Another research on tangible augmented reality is called AR-SEE [13]. Passive Solar Energy Education has presented AR-SEE as an augmented reality application for mobile phones. It is a combination of mobile phone-based AR with the physical model of a house compartment such as roof styles, windows, and other building materials. Those design changes will then affect the internal temperature inside the house and the temperature displayed through particle visualization. Then, the users can realize how passive solar energy affects the temperature inside the house and the energy usage efficiency. The users can interact with the AR-SEE via phone by looking through it and pointing at the big green marker. Then, three buttons are allocated on the phone with one of them for the material selection for the roof of the house, another button for the material for the base of the house, and the last button for X-ray vision function which makes the house partially transparent to allow the user to see the visualization inside the house. Other than that, it uses virtual human with different actions to represent temperature condition. For example, the virtual human is sitting if the temperature is less than 81 degrees Fahrenheit and falling forward if temperature rises above 95 degrees Fahrenheit. Their finding shows that the virtual human visualization enables significantly lower temperature estimation error than the particle visualization.The previous researches which are almost similar to this study are VOMAR [8] and TAR for interior design [1]. A study on tangible AR for interior design has been conducted in [1]. This study introduced an augmented reality system for designing or educating in interior design projects by overlaying virtual furniture in a physical environment on a regular PC system. The system uses marker card as a medium to interact with the virtual furniture. The colour and material of the virtual furniture can be adjusted by adding or hiding the card marker. Another similar study was a cup-shaped handheld compact that has been labelled with marker as the medium interaction between user and virtual furniture [14]. This system includes six main functions, which are picking, allocating, moving, rotating, deleting, and changing mode. The user can pick the virtual object by covering the virtual object with the cup. To allocate the virtual object, user can place the cup that contains the virtual object on the table. User can move the virtual object by sliding the cup. Furthermore, rotating the cup will rotate the virtual object and shaking the cup will delete the object. In interior design scenario, basic manipulation ability of the interaction technique is picking, allocating, moving, rotating, and deleting the virtual furniture. All the virtual furniture manipulations in the previous studies were limited where user cannot change the feature of the virtual furniture.


## 3.2. Studies Related to Spatial Awareness Interaction Technique

Depth sensing and motion tracking are useful features in spatial awareness interaction. It allows user to move in 3D space (x, y, and z position) and to track its orientation in yaw, pitch, and roll [23]. This 6 degrees of freedom of sensing capabilities allows the phone to respond to the physical movements of the user. Table 4 shows a review on existing spatial awareness interaction technique. There are 8 researches that have been reviewed.Table 4
Scoping review on existing spatial awareness interaction technique.
Name of invention Interaction technique: medium Interaction technique Scenario Type of AR Physical setup iSarProjection [15] Spatial (i) Involves only projection visualization technique Medical Projection based (i) Handheld projector(ii) RGBD camera (iii) Physical model  SAR/VR HYBRID SPACE [16] Spatial (i) Wand controller: points at a certain location and presses the controller’s trigger to start the drawing Architecture Projection based (i) Wand controller(ii) Projector  Garden [17] Spatial (i) Durovis Dive headset: user can see the world in VR via tablet that has been mounted on it(ii) Hands and Legs: create voxels to fill in the wireframe and draw objects Game Mobile based (i) Tango tablet(ii) Durovis Dive headset  Tabletop SAR [18] Spatial and tangible:rectangular object, scroll wheel Tangible(i) Rectangular object: specification of zoom region in the virtual volume with desired location and orientation(ii) Push button: locks zoom position in place(iii) Scroll wheel: increases or decreases the zoomed volume.Spatial(i) Handheld input device (push button): selection (ii) Handheld input device (scroll wheel): 2D zooming (iii) 6DOF pointing device: drills down on 3D projected onto the physical objects(iv) 4K monitor: displays detailed views of 3D visualization Information sharing Projection based (i) Handheld projecting device(ii) Physical object (iii) 6DoF Tracking (iv) Projector  Tag It! [19] Spatial (i) Tango: extends Google Glass’ sensing ability by sharing the location and pose of the user as well as tag the virtual information in real world(ii) Glass: overlays virtual tags based on the spatial information received from the Tango(iii) Glass touchpad: user can initiate the wireless connection by using a three-finger touch gesture on it Information sharing See-through based (i) Glass(ii) Glass touchpad(iii) Tango (iv) Target  Social spatial mashup (SSM) [20] Spatial (i) Tango: registers the virtual information on the face of the 3D object after social spatial mashup via tapping the virtual button on the device Information sharing Mobile based (i) Google Project Tango  AR interior designer [21] Spatial A depth camera is used to measure the size of available space in terms of supporting surfaces for new furniture(i) User uses mouse to interact and manipulate virtual furniture Architecture PC based (i) PC + Mouse(ii) Microsoft Kinect  PimpMyKitchen [22] Spatial A system that supports the early design process of kitchens and helps to ease designer and client in meeting session(i) Blank (white cabinet): moveable cabinet placed in an environment of the shape and size of the target kitchen(ii) Identical card: resizing projected components. Initially, the components automatically fit the available space(iii) SAR Swatches: choose colour, loading different design and save design (iv) Stylus: taping the cabinet surface to change materials Architecture Projection based (i) Projector(ii) Blanks (white cabinet)(iii) Card(iv) SAR Swatch(v) StylusProjection-based augmented reality is a normal setup for the spatial augmented reality. General technique of dynamically augmenting physical models was introduced back then. iSarProjection is using a handheld projector attached with an RGBD camera to project the texture on a physical model [15]. The physical model is mapped with corresponding textures, and it develops a real-time dynamic calibration, registration, tracking, and rendering. However, this study only focuses on the visualization in the spatial augmented reality environment.A hybrid space combining spatial augmented reality and virtual reality has been proposed and has obtained a positive feedback from interviewees [16]. The hybrid spatial augmented reality (SAR)/Virtual Reality (VR) hybrid space is a projection-based AR that enables multiusers manipulation with an augmented object, and immersive experiences. Virtual objects can be manipulated using the controller’s grab button. To start drawing, the user can directly point at the target location and press the controller’s trigger. Garden by Sing [17] is a study which used the game scenario to explore the mixed reality (MR) experience that combines both VR with AR. It allows users to transform their physical environment into a virtual garden they can play in. The exploration of mixed reality with motion tracking and depth sensing is going very well here. An immersive experience allows players to move through their surroundings untethered while exploring and interacting with the virtual world by using Durovis Dive headset where users can see the world in VR via tablet mounted on it. It consists of a simple hand and leg interaction where user draws a virtual object via voxel that is created by their hand and leg. Another projection-based AR is the Tabletop SAR [18] where it uses physical object such as cube and cone to interact with the 3D data. It is a combination of spatial augmented reality with the tangible interaction technique. The physical object in their study comes with a push button for selection and scroll wheel for 2D zooming. Users manipulate the physical object and push the button to lock the zooming position in the 3D data visualization area and then increase or decrease the zoom level using the scroll wheel.Besides the projection-based AR, researchers have also studied wearable augmented reality and spatial awareness mobile device such as Google Project Tango. Nassani et al. [19] invented a wearable system that allows people to place and interact with 3D virtual tags placed around them and it is named Tag It!. A head-worn wearable computer (Google Glass) and a chest-worn depth sensor (Tango device) were used as wearable technology. The users can create virtual tags with text content generated by voice input and place it on where they are looking at by wearing the system. Tango is utilized to extend Google Glass’ sensing ability by sharing the location and pose of the user as well as the tagged target position in the real world. The Google Glass then overlays virtual tags based on the spatial information received from the Tango, and the background of the glass display is set to black to act as an optical see-through display. Via the Google Glass touchpad, the user can initiate the wireless connection by using a three-finger touch gesture on it. Besides the Tag It! invention that benefits the user, the social spatial mashup (SSM) [20] for information sharing in public space also improves the interaction among human with overlaying the virtual information surrounding them. The proposed SSM is based on RGB-D SLAM to create a 3D feature map and allows users to locate information and contents in 3D space. Information placing is through tapping button in the Tango. The virtual information is presented in an image (a small logo) form, which is located on the face of the 3D object after social spatial mash up. The virtual information is well registered even with the movement of the Tango device.For the interior design area such as in the study of Tang et al. [21], Kinect was used to detect the spatial information of the physical environment where the furniture reseller can create virtual 3D furniture. The virtual furniture is rendered in the real scene of the home space via AR technology. The study performed an evaluation on the system usability, flexibility, and accuracy where overall result is positive, and the system is helpful, usable, efficient, and effective for home furnishing and furniture purchase. It is related to this study; however, it does not have the mobile ability where user can move and see the furniture in different perspective. Blanks (white cabinet) is a metaphor of the cabinets that are placed in an environment of the kitchen with actual shape and size [22]. It supports the early design process of kitchen. It will ease the discussion session between the designer and the customer. The designer can move the cabinet blanks to the desired location. Then, the designer and the customer can decide on the cupboards, the drawers, and other components. The system automatically arranges the components to fit the available space. The component can be manipulated by using the manipulation tools, which are two identical cards to resize the component and to use SAR Swatches for choosing the colour and loading and saving the kitchen designs. It also allows user to compare different designs simultaneously by quickly flicking through several different designs in a more natural way than using a menu. Final design phase allows the designer and the customer to set the material of the cabinet by tapping the cabinet surface with the Stylus.In conclusion, the previous studies have shown the possibility of the spatial awareness interaction between user and the systems such as the SAR/VR Hybrid Space, Tabletop SAR, Tag It!, Social Spatial Mashup, and Garden.


## 4. Current Kitchen Design Process

We have conducted interview session with the kitchen design company in Malaysia. Figure4 presents the current kitchen design procedures, which start with the face-to-face meeting between the customer and the designer. At the beginning of the process, the customer will need the measurement of the kitchen space and might come with the initial plan of their desired kitchen design to discuss with the designer. Some of them come with an example of their wanted design from the magazine or catalogue. However, the designer will have difficulty to follow the design from the magazine due to lack of measurement information in the magazine. Next, the designer will ask if there is any appliance such as fridge or microwave to be placed in the kitchen. Then, the designer will start to sketch the design after the customer has made the selection of cabinet material, colour, and accessories. Common furniture pieces that the customer requires are sink, stove, kitchen ventilation, and cabinet. The type of furniture, measurement, material, and accessories will have an impact on the price of the kitchen cabinet. After the customer makes the payment, the designer will produce a full 3D design in the design software. The designer and the customer will continue the discussion to finalize the design and then the designer will go to the customer’s home to confirm the actual measurement. After all the procedures have been done, the designer will proceed with the production of kitchen cabinet.Figure 4
Current design procedures.The weaknesses of the current procedure include difficulty of understanding the kitchen design imagination of both customer and designer. The designer cannot understand the customer needs through verbal communication. This problem causes time-consuming delay in the designing process. During the interview section, we have explained the MAR approach to solve the problem to the kitchen designer. The designers feel positive to use the MAR in smartphone platform instead of tablet, as smartphone is a common device that they use daily. The designers suggested that this application should allow the user to design their kitchen at their home and obtain the actual measurement beforehand so that the designer does not have to go to the customer’s home and confirm the measurement. Then, the customer can email or print the output design to show to the designer for further discussion. Based on the designer’s opinion, this application is useful and will ease the designers’ work in the designing process and overcome different imagination between customer and designer. Besides, it will also help to reduce the designing time.


## 5. Proposed Interaction Techniques

Based on the findings of the scoping review, this study has proposed two types of interaction techniques: tangible interaction technique and spatial awareness interaction technique for kitchen design. Tangible interaction technique is used mainly in the situation where the customer and designer are together in a discussion session. Normally, the discussion session is conducted in the kitchen design company. Tangible interaction technique requires physical board as interaction space and cube as medium interaction. The tangible interaction technique will use cube as interaction medium to represent kitchen furniture and kitchen features as shown in Table5. For example, the cubes will represent cabinets, sinks, stoves, and kitchen ventilation labelled with related markers to be arranged on the physical board as shown in Figure 5(a). For sink, stove, and kitchen ventilation, the colour features of this furniture are included in the physical cube where the user can choose the colour of the virtual object by rotating the cube that is labelled with its colour. For the cabinet, the cube is labelled with the surface of cabinet. For example, the user can choose the surface of the cabinet by manipulating the cabinet cubes that are labelled with different type of cabinet’s surface. Figure 5(b) illustrates the virtual cabinet shown after the user places the cabinet cubes on the physical board. For the cabinet accessories and the cabinet colour features, the proposed technique uses the cards as interaction medium. The user can choose the colour and accessories of the cabinet by placing the colour card and the accessories card on the designated physical board.Table 5
Tangible interaction technique method.
Interaction medium Virtual furniture representation Selection of the virtual furniture’s feature Cube Cabinet Cabinet surface Stove Colour Sink Colour Kitchen ventilation Colour  Card  Colour of cabinet Accessories of cabinetFigure 5
(a) User interface of tangible interaction technique. (b) Virtual cabinet is displayed based on the selection.
(a) (b)Spatial awareness interaction techniques do not require physical board because users can design their kitchens directly in their home kitchen using mobile phones. Spatial awareness techniques enable the users to measure available space to add new furniture. Besides, the users can view 1 : 1 scale of the virtual furniture to see the suitability of the virtual furniture with the available space. The user interface of this technique contains four menu buttons representing cabinets, stoves, kitchen ventilation, and sinks as shown in Figure6(a). When user clicks on the cabinet menu button, it will display a list of cabinets for the user to choose. A popup box will be displayed for the user to select the surface type, colour, and accessories of the cabinet as shown in Figure 6(b). If the user clicks on the sink menu button, list of the sink will be displayed. After the user selected the sink they want, a popup will be displayed for the user to select the colour of the sink. A similar process will be used for stove and kitchen ventilation menu button. Spatial awareness interaction techniques can detect walls and floors. Therefore, after the selection has been made the virtual furniture will snap on and not penetrate the physical wall.Figure 6
(a) User interface of spatial awareness interaction technique. (b) Display of popup box.
(a) (b)There are few limitations in spatial awareness interaction technique; namely, this technique cannot be used to fully furnish the kitchen. This is because the spatial awareness interaction technique can detect the existing furniture and act as the boundary or wall of the kitchen space. Thus, the virtual furniture that customer choose will appear in front of the physical furniture. Other than that, customers lack the knowledge in kitchen design; thus, some of the existing water piping or the electric socket area might make some kitchen furniture unable to be located at the location they desired. However, the customer does not know those limitations, and this makes the kitchen design not achievable. This problem can be tackled by having further discussion session about the design between the customer and designer. The discussion session will be easier because they already have the design based on the customer’s idea. This will minimize the gap of the different imagination issue.


## 6. Conclusion

In kitchen design, the traditional approach has a few limitations such as mistake in measurement and different imagination between customer and designer. The user has different final kitchen output, which is usually different from the sketch design from the designer. Hence, this traditional method is inconvenient and not efficient. As a solution, the virtual furniture can be overlaid into real scene by using MAR application. The user can see the overlaid virtual kitchen outcome that could fit with the actual kitchen environment.This paper has presented a scoping review on the tangible and spatial awareness interaction technique and proposed the tangible and spatial awareness interaction technique for the kitchen design authoring tool. The proposed interaction techniques will be developed and refined by conducting a heuristic evaluation with the AR expert. After the refining has been made, both techniques will be improved and evaluated with the kitchen designer and the customer through usability testing.



---

*Source: 5320984-2018-05-21.xml*