# The Comparison of Performance, Efficiency, and Task Solution Strategies in Real, Virtual and Dual Reality Environments

Frederic Raber, Antonio Krüger, Gerrit Kahl

► To cite this version:

Frederic Raber, Antonio Krüger, Gerrit Kahl. The Comparison of Performance, Efficiency, and Task Solution Strategies in Real, Virtual and Dual Reality Environments. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.390-408, 10.1007/978-3-319-22723-8_31 hal-01610846

HAL Id: hal-01610846

https://inria.hal.science/hal-01610846

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# The Comparison of Performance, Efficiency, and Task Solution Strategies in Real, Virtual and Dual Reality Environments

Frederic Raber, Antonio Krüger, and Gerrit Kahl

DFKI GmbH, Saarbrücken, Germany
{frederic.raber, krueger, gerrit.kahl}@dfki.de

**Abstract.** Using virtual models of a real environment to improve performance and design effective and efficient user interfaces has always been a matter of choice to provide control of complex environments. The concept of Dual Reality has gone one step further in synchronizing a real environment with its virtualization. So far, little is known about the design of effective Dual Reality interfaces. With this paper we want to shed light on this topic by comparing the strategies, performance and efficiency in a real, virtualized and a DR setting given a complex task. We propose a cost and efficiency measure for complex tasks, and have conducted an experiment based on a complex shelf planning task. Our results show that for certain tasks interacting with the virtual world yields better results, whereas the best effectivity can be observed in a Dual Reality setup. We discuss these results and present design guidelines for future Dual Reality interfaces.

**Keywords.** Immersion; Dual Reality; Efficiency; Performance differences in real and virtual environments

## 1 Introduction

More and more tasks that were previously successfully conducted in a real-world environment are now being virtualized. For example, car design, which was previously done using miniature models, is now done on a PC using a CAD tool. Cars are designed and tested virtually, and architectural design and structural engineering calculation is done completely on virtual models, replacing their real counterparts. Both virtualization and a real environment have their advantages: On the one hand, virtual tools are fast and easy to use, as the physical demand is reduced. The virtual interface can be enriched by additional visual clues, using coloring of parts of the scene, lighting or textual information. For example, in [16], the work piece is visualized so it cannot be occluded, and additional textural information in the form of process data is displayed. On the other hand, a real-world environment with real objects has better haptic feedback. Size, height and distance estimation is easier in the real world, as we will discuss in the next section. The concept of *Mirror Worlds*, first mentioned by David Gelernter [9] and later by Lifton and Paradiso under the term *Dual Reality* [12],
describes a setting where both worlds, a real and a virtual world, are connected to-
gether and influence each other. Lifton and Paradiso define it as follows:

**Dual Reality** is an environment resulting from the interplay between the real world and the virtual world, as mediated by networks of sensors and actuators. While both worlds are complete unto themselves, they are also enriched by their ability to mutually reflect, influence, and merge into one another. [12]

In Dual Reality, an event in one world can, but does not have to, cause a corre-
sponding action in the other world. The users can act either with the real environment
or with its virtual counterpart. Every action can be reflected in the corresponding
counterpart. Although not directly stated in their paper, a mechanical entity is implic-
itly needed, which synchronizes both worlds. Consider the model of a virtual apart-
ment in Dual Reality: The user can turn on a lamp in the virtual environment. In the
real counterpart, the light is also turned on remotely by the software. In turn, if the
user turns on the light in the real world, this is recognized by the software, which then
turns on the light in the visualization. Our goal is to understand how people interact
with a Dual Reality setup, whether they take advantage of the possibilities of this
concept, which possible problems arise and how this interaction can be improved. We
are interested in questions such as “Are the users acting the same way in a virtual
replica and in the physical environment?”, “How efficient is a Dual Reality setup
compared to the real and virtual world?”, and “Which interface type should I use to
meet my requirements?”

In order to investigate these questions, we have created an experimental setting,
consisting of a real and a virtual environment, representing the same setup and influ-
encing each other. Subjects were asked to perform a complex task requiring both
strategic planning and physical actions. We propose a cost function, which allows us
to judge and compare the efficiency of each task. As an initial step we were interested
to learn more about the differences in terms of performance and behavior between
real-world, virtual-world, and Dual Reality interaction. Dan Montello argues in his
work [14] that the definition of space is a perceptual problem, and gives the defini-
tions of four different sizes of space. The space that can be seen by the user without
locomotion is called the *vista space*. A bigger space that can only be apprehended
with a significant amount of locomotion is called the *environmental space*. Most of
the related work that compares the task solution strategies between a virtual and real
environment does not take Dual Reality settings into account, and uses only a small
space (such as the size of a table), meaning it is clearly limited to a vista space. The
Dual Reality systems that are known from the literature, but which are not evaluated
in terms of performance and efficiency, are situated in the environmental space. Our
experimental setting, which evaluates a real, virtual, and a Dual Reality interface,
resides somewhere between the vista and environmental space: The scene can be
viewed as a whole from a single point, but the detailed information and actions need-
ed for the task can only be performed with significant locomotion. The remainder of
this paper is organized as follows: The next section will first discuss related work and
provide some background information on the topic of Dual Reality. Then we will
present the experimental setup and how we designed the task to be performed in a
real, a virtual, and a Dual Reality world. We then discuss the experiment, including a
pilot study, and its results. Finally, a discussion and conclusion complete the paper.
## 2 Related work

For our work, three different aspects are of importance: first, the differences in perception of a virtual world or object compared to a real environment; second, the research field of *Dual Reality*; and third, comparisons between *interaction in a real and in a virtual* environment. Research in the field of visual perception has shown that users perceive a virtual world significantly differently than the real world. A first problem is the perception of rotation, orientation and shape of a three-dimensional object. Dobbins et al. [7] showed that orientation can be perceived differently, depending on the position of the object in relation to the beholder. Mistakes in the perception of the metric structure of 3D objects from multiple cues were previously discovered by Todd et al. [17]. We expect that differences in visual perception will influence performance and behavior in real and virtual worlds.

There exist several examples of such a setup of complex worlds in the Dual Reality paradigm: Back et al. [4] present a Dual Reality chocolate factory, consisting of a real factory and a virtual model. The applications of this project are, first, to allow a virtual trip through the factory, as well as the remote control of the machines inside the factory for authorized persons. In addition, the states of the machines in the real world, as well as interactions with them, are reflected inside the visualization. Davies and Callaghan [6] examined how human behavior can be captured and learned by sensors. Their goal is to create an autonomous virtual avatar, whose behavior seems natural. They instrumented an apartment with motion sensors, and created a corresponding virtual world in 3D. The behavior of the user in the real world is perceived and also visualized in the virtual component. Conversely, interactions with the virtual world (e.g. turning on the light), will also affect the illumination of the real apartment. Khan et al. [11] created a virtual supermarket, displayed on a CAVE (Cave Automated Virtual Environment). The interaction is realized by a “human joystick” principle: The camera is moved with respect to the user’s position in relation to the center of the CAVE. Their objective was to evaluate the user's experience of pervasive applications within a virtual environment, and to show the potential of evaluating location-based services, especially location-based advertisements in a virtual supermarket. They claim that a virtual world has the advantage of being fully controllable and adaptable to the researchers’ needs, but they did not investigate the differences between their simulated and a real environment. In contrast to the chocolate factory and the virtual avatar mentioned before in this subsection, this visualization has no real counterpart, so it does not form a Dual Reality system. Still, it is the virtualization of an environment as it can exist in the real world. Therefore, the interaction and visualization techniques are related to our experiment.

The design of the two Dual Reality worlds (or the virtualized world), and especially the interaction possibilities, are of interest for our work. We designed our real and virtual environment similarly to these systems in terms of visualization as well as interaction possibilities and techniques, to discover the differences in interaction and behavior between the two environments, as well as a DR setup containing both environments at the same time. In the following we will discuss several studies which compare the performance and behavior of a digitalized 2D representation (e.g. con-
trolled via mouse or touch) with a physical or tangible version for the same task and
domain. Kozak et al. [1] observed in the past that training in the virtual world does
not necessarily lead to an improved performance in the real world. In their study,
subjects had to place a set of cans on a table according to given positions. The first
test group was able to train in advance using the real cans and table. A second group
used a virtual reproduction of the setup, using a data glove and a head-mounted dis-
play. A third group had no training at all. Only the first test group was able to perform
significantly better; the group using the virtual version did not perform better than the
group without training. A simple task where a ruler or several simple geometric ob-
jects have to be aligned to fit a template is presented in [18]. The results show that
subjects needed significantly more time on the touch-based system. In contrast to our
experiment, the tasks here are tasks which are easy to perform, like aligning a ruler to
a given shape. Lucchi et al. [13] compared actions like selection, scaling, rotation and
positioning using a touch-based version or tangible objects. Here as well, the tangible
version outperformed the touch version in terms of time, precision and number of
translations needed to reach the desired goal. Still, a comparison of complex tasks is
absent from this paper; only easy rotation and scaling tasks are performed there. More
complex tasks are presented in [3] and [2]. The participants were asked to solve a
puzzle by using either an interactive surface (based on touch input), or real puzzle
pieces lying on top of the surface. The results show that the virtual version was out-
performed by its tangible counterpart. The behavior and the percentage of time devot-
ed to each sub-task were significantly different. While this is similar to our setup, the
task itself can be considered rather simplistic in terms of strategy and behavior, if for
example compared to a more complex control task in a factory or when finding a
solution to a spatial configuration task.

In summary, several studies have been carried out comparing the performance be-
tween a virtual and a real condition for simple tasks, involving simple actions. We
will extend this work for a more complex task, which requires a strategy, as well as
more complex actions including locomotion. We will introduce a new test condition,
namely the Dual Reality condition, which is also compared against the virtual and real
conditions. An efficiency measure for all three conditions will be proposed. We will
observe and discuss differences in terms of performance, efficiency, as well as the
number and duration of actions conducted in order to complete the task, in order to
give guidelines on the optimal interface for a given environment and task.

3 Experimental setup

We decided to take a pick-and-place task, which had to be extended to form a more
complex task, requiring strategic planning as well as a higher amount of locomotion
as stated in the introduction. Instead of giving the user a specific target location, we
gave him a complex formula which scores the placement of the object, depending on
its position and which other objects are situated in its surroundings. To form a mean-
ingful task, we decided to replicate a realistic task from the retail domain, namely that
of "shelf planning": Retailers have to plan their shelf layouts (i.e. the order and posi-
tion of product placements in a shelf) to optimize their profit. We have designed a real and a virtual environment where real and virtual products could be placed at arbitrary positions on the respective shelves in a shelf unit. In the Dual Reality condition of our experiment, both environments can influence each other, and are always “synchronized”. Each product placement or movement which is done in the virtual environment will also be applied automatically to the real shelf, and vice versa. Normally, in a Dual Reality system, the “synchronization” of the real world is done by machines, such as a robot. Details on how this robot was emulated for our experiment will be given later. For our efficiency calculation, we will assume that this task is done by a robot or an automated process.

Each product is assigned to a price category. Depending on the price and the placement within the shelf unit, the overall profit is calculated. Profit is influenced by the placement (some positions in the shelf unit are more profitable than others) and on which other products have been placed nearby (products of the same price category reduce each other's profit). Details on the profit calculation will be given below. The main task consists of maximizing the profit of the shelf unit as a whole.

## 3.1 Efficiency calculation

To be able to compare the efficiency of the tasks, an efficiency function is essential. In this subsection, we give a general formula which has to be refined for a specific task. We will create a specific formula for our shelf planning task in the next section. We define efficiency as the fraction of the performance $P$ (in terms of score reached for the task) and the cost $C$ needed to achieve this result:

The shelf planning task can be divided into several types of sub-tasks. We will denote this type-set of sub-tasks as $ST$, their elements as $ST_i$ and the number of times this sub-task type was executed during the experiment task as $|ST_i|$. The cost depends on the number of times each sub-task $ST_i$ in $ST$ is conducted, as well as the cost for each of them ($C(ST_i) := \sum_{ST_i \in ST} (|ST_i| * C(ST_i))$)

Each subtask is conducted by an entity, which can either be a real person, or a machine such as a robot. Each entity has a different cost for operating over a certain period of time. For a human this would be his salary; for a machine, the operating cost. The cost of a given subtask is therefore calculated by multiplying the cost per hour $C_{hour}(E)$ of the entity $E$ that conducts the sub-task

by the time needed to complete it (Time(ST_i)) : $C(ST_i) = C_{hour}(E) * Time(ST_i)$

The efficiency formula then is calculated as follows:

$$Efficiency = \frac{P}{\sum_{ST_i \in ST} (|ST_i| * (C_{hour}(E) * Time(ST_i)))}$$

This formula needs to be refined in the next step according to the given experiment task that should be measured. There are three domain-specific variables: *Time on average* needed to complete each individual Subtask $ST$ (*Time(ST_i)*), *Cost per hour* of the entities that fulfill these subtasks ($C_{hour}(E)$) and *Measure of the performance* $P$. It is not our primary goal to show which of the interfaces is most efficient, but to see how changes in different parameters, such as cost per hour/salary of a worker or time
needed for a subtask, can influence the efficiency and make another interface concept become the most efficient one. This should give the reader an idea of whether he should prefer a real, virtual or DR interface for his specific task. Calculating the measures is rather straightforward: for example for a pick-and-place task, the performance P could be the number of placed objects divided by the sum of offsets of each object compared to its target position. There is only one sub-task, namely the pick-and-place subtask, whose average time can be computed in a short trial. The cost per hour is the hourly wage of the human performing the task. In the experiment section, we will determine these variables according to our shelf planning task. The next section will give an overview of the schematic setup of the real environment, followed by the design of the virtual environment, and the Dual Reality system which is a linkage of these two environments.

## 3.2 Schematic setup

The schematic setup is shown in Figure 2 and is the same for all three conditions. The worlds are designed in exactly the same way, illustrated in Figure 1. For the virtual version, we used a 3D scene displayed on a 2D touchscreen for representing a virtual version of the real world.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Environment in the real, virtual and Dual Reality conditions</figcaption></figure>

The products are initially placed in the right shelf unit ("start shelf"), and have to be placed inside the left one ("target shelf"). The touch display running the virtual environment is placed on a table in between the two shelf units. The user stands in front of these two shelf units ("interaction zone"), so he has access to both the real shelves as well as the virtual environment on the display. Behind the shelves is an assistant zone, where the experimenter as well as an assistant can observe the experiment.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Schematic setup of the experiment (all measures in cm)</figcaption></figure>
3.3 Interaction design

Real environment

The subject uses real products and shelves to accomplish the task. We implement-
ed several assistance systems, which should help the subject in achieving a good
score, without reducing the cognitive effort too much. We displayed the actual score
as well as the change with respect to the last step on the screen between the shelves,
giving the subject an idea of whether his actions were going in the right direction.
Additionally, we installed LED lights in different colors on each shelf of the left shelf
unit (target shelf). As soon as the subject touches a product or holds it in his hands,
the LEDs are turned on, indicating if there is not enough space to place the product
(red light); there is enough space, but the score would decrease (yellow light); or there
is enough space and the score would increase (green light) after placing the product
(see Figure 3).

<figure><img src="image_4.png" /><figcaption>Fig. 3. Lights indicate available space and score</figcaption></figure>

Virtual environment

We designed the virtual world to be as similar as possible to the real world in order
to ensure comparability. Therefore we decided against a 2D representation and used a
3D view instead. A screenshot of the virtual model is shown in Figure 4. For addi-
tional functions such as switching views between shelves or zooming, several com-
mand buttons are displayed in the lower left corner of the interface.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Setup in the virtual world condition</figcaption></figure>
Several interaction techniques are provided, based on single-touch input:

- **Selection of a product**: For selecting a product, a single tap on the product is sufficient. Tapping the product again, or selecting another product, deselects it again. The currently selected object is highlighted by a red overlay on the product.

- **Placement of a product**: A selected product can be placed by tapping on the exact position on the shelf where the product should be placed. The program automatically moves the adjacent products aside if the selected space is not sufficient, as long as there is enough space on the shelf. If no space is left on the shelf, a corresponding message appears in the upper right corner of the screen as feedback to the user. The product remains selected and can be placed at some other location.

- **Moving a product**: A product inside the shelf unit can be moved simply by dragging and dropping it. A move is possible only on the same shelf; moving it to another shelf is not allowed using this technique. In that case, a placement action has to be performed.

- **Shelf views**: When clicking on one of the two rightmost buttons, the user gets an overview perspective of the contents inside the left or right shelf unit. This facilitates a better identification and selection of products. In the new visualization, an additional button appears to get back to the shelf view.

- **Zoom**: We implemented a zoom functionality, allowing the user to zoom in or out in the current perspective, to have a more detailed view of a shelf unit or product desk. This corresponds to the action of walking towards the shelf unit.

Users were not forced nor encouraged to use zooming or view switching, if they did not want to. We implemented the assistance functions from the real environment (see last section) in the virtual environment as well: Whenever a product is selected, the shelves inside the target shelf are colored according to available space and possible score after the placement of the selected product. The score is displayed in the upper left corner of the screen.

## Dual Reality environment

In this environment, both the real products and visual cues of the real environment, as well as the virtual system using the touchscreen, can be used. The subject should always have the possibility to switch between the two environments at will. Therefore, both environments always have to be synchronous: Whenever an action is conducted in one of the two worlds, it will be mimicked in the other world. If the subject places a real product on a shelf, the product is also automatically placed in the virtual environment. When the subject uses the touchscreen to place a product in the virtual environment, the real product also has to be transported to its new location.

## 3.4 Pilot study

Prior to our experiment, we conducted a pilot study with a slightly different setup in order to find first differences in people's behavior and performance in a real and in a virtual environment, and to decide which measures might be interesting to observe
in our main experiment. Twenty-seven students volunteered to take part in the pilot study. We tested only a real and a virtual condition without any assistance functions. The products were initially placed on two product tables, and had to be placed inside two target shelves, which were standing opposite each other. All subjects were asked to optimize the product placement in the two shelf units in order to maximize the profit of products from three different price ranges (high, medium, low). For more detail on how the profit was calculated, see the later section on the main experiment.

The results reveal that the profit was significantly higher for the real condition ($M_{real}=0.943, M_{virtual}= 0.909, p=0.02$). We observed a higher number of interactions, especially product movements ($M_{real}=16.62, M_{virtual}=24.35, p=0.04$), in the virtual condition. Most participants used only one hand to interact with the real world and never picked up two products at once. Two-handed interactions were conducted by only 50 percent of all participants. Those subjects performed on average four two-handed interactions per session. The time needed for placing a product was significantly higher for the real environment ($M_{real}=4371, M_{virtual}= 3362, p=0.02$).

# 4 Main experiment

## 4.1 Hypotheses

Although there are small differences in the experimental setup, we expect a similar outcome in terms of performance in the main experiment, as all conditions have the same assistance functions guiding the user to get an optimal profit. Therefore we expect the real condition to perform better than the virtual one (H 1). We did not calculate the efficiency for our pilot experiment, but observed a significantly higher number of interactions and a lower performance in the virtual condition. As the Dual Reality interface should combine the advantages of both worlds, we expect the best efficiency to be in this condition, and the second best for the real setup (H 2), as the latter already outperformed the virtual version in our pilot study. Relatedly, we expect the most interaction with the virtual interface and the least within the real condition. The number of interactions in the Dual Reality setup should lie in between (H 3). As interacting with a virtual interface should be easier in terms of physical demand, we assume more interaction and less workload in that condition (H 4). Regarding the Dual Reality condition, we assume that most people switch between the virtual and real environment either in their physical interactions (H 5) or by switching the view focus (H 6).

## 4.2 Apparatus

Within each shelf unit, one shelf is at eye level and one below. For calculating the “eye level”, we first retrieved the current average body height from the German Federal Bureau of Statistics (DESTATIS)<fnref n="1" />, which is 171 cm. From this value, we sub-

<footnote n="1">¹https://www.destatis.de/DE/Publikationen/Thematisch/Gesundheit/Gesundheitszustand/Koerpfermasse5239003099004.pdf</footnote>
tracted 15cm (a value derived from a pre-test) to arrive at an eye level of about 155cm. We then designed the shelf to be at eye level, so the placed products can be seen best from that height. We arrived at a shelf height of 142cm. The second shelf is placed below this, at 105cm. Shelves in the starting shelf unit had a width of 82cm, whereas the ones in the target unit were only 56cm wide. We designed the latter to be smaller so not all the products inside the start shelf unit can be placed in the target; the subjects had to make a selection. The table between the shelves has a squared shape with a width of 90 cm, where only half of the table is visible to the subject. The monitor is placed on the table at a height of 90cm. On the other side of the shelf is an “assistance zone” for the experimenter and an assistant; this is completely covered by cloth, so it is hidden from the experiment participant. The shelves of both shelf units are accessible from the back as in a puppet theater, so that the assistant can grab and place products undetected by the participants, simulating the “robot” which synchronizes the real world in a Dual Reality environment. The experimenter has a second screen inside the assistance zone, which shows a duplicate screen of the participant's touch display. It allows the experimenter to “synchronize” the virtual world in a Wizard of Oz style, whenever an action is done in the real environment by the participant.

The experiment was designed in a within-subject design: Each participant performed the task in the real, virtual and Dual Reality conditions, one after another. Additionally, we had three sets of products, A, B and C. Each set contained different products, although some of the products appeared in several sets. Each participant had the possibility to become familiar with each interface in advance, so we did not expect any influence on the experiment results, neither from the order of conditions, nor the order of product sets. Nevertheless, we balanced the order in which conditions and product sets were used. The products of a set are placed in the same style and ordering for each experiment. In order to capture eye movements, subjects use a mobile eye tracker in the Dual Reality condition.

## 4.3 Participants

We recruited 17 participants over several university mailing lists. Their ages ranged from 21 to 44 years (average 27.5). We had 10 male and 7 female subjects. Each was paid 10 Euro for the experiment.

## 4.4 Task

The subjects were given a set of products and one empty shelf unit. Every product placement contributed to an overall profit, as we will describe in the next section. The main task was to place the products in the shelves so that the overall profit was maximized. In the DR condition, subjects were free to make use of only the real, only the virtual or both worlds. We set no time limit, and the subjects could change the placement of the products as often as they wanted. The test run was marked as finished once the subject reported being satisfied with the current shelf layout.

The formula to derive the profit for the placements was based on related work from economics [5,15,10]. To enable subjects to better estimate the profit of their shelves,
we based the formula on a few simplistic assumptions. The first aspect taken into account was the effect of the inventory level on the product demand, as proposed by Drèze et al. [8] and implemented by Hwang et al. [10] and Murray et al. [15]. It reflects the fact that products at eye level receive a significantly higher demand than products located below or above. In our formula, we modeled this effect by a deduction of 10 percent for products below eye level. The second aspect is the space and cross-space elasticity as described by Corstjens and Doyle [5]. This earlier theory says, on an abstract level, that if two products of the same price and product category are next to each other, or on the same level in neighboring shelves, the profit of both is reduced.

In detail, the total profit was calculated as follows: Each product has a different price, which is printed on the real product or the 3D model of the product in the virtual interface. We divided the products into three price categories: low-priced products (price < 0.50€), medium-priced products (price between 0.50€ and 1€) and high-priced products (price > 1€). The profit of the target shelf unit is the sum of the profits of its contained products. If a product is placed on a shelf below eye level, its profit is the product price reduced by 10 percent. If more than one product from the same price category is placed at the same eye level, the profit is again reduced. We call such a group of products (on the same level and in the same price category) “colliding products”. For every collision, the profit of each colliding product is reduced by an additional 20%. The reduction can be at most 100%.

More formally, let $C(p) = \text{collisions}$ for a given product p (e.g. the products in the same price level and placed at the same shelf height as p), $Pr(p) = \text{price}$ of a product, as printed on the product, and $H(b) = \text{handicap}$ of a shelf, respecting its height level. It is 0 for the shelf at eye level and 0.1 for the shelf below. Then the overall profit $P$ is inductively defined as follows:

Profit of a product p: $P_p = Pr(p) * (Max(0; 1 - (|C(p)| * 0.2)))$

Profit of a single shelf b: $P_b = (1 - H(b)) * \sum_{products p \ on \ b} (P_p)$

The profit of the target shelf unit is then the sum of the profit of all of its shelves. The setup included more products than could be placed into the shelves. The subjects had to decide which products they wanted to use, and which would remain in the starting shelf. Participants were informed in advance of this effect by the experimenter. Apart from the assistant systems described earlier, we gave no further advice or strategy on how they could best complete the task.

## 4.5 Procedure

First, the participants were given written instructions. After possible questions were answered, a training phase started where first a short introduction to the interface was given. The introduction was done in a live demo by the experimenter. Following this, the subjects were given time to familiarize themselves with the interface. Once they stated they were comfortable with it, the main experiment phase started, where the users executed the given task in one of the three conditions. The starting
condition was selected in a balanced way, as described in section Apparatus. This step was repeated with the second and third conditions. Each condition ended with a short two-page questionnaire, consisting of a NASA-TLX, and an additional page where users could explain their feelings towards efficiency, learnability and enjoyability of the respective condition, and what differences between conditions they perceived.

## 4.6 Efficiency formula refinement for the experiment task

The interface allows two sub-tasks: The placement task, e.g. picking up the product, transporting it to the desired location and dropping it off, and the movement task, which is moving the product inside the shelf without picking it up. In our example, let us assume that we have three different entities that can conduct this task: The expert has the highest qualification and salary. His part in the task is to design the layout for the shelf. The assistant has a lower qualification. He can place the products according to a given layout, but cannot design a shelf layout himself. The robot is the cheapest entity, but can only reproduce an interaction in the real environment if it was done in the virtual environment, as described in our definition of Dual Reality. In our three conditions, we then have the following sub-tasks that have to be conducted, in order to get a shelf layout using real products:

- **Real environment:** The placement of the real products is done directly by the expert; we have only the cost of his placement and movement tasks. Our set of sub-tasks ST therefore consists solely of sub-tasks done by the expert: ST={ST<sub>plc_r_exp</sub>;ST<sub>mv_r_exp</sub>}

- **Virtual environment:** All actions by the expert are done virtually. Therefore, an assistant needs to fill in the real products after the design is finished (sub-task ST<sub>plc_ass</sub>): ST<sub>plc_v_exp</sub>;ST<sub>mv_v_exp</sub>;ST<sub>plc_r_ass</sub>

- **Dual Reality environment:** Some of the expert's actions are done using the real shelf (ST<sub>plc_r_exp</sub>;ST<sub>mv_r_exp</sub>), some virtually (ST<sub>plc_v_exp</sub>;ST<sub>mv_v_exp</sub>). The expert's virtual manipulations are executed in the real world by the robot entity (ST<sub>plc_rob</sub>;ST<sub>mv_rob</sub>):

$$ST=\{ST_{plc_r_exp};ST_{mv_r_exp};ST_{plc_v_exp};ST_{mv_v_exp};ST_{plc_rob};ST_{mv_rob}\}$$

The interaction times needed for each of the subtasks are calculated based on the results of our experiment, which will be given in the next sections. We found the following average interaction times: human placement in the real environment: 4130 msec; human placement in the virtual environment: 1698 msec.

Our experimental setup did not allow us to measure the time needed for a movement, as this action is too fast to be annotated correctly by the experimenter. We estimated the times in a short trial, using a stopwatch: human movement in the real environment: 1933 msec; human movement in the virtual environment: 1633 msec.

To allow a fair comparison, we designed both conditions, the virtual and the real one, as it is typically done by (software) engineers, using a screen for the virtual version, and a physical environment for the real one. Because the real environment typically involves a higher amount of locomotion, it requires more interaction time than
the virtual environment. For the Dual Reality condition, we did not have a robot for our experiment, but there are industrial robots like the KUKA KR-16 which can conduct the placement/movement tasks. We estimated the time needed for those tasks using the datasheet<fnref n="2" /> as follows:

- Grabbing/releasing the product: max. 90° rotation by axis 5, 45° by axis 6 → 316ms

- Pull back/forward to shelf: max. 45° rotation by axes 2/3 → 288ms

- Swivel between start/target shelf: max. 90° by axis 1 → 577 ms

- Movement inside shelf: max 45° rotation by axes 1/2/3 → 288 ms

→ Time(plcmt.) = grab + pull back + swivel + pull forward + release = 1785 ms

→ Time(movemt.) = grab + movement inside shelf + release = 920 ms

## 4.7 Measures—dependent variables

We divide our measures into two categories: task and sub-task measures. A task measure is taken during the whole test-run, and scores the overall result of the complex task. A sub-task measure is taken in relation to a specific sub-task that is conducted during the overall task. For the task, we recorded for all three conditions: **Profit** of the target shelf unit with the user's solution, compared to the optimum solution; **Efficiency** of the task for each condition, according to our efficiency measure; and **Workload** according to the NASA-TLX. In the Dual Reality condition, we additionally documented: **Number of context switches** the user performs within his physical actions and **Number of visual context switches** in terms of visual attention switches between the two environments.

The idea of the *profit* measure is to compare how successful the participants were in accomplishing the given task. The *profit* is a percentual value, comparing the overall profit reached by the subject against an optimum shelf arrangement with the highest possible profit. We calculated this optimum on the given products and space using a brute-force algorithm. Measuring the profit helps us to investigate hypothesis 1. For us, *visual context switches* are only switches where the user looked at the main part of the screen containing the 3D model. Only looking at the score is not a context switch for us. We used the results from our eye tracker to discriminate between those two cases.

For each sub-task, such as placement or movement of a product, we recorded the **number** of times the sub-task was conducted during the task as well as the **time** on average that the user needed to complete the sub-task. The analysis of the measures was done automatically for actions in the *virtual* environment by the software. For activities in the *real* environment, the subject's actions were duplicated in this software by the experimenter on his display inside the assistance zone.

<footnote n="2">²http://www.kuka-robotics.com/res/sps/e6c77545-9030-49b1-93f5-4d17c92173aa_Spez_KR_16_en.pdf</footnote>
# 5 Results

The total time for the whole experiment varied from 15 to 30 minutes, as we set no time limit. On average, each participant needed about 20 minutes. All measures were pairwise compared using a paired t-test. The answers from the questionnaire were evaluated using chi-square.

## 5.1 Task measures

In contrast to our initial hypothesis H 1, the overall profit in the real-world condition was significantly lower than for the virtual condition ($M_{real}=0.93$, $M_{virtual}= 0.96$, SD=0.04, t=3.51, p<0.005), as shown in Table 1. The Dual Reality score lay between the two, but without any significant difference from each of the other conditions ($M_{dr}= 0.945$). Although this profit was best for the virtual condition, the efficiency, according to our efficiency measure, was highest for the Dual Reality setup. The real setup was significantly less efficient ($M_{real}=0.67$, $M_{dr}= 1.09$, SD=0.64, t=2.72, p<0.05), as was the virtual condition ($M_{virtual}= 0.63$, SD=0.59, t=3.2, p<0.05). We could not find any significant difference between the efficiency of the real and virtual interfaces.

We measured a significantly higher workload for the real compared to the virtual ($M_{real}=4.95$, $M_{virtual}= 2.94$, SD=1.01, t=8.2, p<0.005) and Dual Reality ($M_{dr}= 3.02$, SD=1.12, t=7.12, p<0.005) conditions. Regarding the DR condition, we observed 7 of 17 participants switching between the real and virtual environments within their physical actions, for example placing a first product on the real shelf and then another product inside the virtual environment on the screen, or vice versa. Those who did such a context switch did it twice on average during the experiment. 15 of the subjects switched their eye focus between the two environments (26 times on average). We recorded the time users spent in both conditions of the DR setup, but could not find any significance. Some users acted mainly in the real environment, whereas others used the virtual environment all the time. Differences in terms of performance or efficiency could not be shown between those two groups. We evaluated the questionnaire using a chi-square test. Only one of the questions was of high significance in favor of the real environment, all others being insignificant: Users stated that the real interface made it easier to solve the task ($X^2 = 24.93$, p<0.005).

<table><caption>**Table 1.** Results of the experimental task: Real, virtual and Dual Reality (DR)</caption><thead><tr><th>Measure</th><th>Real</th><th>SD-Real</th><th>Virtual</th><th>SD-Virt.</th><th>DR</th><th>SD-DR</th></tr></thead><tbody><tr><td>Profit</td><td>0.926</td><td>0.06</td><td>0.963</td><td>0.04</td><td>0.941</td><td>0.06</td></tr><tr><td>Efficiency</td><td>0.67</td><td>0.24</td><td>0.63</td><td>0.22</td><td>1.09</td><td>0.62</td></tr><tr><td>Workload</td><td>4.95</td><td>0.77</td><td>2.94</td><td>0.49</td><td>3.02</td><td>0.65</td></tr></tbody></table>
Table 2 shows the results for the sub-tasks, regarding the number of times the task was conducted during an experiment for both conditions, as well as the average time needed for the task. As the logging of the experiment was done by an annotation tool used by the experimenter, we could not record the time needed for a movement: a movement task is completed too quickly to be annotated correctly.

<table><caption>**Table 2.** Sub-tasks measure results</caption><thead><tr><th>Measure</th><th>Real</th><th>SD-Real</th><th>Virtual</th><th>SD-Virt.</th><th>DR</th><th>SD-DR</th></tr></thead><tbody><tr><td>Placems.</td><td>13.59</td><td>4.73</td><td>21.65</td><td>9.2</td><td>14.88</td><td>5.3</td></tr><tr><td>Plc. time</td><td>4130</td><td>2260</td><td>1698</td><td>736</td><td>2817</td><td>1861</td></tr><tr><td>Movemts.</td><td>0.18</td><td>0.53</td><td>4.35</td><td>5.6</td><td>1.18</td><td>1.67</td></tr></tbody></table>

In the virtual condition, products were moved significantly more often than in the Dual Reality setup ($M_{virtual}$=4.35, $M_{DR}$=1.18, t=2.26, p<0.05). The DR condition again had significantly more movements than the real condition ($M_{real}$=0.18, t=2.2, p<0.05). We observed a similar result for the number of placements for each condition, where the participants positioned products at a new location more frequently in the virtual condition than in the DR setup ($M_{virtual}$=21.65, $M_{DR}$=14.88, t=3.12, p<0.05) and the real condition ($M_{real}$=13.59, t=3.64, p<0.005). The number of placements between the real and Dual Reality conditions remains insignificant. For the placement times, the real condition was slowest, followed by the DR system ($M_{real}$=4130, $M_{DR}$=2816, t=3.29, p<0.005), which itself was significantly slower than the virtual condition ($M_{virtual}$=1898, t=2.85, p<0.05).

# 6 Discussion

Hypothesis *H* 1 is rejected, as it implies that the performance is lower for the virtual than the real condition. Contrary to this hypothesis, and although the assistance systems are the same in both conditions, the overall profit in the virtual world condition was significantly higher (p<0.005). It seems like an “instrumented” system, e.g. a system that is enriched with assistance functions, works better in a virtual environment. There might be two possible causes of that effect: First, people are used to getting additional information in a virtual system, and are therefore more accustomed to using it. Real environments usually have no assistance functions. Second, we observed a significantly higher amount of interaction within the virtual system, and a smaller workload, as predicted by *H* 4 and confirmed by our results. The virtual system, which requires less body movement in order to complete tasks, seems to enable users to explore the solution space more than the real environment, which could be a cause of the increased performance. Although the workload is highest for the real condition, subjects stated that this condition made it easiest to solve the task.

Our efficiency measure could only partly confirm *H* 2, given for the example cost setting of our main experiment: We assigned the expert a reference cost of $C_{hour}$(expert)=100€. For the other entities we set cheaper costs of $C_{hour}$(assistant)=50€ and $C_{hour}$(robot)=1€. Within this example setting, the real condition was not as effi-
cient as the virtual one. Dual Reality outperformed them both. If we change these
entity costs, the results can also change significantly: If we reduce the expert cost by a
half, so it is equal to the assistant cost, the real interface becomes significantly more
efficient than the virtual condition (M<sub>real</sub>= 1.33, M<sub>virtual</sub>= 0.93, SD=0.47, t=3.56,
p<0.005). Dual Reality again outperforms them both. Every interface has its own
drawbacks: In the real environment, the user needs extra time for the task, as he has to
carry and handle the products, whereas the design on a desktop screen requires an
additional person to fill in the shelf afterwards. The Dual Reality setup allows a virtu-
al or a real-world design, but always has to synchronize with the real world using a
machine entity. Our formula makes it possible to find an equilibrium, based on entity
cost, at which the efficiency of two interfaces is equal. A designer can then investi-
gate which changes in the cost of a specific entity give one interface an advantage
over the other. To do so, two simple steps are required: First we need to equate the
efficiency formulas of both conditions. In the second step, we resolve the equation to
the specific entity cost that we want to modify to achieve our equilibrium. This gives
us the entity cost at which both conditions have an equal efficiency. As an example, if
we want to find the equilibrium for the virtual and DR conditions, using the cost of
the robot entity as a modifier, we set Efficiency<sub>V</sub>=Efficiency<sub>DR</sub> and resolve the equa-
tion by C<sub>hour</sub>(robot). We then get C<sub>hour</sub>(robot)= 111.114€ as an equilibrium value. In
this example, reducing the robot cost gives the DR condition an advantage, whereas
increasing the cost raises the efficiency of the virtual condition.

H 3 has been confirmed: the number of interactions was least for the real and greatest for the virtual condition. This effect can also be based on the workload of the different conditions, which is highest for the real condition, lower for the Dual Reality condition and lowest for the virtual condition. Most of the subjects switched between the virtual and real environment in our Dual Reality condition: About half of the subjects (7 of 17) did such a context switch within their physical actions (“hard” context switch), while nearly every subject (15 of 17) changed at least their eye focus between the two environments. This leads to the assumption that the concept of Dual Reality and the design of our DR condition works, as people took advantage of the possibilities such a system offers. H 5 and H 6, claiming that people switch between the environments within their actions as well as within their visual attention, have also been confirmed. Interestingly, the participants always used the same problem-solving strategy in all three conditions: First they placed high-priced products, then medium- and low-priced products, until the score did not increase anymore, although this mostly does not yield optimal results.

7 Design principles

Based on the results of the user study and the cost/efficiency function that we pro-
pose, we give guidelines on which interface type fits best for given preconditions. The
guidelines presented here mainly apply to our experimental setting. Whether or not
the rules can hold for other complex Dual Reality setups has to be confirmed in future
research. We designed our generic efficiency formula so that it can be translated to an
arbitrary composed task.

Regarding the performance of the task, as we observed in the pilot study, a real in-
terface works best when the user has to interact without additional information or
support functionality. The average user is most accustomed to the real world and can
therefore achieve the best results under those circumstances. That changes if an addi-
tional instrumentation is done, like highlighting parts of the scene, or adding a capa-
bility to view the current score. Virtual environments will outperform their real coun-
terparts in that case, although the number of interactions is likely to be higher within
that environment.

If the focus is on the *efficiency* of the interface, e.g. the cost that is incurred in relation to the result, it depends on the actual cost for each entity. With the settings from our experiment, DR mostly outperformed the other two conditions. But as we have shown in the discussion, a general answer is not possible. The efficiency formula has to be modeled first according to the task, in order to compute the most efficient solution. Using our efficiency formula, it is possible to calculate the point from which one or the other interface type should be preferred, in relation to the cost of the entities involved.

Apart from these analyses, another important aspect is the setup effort of a Dual
Reality environment. Whereas the effort is rather low for a real or virtual setup, a
Dual Reality setup involves a significantly higher effort, especially for the realization
of the synchronization of the real and virtual environment by machines. Our experi-
ment has provided evidence that most people actually use the advantages of the DR
environment and can achieve better results with it. Therefore, a Dual Reality setup
should always be considered, in addition to a conventional real or virtual setting.

8 Conclusion

More and more tasks are being virtualized, such that users can perform them in the
virtual world instead of the real world. Several studies have researched differences
between real and virtual environments using rather simple tasks. Our experiment in-
vestigated a setup with a complex task, involving complex actions with a significant
amount of locomotion, in real, virtual, and Dual Reality settings. We proposed an
efficiency function, and compared the performance, efficiency, and number and tim-
ing of sub-tasks within the three conditions. In contrast to the hypothesis and our pilot
study, the performance in maximizing the profit was significantly better in the virtual
version, although the efficiency is best for the Dual Reality setting. We explored the
impact of entity cost on the efficiency, and identified guidelines which guide the de-
veloper in selecting the right interface for his purpose, depending on the task that has
to be fulfilled as well as the subtasks and entities involved.

9 References

1. Transfer of training from virtual reality. *Ergonomics* 36, 7 (1993), 777–784.
2. Antle, A. N., Droumeva, M., and Ha, D. Hands on what?: Comparing children's mouse-based and tangible-based interaction. In *Proceedings of the 8th International Conference on Interaction Design and Children*, IDC '09, ACM (2009), 80–88.

3. Antle, A. N., and Wang, S. Comparing motor-cognitive strategies for spatial problem solving with tangible and multi-touch interfaces. In *Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction*, TEI '13, ACM (2013), 65–72.

4. Back, M., Kimber, D., Rieffel, E., Dunnigan, A., Liew, B., Gattepally, S., Foote, J., Shingu, J., and Vaughan, J. The virtual chocolate factory: Mixed reality industrial collaboration and control. In *Proceedings of the International Conference on Multimedia*, MM '10, ACM (2010), 1505–1506.

5. Corstjens, M., and Doyle, P. A model for optimizing retail space allocations. *Management Science* 27, 7 (1981), 822–833.

6. Davies, M., and Callaghan, V. iWorlds: Generating artificial control systems for simulated humans using virtual worlds and intelligent environments. *JAISE* 4, 1 (2012), 5–27.

7. Dobbins, A. C., and Grossmann, J. K. Asymmetries in perception of 3D orientation. *PLoS ONE* 5, 3 (03 2010), e9553.

8. Drèze, X., Hoch, S. J., and Purk, M. E. Shelf management and space elasticity. *Journal of Retailing* 70, 4 (1994), 301–326.

9. Gelernter, D. Mirror worlds: or the day software puts the universe in a shoebox... how it will happen and what it will mean. *Oxford University Press*, 1992.

10. Hwang, H., Choi, B., and Lee, M.-J. A model for shelf space allocation and inventory control considering location and inventory level effects on demand. *International Journal of Production Economics* 97, 2 (August 2005), 185–195.

11. Khan, V.-J., Nuijten, K., and Deslé, N. Pervasive application evaluation within virtual environments. In *PECCS*, C. Benavente-Peces and J. Filipe, Eds., SciTePress (2011), 261–264.

12. Lifton, J., and Paradiso, J. A. Dual reality: Merging the real and virtual. In FaVE, F. Lehmann-Grube and J. Sablatnig, Eds., vol. 33 of *Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering*, Springer (2009), 12–28.

13. Lucchi, A., Jermann, P., Zufferey, G., and Dillenbourg, P. An empirical evaluation of touch and tangible interfaces for tabletop displays. In *Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction*, TEI '10, ACM (2010), 177–184.

14. Montello, D. Scale and multiple psychologies of space. In *Spatial Information Theory: A Theoretical Basis for GIS*, A. Frank and I. Campari, Eds., vol. 716 of Lecture Notes in Computer Science. Springer, 1993, 312–321.

15. Murray, C. C., Talukdar, D., and Gosavi, A. Joint optimization of product price, display orientation and shelf-space allocation in retail category management. *Journal of Retailing* 86, 2 (2010), 125–136.

16. Olwal, A., Gustafsson, J., and Lindfors, C. Spatial augmented reality on industrial CNC-machines, 2008.

17. Todd, J., and Norman, J. The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure? *Perception and Psychophysics* 65, 1 (2003), 31–47.

18. Tuddenham, P., Kirk, D., and Izadi, S. Graspables revisited: Multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, CHI '10, ACM (2010), 2223–2232.