# Experiencing virtual geographic environment in urban 3D participatory e-planning: A user perspective

Thibaud Chassin, Jens Ingensand, Sidonie Christophe, Guillaume Touya

► To cite this version:

Thibaud Chassin, Jens Ingensand, Sidonie Christophe, Guillaume Touya. Experiencing virtual geographic environment in urban 3D participatory e-planning: A user perspective. Landscape and Urban Planning, 2022, 224, pp.104432. 10.1016/j.landurbplan.2022.104432. hal-03650229

HAL Id: hal-03650229

https://hal.science/hal-03650229

Submitted on 24 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# Experiencing virtual geographic environment in urban 3D participatory e-planning: A user perspective

Thibaud Chassin<sup>a, b, *</sup>, Jens Ingensand<sup>b</sup>, Sidonie Christophe<sup>c</sup>, Guillaume Touya<sup>c</sup>

## HIGHLIGHTS

* Fluctuation in experiencing VGE is due to conceptual, cognitive, and portrayal gaps.

* The VGE representation impacts mostly the time needed by users to complete a task.

* Spatial abilities strongly affect users' performance while manipulating a VGE.

* Current participatory setups using 3D appear to hinder the potential of VGE.

* The online format adopted in this study was valuable to evaluate users' performance.

## ARTICLE INFO

### Keywords:

Online user study

Virtual Geographic Environment (VGE)

Urban participatory e-planning

3D

User cognition

## ABSTRACT

The adoption of technology in urban participatory planning with tools such as Virtual Geographic Environments (VGE) promises a broader engagement of urban dwellers, which should ultimately lead to the creation of better cities. However, the authorities and urban experts show hesitancy in endorsing these tools in their practices. Indeed, several parameters must be wisely considered in the design of VGE; if misjudged, their impact could be damaging for the participatory approach and the related urban project. The objective of this study is to engage participants (N = 107) with common tasks conducted in participatory sessions, in order to evaluate the users' performance when manipulating a VGE. We aimed at assessing three crucial parameters: (1) the VGE representation, (2) the participants' idiosyncrasies, and (3) the nature of the VGE format. The results demonstrate that the parameters did not affect the same aspect of users' performance in terms of time, inputs, and correctness. The VGE representation impacts only the time needed to fulfill a task. The participants' idiosyncrasies, namely age, gender and frequency of 3D use also induce an alteration in time, but spatial abilities seem to impact all characteristics of users' performance, including correctness. Lastly, the nature of the VGE format significantly alters the time and correctness of users interactions. The results of this study highlight concerns about the inadequacies of the current VGE practices in participatory sessions. Moreover, we suggest guidelines to improve the design of VGE, which could enhance urban participatory planning processes, in order to create better cities.

## 1. Introduction

The role of urban dwellers has been acknowledged as crucial in urban planning practices for decades (Arnstein, 1969). Citizens living in their neighborhoods are cultivating a unique local knowledge driven by their experiences, emotions, or, in other words, their everyday life (Corburn, 2003). In the perspective of local urban governance, this

intimate expertise is valuable for non-omniscient authorities who have an administrative conception of the same territory. The involvement of dwellers in urban participatory planning can mobilize their local expertise, and thus, leverage several issues in implementing urban projects. An active participation can indeed help to reduce oppositions via the reinforcement of the social acceptability of the project (Wüstenhagen et al., 2007), or help to cultivate a reliable project intelligence

<footnote><sup>a</sup> Laboratory of Geographic Information Systems (LaSIG), EPFL, Lausanne, Switzerland</footnote>

<footnote><sup>b</sup> InSIT, University of Applied Sciences and Arts Western Switzerland (HEIG-VD), Yverdon-Les-Bains, Switzerland</footnote>

<footnote><sup>c</sup> LASTIG, Univ Gustave Eiffel, ENSG, IGN, F-94160 Saint-Mandé, France</footnote>

<footnote>* Corresponding author at: thibaud.chassin@epfl.ch</footnote>

<footnote>E-mail addresses: thibaud.chassin@epfl.ch (T. Chassin), jens.ingensand@heig-vd.ch (J. Ingensand), Sidonie.Christophe@ign.fr (S. Christophe), Guillaume.Touya@ign.fr (G. Touya).</footnote>
among citizens that could lead to the appropriation of the project by the population (Joerin et al., 2009). In this study, active participation is defined as a government-initiated process that engages the population following the *spectrum of public participation* described by the International Association for Public Participation (IAP2, 2018). This spectrum describes five levels: inform, consult, involve, collaborate, empower. This time-bound involvement can be conducted in-person or online, and intends to complete specific tasks related to the spectrum, such as providing feedback on certain aspects of the urban project, contributing to the project by suggesting alternatives, etc.

However, engaging the population in participatory sessions is not trivial. Each selected medium used to reach the population has its benefits and drawbacks, which leads to mobilizing only a limited part of the potential participants (Chassin et al., 2021). The inclusivity of a participatory session, which means reaching a targeted population, cannot, therefore, be accomplished by adopting only one medium, but by combining different participatory tools that are selected according to the preferences of the individuals constituting the targeted population. The meaning of inclusivity for a participatory session and its medium differs according to its context of application (e.g., Eastern vs. Western countries) (Zhang et al., 2019); hence, numerous factors should also be considered to reach inclusivity such as cultural, political, economical, contextual or societal factors.

New participatory mediums are developed to bridge these hinders and Information and Communication Technologies (ICT) solutions are often favored (Babelon et al., 2021; Gün et al., 2020; Haklay et al., 2018; Smith & Martín, 2021). Although the use of such tools has yet to be consolidated in planning practices (Afzalan & Muller, 2018), citizens are increasingly using ICT in their daily lives. The ubiquitous use of smartphones and (social) web applications bring the population to cultivate their skills in using digital technology (Ertiö, 2015), i.e., improving their ICT literacy. These digital skills are frequently the focus of concern in the adoption of ICT technologies within participatory practices (van Deursen & van Dijk, 2011), besides other factors such as trust, perceived complexity, data security or past positive experiences (Leroux & Pupion, 2022).

One of the most striking examples of digital literacy development is the Virtual Geographic Environments (VGE) (Lin & Gong, 2001), which are the core of this study. Since 2005 and the birth of Google Earth, the consumption of VGE has been thriving. These 3D representations are currently adopted in a wide variety of sectors (Biljecki et al., 2015), urban planning being one of them. In urban participatory approaches, VGE are a valuable opportunity for the local authorities to better involve the population in planning processes. One of the most compelling benefits of 3D representations is the enhancement of the user's immersion within a hypothetical future. Indeed, these visualizations help users to understand complex aspects of the project (Newell et al., 2021; Smith et al., 2020), which ultimately leads to meaningful contributions from users. This understanding can even be extended to participants not familiar with the surroundings (Onitsuka et al., 2018).

Nevertheless, 3D representations are rarely used outside of exploration studies in urban participatory planning (Gill & Lange, 2015). Skepticism regarding the adoption of VGE in participatory practices could be explained by the complexity of their design and uncertainty around their publication. Several challenges in creating 3D geo-visualizations have yet to be overcome, such as the fusion and visualization of diversified data, or the consideration of the idiosyncrasies of individuals, such as age, gender, spatial abilities (Çöltekin et al., 2017). In addition, because these representations are materializing hypothetical futures, they have a powerful impact on the conduction of participatory sessions and on the population's perception of the urban project. If misused, the same 3D representations could lead to unexpected outcomes (Jacquinod & Bonaccorsi, 2019). The layout of 3D geo-visualizations conveys a meaning that is interpreted by the viewer (Raaphorst et al., 2017). An inadequate use of its content could, for instance, drive the population to perceive a future reality that does not

conform to the built project (or the authorities' idea) (Downes & Lange, 2015), leaving the accountable authorities in a delicate position. The fluid comprehension of VGE, which could cause inadequate results, is articulated around three dimensions: a conceptual gap between the author and the viewers, a cognitive gap between the individuals themselves, and a portrayal gap encompassing the context in which VGE are broadcast, i.e., a 3D snapshot taken at a specific time, with objectives, within a setup and with definite features (Metze, 2020).

This study aims to investigate through a user experiment these three dimensions in order to facilitate the design and the use of VGE in urban participatory planning, and ultimately enhance this practice. This enhancement can be achieved by better describing the fluid comprehension of VGE in order to find solutions to reduce its extent. The next section will describe in more detail these dimensions while introducing research questions specific to each of them. Then, the experimental study will be introduced with its characteristics. The main findings will be illustrated in Section 4 through three metrics articulated around users' understanding, efficiency, and adaptability in the manipulation of distinct VGE. Last, the results will be discussed from a participatory perspective in order to provide guidelines for implementing VGE in urban practices.

## 2. Gaps driving to a fluid comprehension of VGE

### 2.1. *The conceptual gap: creator/viewers*

The creator of a VGE (or any visualizations) defines how to symbolize a snapshot of reality through the adopted visual representation (level of detail, textures, transparency, additional objects, etc.). This process introduces diverse choices, conscious or unconscious, guiding the interpretation of the VGE by the public (Jurík et al., 2016; Koláčný, 1969). In urban planning, the local authorities (or consultancy firms endorsed by the authorities) play this designing role. The guidance offered by the VGE, for instance, by facilitating the understanding of the scene via storytelling (Thöny et al., 2018), could drive the participants to produce relevant feedback or, in contrast, noisy contributions. One delicate choice in the design of VGE vividly discussed in the literature is the level-of-abstraction, which ranges from abstract-based (low level of detail, gray level surfaces) to veristic-based (high level of detail, photo-realistic textures). On the one hand, veristic representations facilitate self-projection and emotional bonding with the scene, enhancing the sense of place (Appleton & Lovett, 2003; Jaalama et al., 2021; Newell et al., 2021; Sheppard & Cizek, 2009). On the other hand, abstract representations are less cluttered, which reduces the cognitive load and focuses the attention of participants (Judge & Harrie, 2020; Santella & DeCarlo, 2004). Several aspects of the urban projects or the tasks to accomplish could lead the adopted representation to either hinder or enhance the participatory practices (Hayek, 2011; Boér, Çöltekin, & Clarke, 2013; Chassin et al., 2019). Hybrid representations are also explored to balance the limits introduced by the two level-of-abstraction alternatives (Brasebin et al., 2016; Lokka et al., 2018; Salter et al., 2009; Semmo et al., 2012). Hence, the local authorities often struggle to select the appropriate type of representation for the design of their approach.

Investigating how viewers interpret VGE based on their representation is crucial to close the conceptual gap induced by creating 3D geo-visualizations. Building this knowledge will ultimately lead to better decisions in the selection of the representation. Therefore, in this study, we are exploring how the representation of a VGE in terms of level of detail and texturing affects users when they perform tasks in an urban participatory session.

### 2.2. *The cognitive gap: Viewers/viewers*

The adoption of ICT tools within urban practices fosters participation by reaching a wider population (González et al., 2008; Kahila-Tani et al., 2019). VGE as an ICT tool also promotes broader participation.
However, the engagement of a multitude of participants also brings wide sociodemographic heterogeneity. Considering that understanding and experiencing VGE differ significantly between individuals (Stanney et al., 1998), this diversity could, in some cases, lead to negative outcomes for the participatory session. Each session has specific objectives and targeted participants, which are defined in advance by the authorities (or their representatives). A medium or representation that is not adequate for certain individuals could lead to irrelevant feedback (i.e., not in line with the objectives of the session), crystallizing opposition (because of misunderstandings), or excluding parts of the targeted individuals (because of a lack of abilities).

Many studies assessed the impact of sociodemographic characteristics on the performance of users related to VGE. Age has been proven to be a determining factor. Seniors are indeed less efficient in conducting cognitive tasks that are intensive (Kessels, Meulenbroek, Fernández, & Olde Rikkert, 2010). Therefore, compared to younger people, they often experience hardship while switching the perspective of the 3D scene (Lokka & Çöltekin, 2020), or while controlling allocentric interactive metaphors, which convey abundant information (Colombo et al., 2017). These two elements are frequently used in VGE. Other sociodemographic characteristics, such as spatial abilities, gender, expertise and education, have also been identified as significant factors in the reduction of 3D performance (Herman et al., 2018b; Wolbers & Hegarty, 2010).

Understanding and considering the population idiosyncrasy is central to urban participatory planning when adopting any participatory medium (including VGE). The consideration of sociodemographic characteristics could enhance the fairness and the inclusivity of the approach. Indeed, determining which characteristics of the population affect the interpretation of a VGE could help to design strategies that reduce the cognitive gap between individuals. Hence, this study aims at identifying which interactive aspects of VGE are altered by individual characteristics and how strongly.

## 2.3. The portrayal gap: Broadcast format

The publication of VGE in an urban participatory context compels the local authorities to choose a specific format, ranging from static photomontage to immersive Virtual Reality (VR), including others, such as video or web applications (see, Velarde et al., 2017; Virtanen et al., 2018; Chowdhury & Schnabel, 2019). These formats are not “created equal”, because they have different levels of realism or immersiveness (Çöltekin et al., 2016), which transforms the narratives or the methods of portraying the information included in VGE. The selection of a VGE version (i.e., its format) forces users to undergo a specific experience, which will ultimately affect their perceptions and behaviors. Several studies aim to better understand how users perform on particular tasks with different VGE formats, such as between static and interactive VGE (Herman et al., 2018a; Jurík et al., 2020), pseudo 3D (screen-based) and real 3D (stereoscopic-based) (Jurík et al., 2020), and desktop-based and VR-based (Dong et al., 2020). We observed various participatory sessions, and noted that common practices of 3D geovisualizations are: (1) static images (photomontage or architectural design) supported by 2D map documents, and (2) projected videos that participants watch (often once) at the beginning of a session before working with 2D maps. A few interactive VGE that were employed were manipulated by trained operators, leaving the participants as passive users. However, active interaction is a crucial aspect of VGE (Sheppard & Cizek, 2009; Smith et al., 2020; Thony et al., 2018).

The current practice of VGE in urban participatory planning does not seem to consider the benefits of active interaction in their design. We argue that this hindrance weakens the overall participatory approach, inhibiting the participants from sharing relevant opinions, insights or stories. With this study, we therefore analyze how users perform on participatory tasks within different contexts of interactivity.

# 3. Methods

To respect the safety restrictions on in-person gatherings introduced by the global COVID-19 pandemic (April 2020), we adopted an online setup to conduct this experimental study. The design of a digital study introduces many challenges that are less critical to address in an on-site setup, such as biases in user representativeness, data correctness, and dropout rate (Zhang, 2000). To limit the latter, we reduced the study to its essentials, and emphasized user-friendliness through simplicity of use, user engagement, and low entry costs. However, the core of the study comprised 20 min of arduous VGE-based tasks that could drive users to exhaustion and discouragement. To keep participants engaged with the study, we introduced gamified elements, such as a personal score, a reduced sociodemographic section (based on a fill-in-the-gaps paragraph written in the first person), and a badge system (personalized digital reward for the completion of the study).

The study was open to everyone and accessible online via a personal computer. We used several communication channels to reach a wide range of participants. Besides traditional social networks (e.g. Facebook, LinkedIn) activated via personal accounts or specific working groups, we adopted other means such as Reddit (a discussion website), Georezo (a French forum for Geographical Information Science professionals) and a newsletter about topographic sciences. The study took place during the 2020 academic break. We collected 107 completed responses from the participants.

## 3.1. Procedure

The study was only available on personal computers; smartphones were not compatible. This choice accommodated two desired conditions: development time and quiet environment. On the one hand, implementing a web app that is responsive for any kind of device can be time-consuming. On the other hand, the use of a smartphone induces frequent interruptions by notifications (Mehrotra et al., 2016), or divided attention (Wilmer et al., 2017), which are not suitable for the complexity of the tasks that were needed to be fulfilled. We assumed that the use of a personal computer encourages participants to complete the study in a calmer environment, seated at a desk, and hypothetically, with fewer interruptions. We framed the study procedure in three sections: welcome & training (landing page, terms of use, sociodemographic, tutorial), survey tasks, and feedback & results (comments, score page, acknowledgment, personal feedback). The tutorial aimed to outline the tasks that users had to perform throughout the study. Following the tutorial, participants were invited to start their tasks; the process is depicted in Fig. 1. Users were confronted with six batches, and each batch was composed of a set of tasks for a specific representation and metaphor of the VGE (see next section for the details). To complete the study, participants had to achieve 18 tasks in total. At the end, the participant could provide feedback about the study before reaching the last page, where the results were shown, along with some acknowledgments.

## 3.2. Descriptions of the batches

Several VGE were designed for this study, and each VGE illustrated a reduced city model of 16–19 buildings (the technical details are described in the Supplementary materials section). Various batches were implemented to address the research questions, four representations were selected in order to assess their impacts on the participants’ performance. In addition, three metaphors were implemented to simulate different participatory session setups. Lastly, four tasks were designed, which mimicked participatory activities or concerns that we observed in Switzerland. The batches were designed to evaluate the performance of the participants while discovering a 3D scene. From the perspective of urban participatory e-planning, we focussed on the understanding of the VGE that users have, according to the different tested parameters. Before
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Graphic procedure of the experimental study. The batch order was random for each participant. The number of times a batch is shown in the first, second, third, or fourth position is depicted by the table present within the Figure.</figcaption></figure>

engaging a participant to complete participatory tasks (such as giving an opinion or proposing alternative planning solutions), which employ a VGE, it is crucial for the authorities to ensure that users understand accurately the project depicted within the 3D scene. Without this comprehension, the relevance of these participants’ inputs can be questioned. Therefore, the assessment of the user performance investigates these first understandings.

### 3.2.1. VGE representations

Heterogeneous representations applied to the VGE were produced to

investigate their impact on the performance of participants. A representation is defined as a combination of style elements (the textures of 3D objects) and Level of Detail (LoD). We selected two abstract LoD to shape the buildings: LoD 1 and LoD 2. The LoD 1 is created by extruding the facades of a building from its footprint, and the LoD 2 adds the shape of the roof to the LoD 1 model (Biljecki, Ledoux, & Stoter, 2016). These buildings were textured with three different methods: (1) *color*, where the dominant color of the facade and the roof was applied to the surface of the 3D model; (2) *texture*, where a photorealistic texture covered the raw building; and (3) *focus*, where one of the buildings was modeled

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** VGE representations used in this experiment.</figcaption></figure>
with remarkable colors (i.e., white for the facades and red for the roof). From these two aspects we designed four representations: *LoD 2 + color* (Col2), *LoD 1 + color* (Col1), *LoD 2 + textures* (Tex2), and *LoD 2 + focus* (Foc2) (Fig. 2). The representation *LoD 2 + color* was the control; and the other representations were created by adjusting one aspect of the control. From the three resulting representations, we aspired to evaluate the impact of three features on user perception: (1) a decrease in the scenes' details via the reduction of the buildings' LoD; (2) the addition of visual complexity with the photorealistic texture; and (3) the integration of visual cues via the remarkable building colors, with these cues aimed at highlighting the concerned urban project within its surroundings. To limit the duration of the study, we restricted the use of these four representations only to specific tasks. Fig. 1 shows the order of their display.

### 3.2.2. Metaphors

Three metaphors were implemented in order to simulate the practice of VGE in urban participatory planning to assess their impact on users' performance. The first metaphor (batches 1 to 4, Fig. 1) aims at simulating the use of an interactive VGE that can be manipulated by the participants at any time during a participatory session. The second metaphor (named *memory*, batch 5, Fig. 1) intends to reproduce the broadcast of a video only seen once by the participants at the beginning of a session; this setup was translated by having the VGE and the pre-selected answers not visible simultaneously. The third metaphor (named *static*, batch 6, Fig. 1) mimics the use of static images, such as photomontages, or architectural designs to foster the discussion between participants in a participatory context. Therefore, no interaction was possible with the VGE. The two last metaphors were introduced to evaluate the differences between interactive VGE and typical urban participatory processes.

The interaction with the VGE was possible around a two-degrees-of-freedom rotation (roll with ↑↓ and yaw with ←→). These basic controls aspired to enhance the engagement of the participants with the VGE by making users active in the interaction, without adding extensive complexity for the lay users. The manipulation of the VGE followed some rules to accommodate the objectives of the study: (1) the entire 3D scene was always visible during rotations; (2) zoom and pan control were not permitted; and (3) participants could pivot around the scene (yaw rotation) at their discretion, but, the roll was restricted between 0 (horizontal view) and $\pi/4$ (bird's-eye view). This last rule aimed to restrain the participants to see the model from below, or through a vertical view. A vertical view could be relevant in a participatory context, for instance, by helping the orientation of the participants within the VGE. However, due to its similarities with the 2D map, this point of view was antagonistic with some tasks designed in the study.

### 3.2.3. Tasks

We designed the tasks considering two factors: (1) to be suitable for the analysis; and (2) to be realistic enough for a participatory approach perspective. Each task had a comparable layout: on the top was a question describing the task to complete; on the left was the VGE with which users could interact to resolve the task; and on the right were four pre-selected answers in the form of text, images, and clickable elements. The tasks' distribution within the batches is presented in Fig. 1. We designed four different tasks that were offered to the participants:

* **Guess layout** – participants were asked to identify the 2D footprint that matched the depicted VGE. This task was conducted to assess how participants were able to switch from a VGE to a 2D map. This activity is relevant for a participatory context, because 2D documents are often used in session with participants.

* **Guess lowest/highest** – participants had to determine the lowest/highest building of the VGE by selecting the correct building on the 2D map that they had previously chosen. This task aimed to evaluate how participants perceive the buildings' heights in an interactive VGE setup. With the progressive densification of the cities, the height

of the buildings is a major concern of the local population, and is frequently raised in urban participatory sessions (Ruming, 2018; Ruming et al., 2012). VGE are efficient for communicating about volumes, i.e., via a 3D scene, and the heights of the buildings are more straightforwardly perceived than with other supports such as 2D maps (Schroth et al., 2011; Sheppard & Cizek, 2009).

* **Guess viewpoint** – participants had to assess if two green spheres (embedded in the VGE) were visible or not from a viewpoint symbolized by a red sphere. This task is also relevant for a participatory context. Indeed, integrating a future project raises concerns by citizens about the visibility of the new project in the local landscape. A question often raised in a participatory session that uses a VGE is “Can I see the project from my window?”.

### 3.3. Participants' demographics

From the 17th of July to the 27th of October 2020, 107 participants (35% female) took part in the study. The age of the participants ranged from 17 to 67 years old (med = 33), with most of them holding a graduate-level degree or higher (83.2%). Their consumption of 3D was equally distributed. We estimated their spatial abilities and perceptions via a psychological test, namely the Water-Level-Problem (Piaget & Inhelder, 1956), which is an elementary exercise widely used for this purpose in the scientific literature (Linn & Petersen, 1985; Voyer et al., 1995). Its goal is to ask a participant to draw the water level of a glass that has been tilted; any incorrect answers suggest poor spatial abilities. Table 1 summarizes the sociodemographic characteristics of the participants. In the feedback survey, the participants evaluated the overall study positively, with an average mark of 4.01 out of 5 (2 individuals had no opinion). Concerning the difficulty, the opinion was divided between an average and relatively hard study, with a mark of 4.25 out of 7 (2 individuals had no opinion).

### 3.4. Data collection and analysis methods

During the experiment, data on user performance was collected alongside sociodemographic characteristics of the participants. We evaluated the performance via three variables: (1) the correctness of the answers (count of accurate answers), (2) the inputs count (number of times a user pressed a control, i.e., the count of camera positions), (3) and the time spent to achieve a specific task (from a new task display to

<table><caption>**Table 1** Sociodemographic characteristics of the study's participants.</caption><thead><tr><th>Sociodemographic characteristics</th><th>Mean (sd)</th><th>N (%)</th></tr></thead><tbody><tr><td><strong>Gender</strong></td><td></td><td></td></tr><tr><td>Male</td><td></td><td>69 (64.5)</td></tr><tr><td>Female</td><td></td><td>38 (35.5)</td></tr><tr><td><strong>Age</strong></td><td>37.7 (12.3)min = 17; max = 67</td><td></td></tr><tr><td><strong>Education</strong></td><td></td><td></td></tr><tr><td>Master or higher</td><td></td><td>64 (59.8)</td></tr><tr><td>Bachelor</td><td></td><td>25 (23.4)</td></tr><tr><td>Undergraduate *</td><td></td><td>4 (3.7)</td></tr><tr><td>Other</td><td></td><td>13 (12.2)</td></tr><tr><td>No Answer *</td><td></td><td>(0.9)</td></tr><tr><td><strong>3D Consumption</strong></td><td></td><td></td></tr><tr><td>Daily</td><td></td><td>27 (25.2)</td></tr><tr><td>Weekly</td><td></td><td>31 (29)</td></tr><tr><td>Monthly</td><td></td><td>29 (27.1)</td></tr><tr><td>Yearly or less</td><td></td><td>20 (18.7)</td></tr><tr><td><strong>Water-Level-Problem</strong></td><td></td><td></td></tr><tr><td>Pass</td><td></td><td>89</td></tr><tr><td>Fail</td><td></td><td>(83.2)</td></tr><tr><td></td><td></td><td>18 (16.8)</td></tr></tbody></table>

<footnote>* categories not weighed in the statistical analysis.</footnote>
the validation of the answer). Also, further information on user interaction was gathered to better describe how the scene was manipulated: camera positions, duration of the inputs, and rotation orientation (Chassin et al., 2021). From these variables, we assumed that hardship in manipulating the VGE will be translated by a drop in correctness and an increase in both the inputs count and the time spent on a specific task.

The statistical analysis was executed with Python. Based on the results, non-parametric tests were used. Table 2 describes the analysis that was undertaken. For categorical data, three steps were conducted: (1) overall differences analysis with a Kruskal-Wallis test (KW<sub>H</sub>); (2) if significant, one-by-one comparison with a Wilcoxon rank-sum test (W<sub>U</sub>); and (3) false discovery rate correction via a Benjamin-Hochberg procedure (p<sub>BH</sub>). For continuous ordered data, a Kendall rank correlation coefficient (K<sub>r</sub>) was calculated to assess their relation. The outputs of all the statistical tests that were conducted in this study are depicted in the Supplementary Materials section.

# 4. Results

## 4.1. Conceptual gap: Batches comparison

The overall results related to the comparison of the different batches (representations, metaphors) are shown in Fig. 3, in terms of input count, task completion time, and correctness score. For each batch, only the two first tasks, i.e., *Guess layout* and *Guess highest*, were considered. The tasks *Guess viewpoint* and *Guess lowest*, being specific to only four or two batches, respectively, have not been assessed for the overall results. The sixth batch does not appear on the input count graph, because participants could not interact with the static images.

### 4.1.1. Impact of the VGE representations on user performance

To appreciate which aspects of the VGE representations impact the participants' performance, we analyzed the inputs, time, and correctness. These three parameters were recorded during each of the tasks for the four different representations (described in Section 3.2.1.). The results are shown in Fig. 3.

For the count of inputs, we did not identify any significant difference between the representations (KW<sub>H</sub> = 2.776, p = 0.427). Thus, we did not carry out additional tests.

For the completion time, we noticed a meaningful discrepancy between the representations (KW<sub>H</sub> = 23.064, p < 0.001), therefore we can reject the null hypothesis, i.e., the participants performed differently in at least one representation. To identify which representation took more time to complete, we applied a Wilcoxon Rank Sums test. We did not establish divergence between the pairs **Col1** vs. **Foc2** (W<sub>U</sub> = 0.935, p<sub>BH</sub> = 0.404), and **Tex2** vs. **Col2** (W<sub>U</sub> = 0.544, p<sub>BH</sub> = 0.587) which implies that the participants required analogous time to complete their tasks on these representations. For the remaining pairs, we observed significant differences: **Col1** vs. **Tex2** (W<sub>U</sub> = -3.181, p<sub>BH</sub> < 0.01), **Col1** vs. **Col2**

<caption>**Table 2**</caption>

Statistical tests applied to the data according to its type. The parameters of this study linked to the data type are shown in the right column.

<table><thead><tr><th>Data type</th><th>Tests used</th><th>Parameter evaluated</th></tr></thead><tbody><tr><td>Categorical data (cat > 2)</td><td>(1) Kruskal-Wallis for overall significance<br>(2) Wilcoxon rank-sum test for paired significance<br>(3) P-values false discovery rate correction via Benjamin-Hochberg procedure</td><td>Batches</td></tr><tr><td>Paired data (2 categories comparison)</td><td>(1) Wilcoxon rank-sum test</td><td>Gender, Spatial Cognition Test</td></tr><tr><td>Continuous ordered data</td><td>(1) Kendall rank correlation coefficient</td><td>Education, Age, 3D consumption</td></tr></tbody></table>

## A. Participants' Inputs count

<figure><img src="image_5.png" /></figure>

## B. Participants' completion time

<figure><img src="image_6.png" /></figure>

## C. Participants' correctness score

<figure><img src="image_7.png" /></figure>

**Fig. 3.** Comparison of the performances of the participants interacting with different VGE representations (or batches) according to three aspects: count of the inputs (A), the time of completion (B) and answer correctness (C). This figure shows the aggregated values of users' performance for the tasks *Guess Layout* and *Guess Highest*. The p-value of each computed pair of VGE representations is shown on the right side of each figure.

(W<sub>U</sub> = -2.440, p<sub>BH</sub> < 0.05), **Tex2** vs. **Foc2** (W<sub>U</sub> = 4.095, p<sub>BH</sub> < 0.001), and **Foc2** vs. **Col2** (W<sub>U</sub> = -3.490, p<sub>BH</sub> < 0.001). From these results, we can claim that participants were quicker, on average, to complete their tasks with the representations **Col1** and **Foc2**. The two other representations, namely **Tex2** and **Col2**, led to longer completion time.

For the participants' correctness, we demonstrated with a Kruskal-Wallis test that at least one batch is distinct from the others, and the null hypothesis is indeed rejected (KW<sub>H</sub> = 14.789, p < 0.01). However, after one-to-one comparison, the Benjamin-Hochberg procedure (or false discovery rate) did not produce any significant differences.
### 4.1.2. Impact of the metaphors on user performance

To investigate which parameters of the participants' performance are affected by the participatory setup (portrayal gap), we added the batches 5 and 6, namely *memory* and *static* to the analysis. The results are depicted in Fig. 3, with batches *memory* and *static* that follow the first four batches.

For the count of inputs, we established a significant difference over the batches ($KW_H = 12.058, p < 0.017$). According to the Wilcoxon Rank Sums paired test, we observed that the metaphor *memory* does not differ from the batches (i.e., representations) **Col1** ($W_U = 2.418, p_{BH} = 0.052$) and **Foc2** ($W_U = 1.546, p_{BH} = 0.244$). However, a significant divergence was revealed with **Tex2** ($W_U = 2.798, p_{BH} < 0.05$) and **Col2** ($W_U = 3.092, p_{BH} < 0.05$). Therefore, this result indicates that the participants were manipulating the VGE less for executing their *memory* task than they did in the batches **Col2** and **Tex2**.

For the participants' completion time, we noticed that the addition of the two metaphors demonstrates a greater value for the Kruskal-Wallis test ($KW_H = 86.274, p < 0.001$). The Wilcoxon Rank sums test indicates a strong contrast between the two metaphors (*memory* and *static*) and the rest of the batches. Therefore, we can presume that the participants spent more time on the tasks associated with the batches, *memory* and *static*, than on the first four batches. Furthermore, we did not notice a significant distinction within the pair *memory* vs. *static* ($W_U = -0.873, p_{BH} = 0.410$).

For the participants' correctness, we established that introducing the two metaphors led to a higher value for the Kruskal-Wallis test ($KW_H = 110.544, p < 0.001$). We observed sharp contrasts for all the pairs containing *memory* or *static* ($p_{BH} < 0.001$), with less pronounced differences only for the pair **Col1** vs. *static* ($W_U = 2.772, p_{BH} < 0.05$). Consequently, participants seem to have a better performance with the default interactive VGE, which was used in the first four batches, rather than with other metaphors. The pair *memory* vs. *static* did not show a significant difference.

## 4.2. Comparison of population sociodemographic characteristics

The second section of the results aims to present the variations introduced by the idiosyncrasies of the users in terms of performance (inputs, time, correctness). These results are depicted in Fig. 4 and Table 3. The significance of five sociodemographic parameters (gender, age, education, 3D consumption and Water-Level-Problem) has been evaluated. All the batches were aggregated, resulting in one value per criterion per individual.

### 4.2.1. Gender

The performance of the two gender groups is comparable for the criteria: input count and completion time. We applied a Wilcoxon rank sums test on each of these aspects; the analysis did not show a correlation for either aspects: input count ($W_U = 0.901, p = 0.367$), and time completion ($W_U = 1.435, p = 0.151$). Therefore, males and females analogously manipulated the VGE (independently of the batches or tasks) before submitting their results. However, the participants' correctness score manifests a notable difference between the two gender groups ($W_U = 2.916, p = 0.004$), where males appear to have a higher score than females.

### 4.2.2. Age

We considered the age parameter as a ranked continuous value. Therefore, we did not create age categories. Thus, we calculated a Kendall rank correlation coefficient to identify any correlation between the age of the participants and their performance. We noticed a positive correlation for the input count ($K_t = 0.143, p < 0.05$) and the completion time ($K_t = 0.194, p = 0.005$). This correlation demonstrates, therefore, an influence of age on the manipulation of the VGE: the elderly appear to manipulate the VGE, in terms of input and time, more than the youth. We did not observe this correlation for the participant's

score accuracy ($K_t = -0.108, p = 0.122$). Furthermore, we also observed a compelling negative correlation between correctness score and age for the female gender group, illustrated in Fig. 4(C), ($K_t = -0.384, p = 0.001$).

### 4.2.3. Other parameters

We also described the relation between other sociodemographic characteristics (namely education, 3D consumption and water-level-problem) and the participants' performance (see Table 3). We did not notice differences in users' performance according to the education level of the participants (analyzed through a Kendall rank correlation coefficient). The 3D consumption presents a significant positive correlation for the completion time ($K_t = -0.240, p < 0.01$). Therefore, the more acquainted participants are with 3D, the faster they complete their tasks. Lastly, we observed apparent differences between the participants according to the result of the Water-Level-Problem (pass vs. fail). Individuals lacking spatial abilities (fail group) performed significantly below the other group in completing their tasks for all the parameters, with higher input count ($p < 0.01$), longer completion time ($p < 0.05$) and lower correctness score ($p < 0.01$).

## 4.3. User preferences in interacting with VGE representation

During the feedback section of the online survey, the participants were asked to weigh their perceived complexity on the *Guess highest* task according to the four VGE representations. Two questions were asked: the users had to select the most complex representation first, and then the easiest one. This information is pertinent to the study, adding a subjective opinion on the hardship generated by VGE representations. The perceived hardship can be compared to the factual performances of users assessed in the previous sections. The results are depicted in Fig. 5 for 98 votes. A few participants voted for the same scene (being at the same time the easiest and the hardest), therefore, they were removed from the results. Most of the participants voted for the portrayal of **Foc2** (37.8%) as the easiest portrayal to perceive the building height. Regarding the hardest representation, the participants voted for **Tex2** (35.7%). The representation **Col1** is in the second position for both the easiest (27.5%) and hardest (23.5%). Surprisingly, we observed in these votes that eight participants answered incorrectly to the tasks related to the representation that they perceived as the easiest.

# 5. Discussion

## 5.1. VGE representations affect the users' performance time

We have compared different VGE representations (i.e., LoD and style) in order to assess their effect on users' performance. The data does not demonstrate a connection of this parameter to the input count and the correctness score. However, the analysis shows a decrease in completion time for the representations having a lower LoD or a remarkable element.

The representation **Foc2** has been selected by the participants as the easiest scene to determine the highest building. The distinct building within the VGE, highlighted by color saliency, was the solution of the task *Guess highest*. Salient features highlight specific elements in a bushy landscape (2D or 3D) (Wolfe & Horowitz, 2004). Therefore, these features could focus the users' gaze and their attention. The users could, thus, be guided through cluttered VGE, significantly reducing the time needed to complete a task, as pinpointed in this user study. In an urban participatory approach, this method could focus the participants on a specific thematic or component that should be addressed in a participatory session, as hypothesized in the literature (Chassin et al., 2019). However, despite catching users' attention, no conclusions can be drawn on keeping their attention engaged in a lengthy session.

The representation **Tex2** has been considered by the participants as the most difficult representation to determine the highest building of the
<figure><img src="image_8.png" /><figcaption>**A. Participants' inputs count**</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**B. Participants' completion time**</figcaption></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /><figcaption>**C. Participants' correctness score**</figcaption></figure>

<figure><img src="image_12.png" /></figure>

**Fig. 4.** Evaluation of the variation of performances according to users' idiosyncrasies. The variations are depicted according to the age of the participants (abscise) and separated by gender group: male in blue, on the left, and female in orange, on the right. For each diagram, a regression line with its confidence interval is plotted. We assessed three aspects: count of the inputs (A), the time of completion (B) and answer correctness (C). This figure shows the aggregated values for the six batches (and five for the inputs count). The p-value of each computed parameter is shown on the right of the figure (D). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
<caption>**Table 3**</caption>

Supplementary findings about sociodemographic characteristics, namely education, 3D consumption, and Water-Level-Problem test.

<table><thead><tr><th>Sociodemographic characteristics</th><th>Inputs</th><th>Times</th><th>Correctness</th></tr></thead><tbody><tr><td>Education</td><td>–</td><td>–</td><td>–</td></tr><tr><td>3D Consumption</td><td>–</td><td>***</td><td>–</td></tr><tr><td>Water-Level-Problem</td><td>***</td><td>*</td><td>***</td></tr></tbody><tfoot><tr><td colspan="4">–: p-value > 0.05; *: p-value < 0.05; **: p-value < 0.01</td></tr></tfoot></table>

<figure><img src="image_13.png" /><figcaption>**Fig. 5.** Results of the participants' vote on the hardest and easiest VGE representation while performing the task *Guess highest*.</figcaption></figure>

VGE. Realistic representations incorporate plenty of information, which increases the cognitive load conveyed by the 3D scene (Skulmowski & Rey, 2020). Cluttered information present on the texture (doors, windows, floors, paints, roof tiles, etc.) drives the users to intensify their attention, which could lead to poorer performance compared to more abstract representations (Liao, Dong, Peng, & Liu, 2017; Smallman & John, 2005). The results of our study are in line with the literature regarding the completion time aspect for the representations **Col1** and **Foc2**. However, we did not observe significant differences between **Tex2** and **Col2**, indicating a relatively small impact of visual cluttering on users' performance, which has already been noted in the literature (Lokka & Çöltekin, 2019). Therefore, the use of abstract or realistic representation in an urban participatory context should not be evaluated by its impact on users' performance, but according to the urban project dimensions, such as temporality, tasks, and uncertainties (Chassin et al., 2019; Hayek, 2011; Judge & Harrie, 2020).

The representation **Col1** was perceived both as helping and hindering in distinguishing the building heights. On the one hand, the positive feedback from the participants is illustrated by an improvement of users' performance (in terms of completion time) without loss of correctness. During a participatory session, participants could, therefore, efficiently focus on a specific task, avoiding spending time and energy on perceiving and understanding the specificities of the elements that are displayed within a VGE. On the other hand, this antagonism reported by the participants is also supported by the fact that eight users, who voted **Col1** to be the easiest, did not select the correct answer on the related task.

This representation indeed appears to be confusing by its apparent simplicity. The lack of differentiation between the buildings, introduced by the basic cubic shape, appears to decrease the legibility of the overall VGE (this poor contrast was unfortunately accentuated by similar colors).

## 5.2. Several sociodemographic characteristics affect users' performance

We have assessed the impact of five sociodemographic characteristics, namely age, gender, education, 3D consumption and Water-Level-Problem on users' performance based on 3D tasks. The results show a clear correlation between the correctness score of the participants and their gender or interpretation of the Water-Level-Problem. Further relations have also been established, such as input count vs. age & Water-Level-Problem, and completion time vs. age & 3D consumption & Water-Level-Problem.

Sociodemographic characteristics act as an essential feature of participatory approaches. Each session aspires indeed to reach a specific targeted population that could extensively fluctuate (e.g., gender, age, location, profession) according to the context of the session (i.e., addressed thematic). However, the tools that are adopted by the municipal officials can mobilize these targeted groups only to a certain degree (Chassin et al., 2021); this limited extent also affects urban 3D participatory e-planning solutions. A suited solution to limit this hindrance, and to promote more inclusivity of the participatory sessions, would be to diversify the participatory tools, which are adopted to reach the targeted public. However, implementing this approach can be complex depending on various factors: political, economic, cultural, societal, etc. Therefore, acknowledging and understanding the differences between participants is crucial in the design of participatory tools (e.g., VGE), which should be responsive to users' backgrounds, skills, and expectations. Based on the findings described in the previous section which demonstrate a partial impact of the VGE representation on user performance, we would recommend designing a custom representation that results from a dialogue between a user and a digital agent (Christophe, 2011), which can be embedded, if users' social-demographic characteristics are known in advance. A personalized representation will have a limited impact on the user performance but could enhance their motivation to be engaged with the VGE.

One vivid debate in the adoption of VGE in urban participatory planning is the mobilization of the elderly and the young (Bouzguenda et al., 2021). The results suggest that age is related to a loss of efficiency (input count and completion time), or at least to more exploration of the VGE. However, no loss of correctness score has been established for the age parameter, in the online context of the study that attracted participants who were certainly more tech-savvy than the broader population. Training individuals beforehand could balance this reduced efficiency, as suggested by the results for 3D consumption. Therefore, in a participatory session context, an optional tutorial could be implemented to introduce and facilitate the use of VGE tools for the individuals who desire to do so.

Nonetheless, one population segment presents a nonconformity. We observed a strong decrease in the correctness score according to age for the female demographic group. The number of individuals being relatively low, this assumption deserves particular attention and further investigation. This drop implies the need for participatory approaches to adapt the practice. The 18% of participants who failed on the Water-Level-Problem (thus lacking spatial abilities) reinforced this need to design flexible approaches. A diversification of the participatory tools (not only spatial-based) could support a better inclusion of these demographic groups. Nevertheless, two challenges arise: (1) how to reduce the spatial heaviness of mediums used in participatory sessions on issues that are profoundly inscribed in the territory; and (2) how to consider the local expertise of citizens having a different spatial perception.
## 5.3. Towards an enhancement of VGE practices in urban participatory planning

We investigated the practicability of typical participatory setups, namely video broadcasting and architectural static visuals, by implementing two metaphors (memory and static). Our analysis demonstrates a clear lack of performance of these methods compared to the use of an interactive VGE for tasks related to the evaluation of buildings' heights and the transition between 2D vs. 3D views.

3D geovisualizations are acknowledged to foster the sense of place, i. e., to establish a connection between the real world and the virtual representation of a place (Newell & Canessa, 2015). This sense of place is crucial in urban participatory planning, notably by allowing participants to relate to the virtual representation in order to discuss tangible urban issues. Furthermore, the interactivity promoted by VGE supports the understanding and representativeness of the scene that is portrayed (Sheppard & Cizek, 2009). The results of this study on the interactive setup, namely active interaction (interactive VGE), passive interaction (memory), and non-interaction (static) can be compared to previous results found in the literature.

First, the use of static images limits the amount of information that an interactive 3D geovisualization can convey. Indeed, the performance of the participants is reduced with the use of static images, as demonstrated with the significant differences in users' correctness score. Similar findings can be found in the literature (Herman et al., 2018a; Keehner et al., 2008; Kubíček et al., 2019; Schroth et al., 2011). However, surprisingly, we observed that interactivity appears to reduce the time needed for completing a task, which is not in line with other studies (Herman et al., 2018a; Jurík et al., 2020). This divergence could be linked to the task to complete, the type of data that is depicted in the VGE, or to the scale of the VGE. In our setup, VGE had a reduced spatial extent with a limited manipulation that could lead users to restrict their exploration.

Second, the use of video that is broadcast once at the beginning of a participatory session forces the participants to memorize and understand complex aspects regarding a future urban project. The subsequent contributions of participants are relevant only if they can accurately and instantly recall the information that they had watched. However, our results (in a memory setup that uses interactive VGE) demonstrate a drop in users' correctness score. This result is in line with the literature that established the existence of immediate inaccuracies in route navigation recall after seeing a video with an egocentric perspective; these inaccuracies tend to increase after a delay (Lokka et al., 2018). There is evidence to suggest that active interaction facilitates the encoding of (spatial) information compared to a passive setup (Chrustil & Warren, 2012). This encoding is essential for learning and also for developing comprehensive knowledge about a (spatial) situation. Passive setup can be improved by recording beforehand a set of optimized viewpoints that would be passively displayed during the exploration of the 3D models (Keehner et al., 2008).

The metaphors, memory and static, appear to be less suitable in a participatory context than an interactive VGE. The participants perform better with interactive VGE, which allows them to select their own point of view and not be restrained by vantage views carefully selected beforehand by the authorities. By restricting the free exploration of the VGE, the local authorities are empowered to share a carefully selected collection of perspectives, which aims only to show favorable aspects of the project in order to influence citizens' judgments.

## 5.4. Limits of the survey

This study was conducted online, and we had no control over the participants completing the experiment. The population that takes part in online surveys is often skewed toward male, tech-savvy, educated people (Bethlehem, 2010). Thus, we observed a preponderance of these sociodemographic characteristics within our study, which are not

representative of the broad population. This imbalance is notable for the elderly, who were mainly tech-savvy and therefore relatively comfortable in manipulating VGE. Our results should be considered accordingly.

Furthermore, buildings were the only city elements depicted within the VGE. Other common elements, such as trees, garden, street furniture, light poles, and vehicles were not modeled. This restriction limited the time needed for the development of the study, and reduced the size of the VGE. Indeed, the technical setup of the participants was not controlled. The performance of the users' computers could have produced delays in the display of the VGE. This situation was raised once in the comment section for the batch **Tex2**. Moreover, another participant declared having to restart the study several times because their computer was turning off during the survey.

The results of this research are based on an experimental study where the participants manipulated VGE for approximately 20 min. However, a typical in-person participatory session is spread over several hours. Therefore, our experimental setup cannot simulate this expanded time of engagement that could drive the participants to exhaustion, reducing their performances in manipulating VGE or tainting their judgment.

# 6. Conclusion

In this study, we developed a user experiment to investigate the performance of the users (n = 107) when interacting with a VGE. Tasks related to urban participatory sessions were implemented, and the results were evaluated through three gaps: (1) a conceptual gap portrayed by the design of multiple levels-of-abstraction for the VGE; (2) a cognition gap observed via the number of participants and their variation in terms of the sociodemographic characteristics; and (3) a portrayal gap considered over the simulation of three participatory setups using VGE.

The results demonstrated that a higher level-of-abstraction or salient elements significantly decreased the time needed to complete the tasks of the experiments. Time is also an important aspect influenced by sociodemographic characteristics and past experiences, namely age and consumption of 3D tools. Furthermore, we observed that participants who failed the Water-Level-Problem experienced hardship when interacting with VGE. This last result is crucial for the adoption of VGE in urban participatory practices because these users account for nearly 20% of the total of participants. The local authorities should, therefore, consider other mediums (or tools) to collect the contribution of these urban dwellers in order to, first, limit bias in the decision, and second, increase the inclusivity of the approach.

Moreover, this study showed that the current practices of 3D geovisualizations in urban participatory sessions are poorly adapted to the potential of VGE. Using videos or static images seems to limit the performance of the participants on tasks that are related to participatory planning. Therefore, this study highlights the need to enhance current practices with interactivity in order to improve the quality of the participatory approaches via better contributions and a better understanding of the urban projects. Ten years ago, E. Lange stated "We can visualise. Now what?" to emphasize that the technical obstacles in 3D were mainly lifted (Lange, 2011). Since then, many studies have contributed to the assessment of users' perception and behavior while facing VGE. This knowledge base provides several guidelines to design better VGE, more centered on users. It is now time to lift the skepticism of the local authorities in order to finally democratize the practice of interactive VGE in urban participatory planning. Therefore, we aim to develop a prototype of an online platform for urban 3D participatory e-planning, which would be augmented by the findings of this study and the current state of scientific knowledge in participatory planning. After its implementation, we aspire to consult the authorities in the adoption of this new generation of participatory tool.

The last contribution of this user study is the surprising success of the online format. The experiment was planned in-person, like most of the studies about user cognition. However, with the unpredictable context
of the COVID-19 pandemic, we were forced to think outside-the-box and to adopt a different method, similarly to others (Griffin et al., 2021). From a user study that was planned to take place for a couple of hours, we had to transform our procedure to suit the online format. Some aspects had to be deleted and others to be added, such as the gamification elements. This process was challenging, and the fact that participants could complete the study freely at any time, on their personal devices, surely added new and uncontrolled parameters that might be difficult to assess. Nonetheless, this flexibility allowed us to reach more than a hundred people during the summer months following the first wave of the pandemic. The resulting conclusions of this study are also based on solid data, which were only made possible by the online format. With this study and the availability of the in-house original files in open source, we contribute to the development of potential methods to address challenges in conducting online user studies.

## 7. URLs

Google Earth: [google.com/earth/](https://www.google.com/earth/); Facebook: [facebook.com](https://www.facebook.com); LinkedIn: [LinkedIn.com](https://www.linkedin.com); Reddit: [Reddit.com](https://www redd.it); GeoRezo: [georezo.net](https://www.georezo.net); OpenStreetMap: [openstreetmap.org](https://www.openstreetmap.org); Blender: [Blender.org](https://www.blender.org); (version 2.82); Three.js: [Threejs.org](https://www.threejs.org) (version 0.114.0); Python: [Python.org](https://www.python.org) (version 3.7.5).

### CRediT authorship contribution statement

**Thibaud Chassin:** Conceptualization, Methodology, Software, Formal analysis, Writing – original draft. **Jens Ingensand:** Conceptualization, Methodology, Supervision. **Sidonie Christophe:** Conceptualization, Methodology, Validation. **Guillaume Touya:** Conceptualization, Methodology, Validation, Writing – review & editing.

### Acknowledgements

This study was partly funded by the Computers & Geosciences Research Scholarships co-sponsored by Elsevier and the International Association for Mathematical Geosciences (IAMG). We would also like to thank the participants for their generous time and valuable feedback that made this project possible.

## Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.landurbplan.2022.104432.

## References

Afzalan, N., & Muller, B. (2018). Online Participatory Technologies: Opportunities and Challenges for Enriching Participatory Planning. *Journal of the American Planning Association*, **84**(2), 162–177. https://doi.org/10.1080/01944363.2018.1434010

Appleton, K., & Lovett, A. (2003). GIS-Based Visualisation of Rural Landscapes: Defining 'sufficient' Realism for Environmental Decision-Making. *Landscape and Urban Planning*, **65**(3), 117–131. https://doi.org/10.1016/S0169-2046(02)00245-1

Arnstein, S. R. (1969). A Ladder Of Citizen Participation. *Journal of the American Institute of Planners*, **35**, 216–224. https://doi.org/10.1080/01944366908977225

Babelon, I., Pánek, J., Falco, E., Kleinhans, R., & Charlton, J. (2021). Between Consultation and Collaboration: Self-Reported Objectives for 25 Web-Based Geoparticipation Projects in Urban Planning. *ISPRS International Journal of Geo-Information*. https://doi.org/10.3390/ijgi10110783

Bethlehem, J. (2010). Selection Bias in Web Surveys. *International Statistical Review* = *Revue Internationale de Statistique*, **78**(2), 161–188. https://doi.org/10.1111/j.1751-5823.2010.00112.x

Biljecki, F., Ledoux, H., & Stoter, J. (2016). An improved LOD specification for 3D building models. *Computers, Environment and Urban Systems*, **59**, 25–37. https://doi.org/10.1016/j.compervurbsys.2016.04.005

Biljecki, F., Stoter, J., Ledoux, H., Zlatanova, S., & Çöltekin, A. (2015). Applications of 3D City Models: State of the Art Review. *ISPRS International Journal of Geo-Information*. https://doi.org/10.3390/ijgi4042842

Boér, A., Çöltekin, A., & Clarke, K. C. (2013). An evaluation of web-based geovisualizations for different levels of abstraction and realism – what do users predict? *26th International Cartographic Conference, Dresden, Germany, 25 August 2013 - 30 August 2013*. https://doi.org/10.5167/uzh-87810

Bouzguenda, I., Fava, N., & Alalouch, C. (2021). Would 3D Digital Participatory Planning Improve Social Sustainability in Smart Cities? An Empirical Evaluation Study in Less-Advantaged Areas. *Journal of Urban Technology*, *April*, 1–31. https://doi.org/10.1080/10630732.2021.1900772

Brasebin, M., Christophe, S., Jacquinod, F., Vinesse, A., & Mahon, H. (2016). 3d Geovisualization & Stylization to Manage Comprehensive and Participative Local Urban Plans. *ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences*, *IV-2/W1*(October), 83–91. https://doi.org/10.5194/isprs-annals-IV-2-W1-83-2016

Chassin, T., Cherqui, A., Ingensand, J., & Joerin, F. (2021). Impact of Digital and Non-Digital Urban Participatory Approaches on Public Access Conditions: An Evaluation Framework. *ISPRS International Journal of Geo-Information*, **10**(8), 563. https://doi.org/10.3390/ijgi10080563

Chassin, T., Ingensand, J., Lotfian, M., Ertz, O., & Joerin, F. (2019). Challenges in Creating a 3D Participatory Platform for Urban Development. *Advances in Cartography and GIScience of the ICA, I*(July), 1–8. https://doi.org/10.5194/ica-adv-1-3-2019

Chassin, T., Ingensand, J., Touya, G., & Christophe, S. (2021). How do users interact with Virtual Geographic Environments? Users’ behavior evaluation in urban participatory planning. *Proceedings of the ICA, 4*. https://doi.org/10.5194/ica-proc-4-19-2021

Chowdhury, S., & Schnabel, M. A. (2019). Laypeople’s Collaborative Immersive Virtual Reality Design Discourse in Neighborhood Design. *Frontiers in Robotics and AI, 6* (October), 97. https://doi.org/10.3389/frobt.2019.00097

Chrastil, E. R., & Warren, W. H. (2012). Active and Passive Contributions to Spatial Learning. *Psychonomic Bulletin & Review*, **19**(1), 1–23. https://doi.org/10.3758/s13423-011-0182-x

Colombo, D., Serino, S., Tuena, C., Pedroli, E., Dakanalis, A., Cipresso, P., et al. (2017). Egocentric and Allocentric Spatial Reference Frames in Aging: A Systematic Review. *Neuroscience and Biobehavioral Reviews*, **80**(September), 605–621. https://doi.org/10.1016/j.neubiorev.2017.07.012

Çöltekin, A., Bleisch, S., Andrienko, G., & Dykes, J (2017). Persistent challenges in geovisualization – a community perspective. *International Journal of Cartography*, **3**, 115–139. https://doi.org/10.1080/23729334.2017.1302910

Çöltekin, A., Lokka, I. E., & Zahner, M. (2016). On The Usability and Usefulness Of 3d (Geo)visualizations– A Focus On Virtual Reality Environments. *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*. https://doi.org/10.5194/isprsarchives-xli-b2-387-2016

Corburn, J. (2003). Bringing Local Knowledge into Environmental Decision Making. *Journal of Planning Education and Research*, **22**(4), 420–433. https://doi.org/10.1177/0739456X03022004008

Christophe, S. (2011). Creative Colours Specification Based on Knowledge (COLorLEGend System). *The Cartographic Journal*, **48**(2), 138–145. https://doi.org/10.1177/1743277411Y.00000000012

van Deursen, A., & van Dijk, J. (2011). Internet Skills and the Digital Divide. *New Media & Society*, **13**(6), 893–911. https://doi.org/10.1177/146144810386774

Dong, W., Yang, T., Liao, H.a., & Meng, L. (2020). How Does Map Use Differ in Virtual Reality and Desktop-Based Environments? *International Journal of Digital Earth*, **13** (12), 1484–1503. https://doi.org/10.1080/17538947.2020.1731617

Downes, M., & Lange, E. (2015). What You See Is Not Always What You Get: A Qualitative, Comparative Analysis of Ex Ante Visualizations with Ex Post Photography of Landscape and Architectural Projects. *Landscape and Urban Planning*, **142**(October), 136–146. https://doi.org/10.1016/j.landurbplan.2014.06.002

Ertti, T.-P. (2015). Participatory Apps for Urban Planning—space for Improvement. *Planning Practice and Research*, **30**(3), 303–321. https://doi.org/10.1080/02697459.2015.1052942

Gill, L., & Lange, E. (2015). Getting Virtual 3D Landscapes out of the Lab. *Computers, Environment and Urban Systems*, **54**(November), 356–362. https://doi.org/10.1016/j.compervurbsys.2015.09.012

González, A., Gilmer, A., Foley, R., Sweeney, J., & Fry, J. (2008). Technology-Aided Participative Methods in Environmental Assessment: An International Perspective. *Computers, Environment and Urban Systems*, **32**(4), 303–316. https://doi.org/10.1016/j.compervurbsys.2008.02.001

Griffin, A., Kubíček, P., & Kettunen, P. (2021). Workshop on Adaptable Research Methods For Empirical Research with Map Users. *Edited by ICA Commission on Cognitive Issues in Geographic Information*. Visualization.

Gün, A., Demir, Y., & Pak, B. (2020). Urban Design Empowerment through ICT-Based Platforms in Europe. *International Journal of Urban Sciences*, **24**(2), 189–215. https://doi.org/10.1080/12265934.2019.1604250

Haklay, M., Jankowski, P., & Zwoliński, Z. (2018). Selected Modern Methods and Tools for Public Participation in Urban Planning – A Review. *Quæstiones Geographicae*, **37** (3), 127–149. https://doi.org/10.2478/quadeo-2018-0030

Hayek, U. W. (2011). Which Is the Appropriate 3D Visualization Type for Participatory Landscape Planning Workshops? A Portfolio of Their Effectiveness. *Environment and Planning B, Planning & Design*, **38**(5), 921–939. https://doi.org/10.1068/b36113

Herman, L., Jurík, V., Stachón, Z., Vrbík, D., Russnák, & Rezník, T. (2018a). Evaluation of User Performance in Interactive and Static 3D Maps. *ISPRS International Journal of Geo-Information*, **7**(11), 415. https://doi.org/10.3390/ijgi7110415

Herman, L., Rezník, T., Stachón, Z., & Russnák, J. (2018). The Design and Testing of 3DmoveR: An Experimental Tool for Usability Studies of Interactive 3D Maps. *Cartographic Perspectives*, **90**(September), 31–63. https://doi.org/10.14714/CP90.1411
IAP2. (2018). IAP2 Spectrum of Public Participation. https://cdn.ymaws.com/www.iap2.org/resource/resmgr/pillars/Spectrum_8.5x11 Printed.pdf.

Jaalama, K., Fagerholm, N., Julin, A., Virtanen, J.-P., Maksimainen, M., & Hyppä, H. (2021). Sense of Presence and Sense of Place in Perceiving a 3D Geovisualization for Communication in Urban Planning – Differences Introduced by Prior Familiarity with the Place. *Landscape and Urban Planning*, 207(103996). https://doi.org/10.1016/j.landurbplan.2020.103996

Jacquinod, F., & Bonaccorsi, J. (2019). Studying Social Uses of 3D Geovisualizations: Lessons Learned from Action-Research Projects in the Field of Flood Mitigation Planning. *ISPRS International Journal of GeomInformation*, 8. https://doi.org/10.3390/ijgi8020084

Joerin, F., Desthieux, G., Beuze, S. B., & Nembriani, A. (2009). Participatory Diagnosis in Urban Planning: Proposal for a Learning Process Based on Geographical Information. *Journal of Environmental Management*, 90(6), 2002–2011. https://doi.org/10.1016/j.jenvman.2007.08.024

Judge, S., & Harrie, L. (2020). Visualizing a Possible Future: Map Guidelines for a 3D Detailed Development Plan. *Journal of Geovisualization and Spatial Analysis*, 4(1). https://doi.org/10.1007/s41651-020-00049-4

Jurík, V., Herman, L., Kubíček, P., Stachon, Z., & Šašinka, C. (2016). Cognitive Aspects of Collaboration in 3d Virtual Environments. *ISPRS - International Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences, XLI-B2*(June), 663–670. https://doi.org/10.5194/isprsarchives-XLI-B2-663-2016

Jurík, V., Herman, L., Snopková, D., Galang, A. J., Stachon, Z., Chmelfík, J., et al. (2020). The 3D Hype: Evaluating the Potential of Real 3D Visualization in Geo-Related Applications. *PloS One*, 15(5), e0233353. https://doi.org/10.1371/journal.pone.0233353

Kahila-Tani, M., Kytta, M., & Geertman, S. (2019). Does Mapping Improve Public Participation? Exploring the Pros and Cons of Using Public Participation GIS in Urban Planning Practices. *Landscape and Urban Planning*, 186(June), 45–55. https://doi.org/10.1016/j.landurbplan.2019.02.019

Keehner, M., Hegarty, M., Cohen, C., Khooshabeh, P., & Montello, D. R. (2008). Spatial Reasoning with External Visualizations: What Matters Is What You See, Not Whether You Interact. *Cognitive Science*, 32(7), 1099–1132. https://doi.org/10.1080/03640210801898177

Kessels, R., Meulenbroek, O., Fernández, G., & Olde Rikkert, M. (2010). Spatial Working Memory in Aging and Mild Cognitive Impairment: Effects of Task Load and Contextual Cueing. *Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition*, 17(5), 556–574. https://doi.org/10.1080/13825585.2010.481354

Kolácný, A. (1969). Cartographic Information—a Fundamental Concept and Term in Modern Cartography. *The Cartographic Journal*, 6(1), 47–49. https://doi.org/10.1179/caj.1969.6.1.47

Kubíček, P., Šašinka, C., Stachon, Z., Herman, L., Jurík, V., Urbánek, T., et al. (2019). Identification of Altitude Profiles in 3D Geovisualizations: The Role of Interaction and Spatial Abilities. *International Journal of Digital Earth*, 12(2), 156–172. https://doi.org/10.1080/17538947.2017.1382581

Lange, E. (2011). 99 Volumes Later: We Can Visualise. Now What? *Landscape and Urban Planning*, 100(4), 403–406. https://doi.org/10.1016/j.landurbplan.2011.02.016

Leroux, E., & Pupion, P.-C. (2022). Smart Territories and IoT Adoption by Local Authorities: A Question of Trust, Efficiency, and Relationship with the Citizen-User-Taxpayer. *Technological Forecasting and Social Change*, 174(121195), 121195. https://doi.org/10.1016/j.techfore.2021.121195

Liao, H., Dong, W., Peng, C., & Liu, H. (2017). Exploring Differences of Visual Attention in Pedestrian Navigation When Using 2D Maps and 3D Geo-Browsers. *Cartography and Geographic Information Science*, 44(6), 474–490. https://doi.org/10.1080/15230406.2016.1174886

Lin, H., & Gong, J. (2001). Exploring Virtual Geographic Environments. *Geographic Information Sciences*, 7, 1–7. https://doi.org/10.1080/10824000109480550

Linn, M. C., & Petersen, A. C. (1985). Emergence and Characterization of Sex Differences in Spatial Ability: A Meta-Analysis. *Child Development*, 56(6), 1479. https://doi.org/10.2307/1130467

Lokka, I. E., & Çöltekin, A. (2019). Toward Optimizing the Design of Virtual Environments for Route Learning: Empirically Assessing the Effects of Changing Levels of Realism on Memory. *International Journal of Digital Earth*, 12(2), 137–155. https://doi.org/10.1080/17538947.2017.1349842

Lokka, I. E., & Çöltekin, A. (2020). Perspective switch and spatial knowledge acquisition: effects of age, mental rotation ability and visuospatial memory capacity on route learning in virtual environments with different levels of realism. *Cartography and Geographic Information Science*, 47, 14–27. https://doi.org/10.1080/15230406.2019.1595151

Lokka, I. E., Çöltekin, A., Wiener, J., Fabrikant, S. I., & Röcke, C. (2018). Virtual Environments as Memory Training Devices in Navigational Tasks for Older Adults. *Scientific Reports*, 8(1), 10809. https://doi.org/10.1038/s41598-018-29029-x

Mehrotra, A., Pejovic, V., Vermeulen, J.o., Hendley, R., & Musolesi, M. (2016). In *My Phone and Me*. New York, NY, USA: ACM. https://doi.org/10.1145/2858036.2858566.

Metze, T. (2020). Visualization in Environmental Policy and Planning: A Systematic Review and Research Agenda. *Journal of Environmental Policy & Planning*, 22(5), 745–760. https://doi.org/10.1080/1523908X.2020.1798751

Newell, R., & Canessa, R. (2015). Seeing, Believing, and Feeling: The Relationship between Sense of Place and Geovisualization Research. *Spaces & Flows: An International Journal of Urban and Extraurban Studies*, 6(4), 15–30. https://doi.org/10.18848/2154-8676/CGP/v06i04/15-30

Newell, R., McCarthy, N., Picketts, I., Davis, F., Hovem, G., & Navarrete, S. (2021). Communicating Complexity: Interactive Model Explorers and Immersive

Visualizations as Tools for Local Planning and Community Engagement. *Facets (Ottawa)*, 6(1), 287–316. https://doi.org/10.1139/facets-2020-0045

Onitsuka, K., Ninomiya, K., & Hoshino, S. (2018). Potential of 3D Visualization for Collaborative Rural Landscape Planning with Remote Participants. *Sustainability: Energy Practice and Policy*, 10(9), 3059. https://doi.org/10.3390/su10093059

Piaget, J., & Inhelder, B. (1956). *The Child's Conception of Space*. Humanities Press International.

Raaphorst, K., Duchhart, I., van der Knaap, W., Roeleveld, G., & van den Brink, A. (2017). The Semiotics of Landscape Design Communication: Towards a Critical Visual Research Approach in Landscape Architecture. *Landscape Research*, 42(1), 120–133. https://doi.org/10.1080/01426397.2016.1257706

Ruming, K. (2018). Post-political Planning and Community Opposition: Assigning and Challenging Consensus in Planning Urban Regeneration in Newcastle, New South Wales. *Geographical Research*, 56(2), 181–195. https://doi.org/10.1111/1745-5871.12269

Ruming, K., Houston, D., & Amati, M. (2012). Multiple Suburban Publics: Rethinking Community Opposition to Consolidation in Sydney. *Geographical Research*, 50(4), 421–435. https://doi.org/10.1111/j.1745-5871.2012.00751.x

Salter, J. D., Campbell, C., Journeyne, M., & Sheppard, S. R. J. (2009). The Digital Workshop: Exploring the Use of Interactive and Immersive Visualisation Tools in Participatory Planning. *Journal of Environmental Management*, 90(6), 2090–2101. https://doi.org/10.1016/j.jenvman.2007.08.023

Santella, A., & DeCarlo, D. (2004). Visual Interest and NPR. *Proceedings of the 3rd International Symposium on Non-Photorealistic Animation and Rendering - NPAR '04*. New York, New York, USA: ACM Press. https://doi.org/10.1145/987657.987669

Schroth, O., Pond, E., Campbell, C., Cizek, P., Bohus, S., & Sheppard, S. R. J. (2011). Tool or Toy? Virtual Globes in Landscape Planning. *Future Internet*, 3(4), 204–227. https://doi.org/10.3390/fi3040204

Semmo, A., Trapp, M., Kyprianidis, J. E., & Döllner, J. (2012). "Interactive Visualization of Generalized Virtual 3D City Models Using Level-of-Abstraction Transitions". *Computer Graphics Forum: Journal of the European Association for*. Computer Graphics, 31(3pt1), 885–894. https://doi.org/10.1111/j.1467-8659.2012.03081.x

Sheppard, S. R. J., & Cizek, P. (2009). The Ethics of Google Earth: Crossing Thresholds from Spatial Data to Landscape Visualisation. *Journal of Environmental Management*, 90(6), 2102–2117. https://doi.org/10.1016/j.jenvman.2007.09.012

Skulmowski, A., & Rey, G. D. (2020). The Realism Paradox: Realism Can Act as a Form of Signaling despite Being Associated with Cognitive Load. *Human Behavior and Emerging Technologies*, 2(3), 251–258. https://doi.org/10.1002/hbe2.190

Smith, H. S., & John, M. S. (2005). Naive Realism: Misplaced Faith in Realistic Displays. *Ergonomics in Design: The Magazine of Human Factors Applications*, 13(3), 6–13. https://doi.org/10.1177/106480460501300303

Smith, A., & Martín, P. P. (2021). Going beyond the Smart City? Implementing Technopolitical Platforms for Urban Democracy in Madrid and Barcelona. *Journal of Urban Technology*, 28(1–2), 311–330. https://doi.org/10.1080/10630732.2020.1786337

Smith, K., Berry, R., & Clarke, L. E. (2020). Exploring the Potential of Google Earth as a Communication and Engagement Tool in Collaborative Natural Flood Management Planning. *The Geographical Journal*, 186(2), 171–185. https://doi.org/10.1111/geoj.12323

Stanney, K. M., Mourant, R. R., & Kennedy, R. S. (1998). Human Factors Issues in Virtual Environments: A Review of the Literature. *Presence*, 7(4), 327–351. https://doi.org/10.1162/105474698565767

Thöny, M., Schnürer, R., Sieber, R., Hurni, L., & Pajarola, R. (2018). Storytelling in Interactive 3D Geographic Visualization Systems. *ISPRS International Journal of Geo-Information*, 7(3), 123. https://doi.org/10.3390/ijgi07030123

Velarde, J., Parejo, M., Blanco, J., & Moruno, L. (2017). Use of Video and 3D Scenario Visualisation to Rate Vegetation Screens for Integrating Buildings into the Landscape. *Sustainability: Science Practice and Policy*, 9(7), 1102. https://doi.org/10.3390/su9071102

Virtanen, J.-P., Hyppä, H., Kurkela, M., Vaaja, M. T., Puustinen, T., Jaalama, K., et al. (2018). Browser Based 3D for the Built Environment. *Nordic Journal of Surveying and Real Estate Research*, 13(1), 54–76. https://doi.org/10.30672/njsr.67846

Voyer, D., Voyer, S., & Bryden, M. P. (1995). Magnitude of Sex Differences in Spatial Abilities: A Meta-Analysis and Consideration of Critical Variables. *Psychological Bulletin*, 117(2), 250–270. https://doi.org/10.1037/0033-2909.117.2.250

Wilmer, H. H., Sherman, L. E., & Chein, J. M. (2017). Smartphones and Cognition: A Review of Research Exploring the Links between Mobile Technology Habits and Cognitive Functioning. *Frontiers in Psychology*, 8(April), 605. https://doi.org/10.3389/fpsyg.2017.00605

Wolbers, T., & Hegarty, M. (2010). What Determines Our Navigational Abilities? *Trends in Cognitive Sciences*, 14(3), 138–146. https://doi.org/10.1016/j.tics.2010.01.001

Wolfe, J. M., & Horowitz, T. S. (2004). What Attributes Guide the Deployment of Visual Attention and How Do They Do It? *Nature Reviews. Neuroscience*, 5(6), 495–501. https://doi.org/10.1038/nrn1411

Wüstenhagen, R., Wolsink, M., & Bürer, M. J. (2007). Social Acceptance of Renewable Energy Innovation: An Introduction to the Concept. *Energy Policy*, 35(5), 2683–2691. https://doi.org/10.1016/j.enpol.2006.12.001

Zhang, L., Geertman, S., Hooimeijer, P., & Lin, Y. (2019). The Usefulness of a Web-Based Participatory Planning Support System in Wuhan, China. *Computers, Environment and Urban Systems*, 74(March), 208–217. https://doi.org/10.1016/j.compenvurbs.2018.11.006

Zhang, Y. (2000). Using the Internet for Survey Research: A Case Study. *Journal of the American Society for Information Science*. https://doi.org/10.1002/(sici)1097-4571 (2000)51:1<57::Aid-asi9>3.0.co;2-w