# State of the Art of Molecular Visualization in Immersive Virtual Environments

David Kuťák, Pere-pau Vázquez, Tobias Isenberg, Michael Krone, Marc Baaden, Jan Byška, Barbora Kozlíková, Haichao Miao

<figure><img src="image_1.png" /></figure>

## To cite this version:

David Kuťák, Pere-pau Vázquez, Tobias Isenberg, Michael Krone, Marc Baaden, et al. State of the Art of Molecular Visualization in Immersive Virtual Environments. Computer Graphics Forum, 2023, 42 (6), pp.e14738:1–e14738:29. 10.1111/cgf.14738. hal-04006531

HAL Id: hal-04006531

https://inria.hal.science/hal-04006531

Submitted on 27 Feb 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
State of the Art of Molecular Visualization
in Immersive Virtual Environments

David Kut'ák¹,² ID, Pere-Pau Vázquez³ ID, Tobias Isenberg⁴ ID, Michael Krone⁵,⁹ ID, Marc Baaden⁶ ID, Jan Byška¹,⁷ ID,
Barbora Kozlíková¹ ID, Haichao Miao⁸ ID

¹Visitlab, Masaryk University, Brno, Czech Republic

³ViRVIG group, Universitat Politècnica de Catalunya, Barcelona, Spain

⁵Big Data Visual Analytics, University of Tübingen, Germany

⁷Department of Informatics, University of Bergen, Norway

⁹Department of Computer Science and Engineering, Tandon School of
Engineering, New York University, New York, USA

²AIT Austrian Institute of Technology, Vienna, Austria

⁴Université Paris-Saclay, CNRS, Inria, LISN, France

⁶Laboratoire de Biochimie Théorique, UPR 9080 CNRS, France

⁸Center for Applied Scientific Computing, LLNL, USA

<figure><img src="image_3.png" /><figcaption>**Figure 1:** For this state-of-the-art report, we surveyed the literature for papers focusing on molecular visualization in immersive environments. Many of them explore educational use cases (☑) or collaborative environments (♣). We also report on various enabling technologies, such as head-mounted displays (walnut) or augmented/mixed reality (Laptop). Furthermore, we also report on papers tackling interaction techniques (cell phone) or providing solutions over the web (台). Image sources: [GJB*20; MB21; RFK*21; OBD*19; CŠR*20] (permission for re-use obtained).</figcaption></figure>

**Abstract**

Visualization plays a crucial role in molecular and structural biology. It has been successfully applied to a variety of tasks, including structural analysis and interactive drug design. While some of the challenges in this area can be overcome with more advanced visualization and interaction techniques, others are challenging primarily due to the limitations of the hardware devices used to interact with the visualized content. Consequently, visualization researchers are increasingly trying to take advantage of new technologies to facilitate the work of domain scientists. Some typical problems associated with classic 2D interfaces, such as regular desktop computers, are a lack of natural spatial understanding and interaction, and a limited field of view. These problems could be solved by immersive virtual environments and corresponding hardware, such as virtual reality head-mounted displays. Thus, researchers are investigating the potential of immersive virtual environments in the field of molecular visualization. There is already a body of work ranging from educational approaches to protein visualization to applications for collaborative drug design. This review focuses on molecular visualization in immersive virtual environments as a whole, aiming to cover this area comprehensively. We divide the existing papers into different groups based on their application areas, and types of tasks performed. Further, we also include a list of available software tools. We conclude the report with a discussion of potential future research on molecular visualization in immersive environments.

**CCS Concepts**

• *Computing methodologies* → *Virtual reality*; • *Human-centered computing* → *Scientific visualization*; • *Applied computing* → *Molecular structural biology*;

# 1. Introduction

The significant benefits of virtual environments for the visualiza-
tion of scientific data have been established decades ago [Bry93;
Haa96; vDFL*00; LSSB12; MGK*13]. Yet only relatively recent
technological advancements—partially driven by the video game

industry [LTD*13]—have led to an increase in the widespread avail-
ability of affordable immersive hardware, especially in the form of
head-mounted displays (HMDs) [Mor16]. This development, in turn,
has led to the establishment of the subfield of Immersive Analytics
[DMI*18; FP21] within visualization research. As part of this work,
research has been carried out and tools have been developed that
rely on virtual or augmented reality—usually summarized under the umbrella term *mixed reality*, occasionally also called *extended reality* or the “metaverse” [LBZ*21]—that are now available outside of visualization research labs and have become a feasible and promising way of data exploration and/or analysis [MJK18] for many labs in the domain sciences. Furthermore, the omnipresence of small mobile devices, such as smartphones, with both a display and a camera now allows virtually everyone to experience some form of mixed reality—even if only with a monoscopic display—, for instance for educational purposes. Due to the importance and complexity of the three-dimensional structure of molecules, molecular visualization started to adopt immersive technologies already decades ago [AEFQ96]. Besides providing access to the intricacies of molecular shapes, a mixed reality approach also has the considerable benefit that it can show relevant non-surface data that affects many molecular interactions (e. g., electrical fields or charges), and it inherently facilitates collaborative data exploration and analysis scenarios [SWB*22].

Researchers have proposed many techniques, setups, and application cases to use immersive technology for the visualization of molecular data. Several surveys in the past have covered research on genome [NHG19] or molecular visualization [KKL*15; KKF*17] in general, on immersive environments for (bio-)medical visualization [VBB*04; VMR*21; FFV22], or for visualization in general [vDFL*00; vDLS02; OOKO15; DMI*18; FP21]. In our overview, however, we focus on the analysis of the state of the art of molecular visualization in mixed reality. Our specific focus is the use of modern head-mounted platforms, but we also discuss relevant work in the related areas, such as “general” VR in form of CAVES [CSD*92] and stereo rendering, as well as hand-held augmented reality. The research overviews most closely related to our own are those by Goddard et al. [GBS*18] and Calvelo et al. [CPG20]. In contrast to Goddard et al.’s survey, we examine the research landscape more broadly and without a particular own set of approaches as a comparison. Compared to Calvelo et al., we do not focus only on one specific application case or easily accessible software—we cover the field of immersive molecular visualization as a whole, considering wider range of challenges and future opportunities. Furthermore, we take a visualization-centric perspective on the existing work, while Calvelo et al. primarily provide an application-centric perspective, i.e., an overview of the available VR tools that can be used to view molecules in relation to COVID-19. Recent work also suggests advantages of immersive hardware specifically for molecular visualization, such as for perceiving the volume of a pocket [CŠR*20], for seeing data aspects previously unnoticed [GBS*18], or for engaging students in education [BWH20]. In our survey, we thus focus also particularly on these benefits and analyze specific application scenarios of immersive visualization approaches. Finally, we classify all approaches that we review in this report based on their application domain and supported tasks, noting that much past work focused on applications in education, often created by researchers within the bio-molecular research fields, instead of originating from visualization researchers (see Figure 1).

## 2. Background

We start our discussion by introducing the necessary knowledge, background, and terminology for understanding this report.

### 2.1. Definition of Immersive Environments

As we later need to distinguish between different aspects of virtual environments, we first clarify the respective terms. In the literature, there is no clear consensus about how to differentiate between *extended reality* (XR), *augmented reality* (AR), *mixed reality* (MR), and *virtual reality* (VR) and the terms are, consequently, not always used consistently. To avoid misunderstandings, we stick to the terminology established in the scientific literature, which follows the classical definition of Milgram and Kishino [MK94], who proposed to use the term MR for the whole “virtuality continuum” between AR and VR:

**Virtual Reality (VR)** puts the user into a purely virtual, fully immersive, simulated environment, which shuts them off from the real environment.

**Augmented Reality (AR)** combines digital information with the real world (either by embedding it into a live video stream or using see-through displays). The digital information can be only overlaid on top of the real environment or the virtual content can interact with real-world objects. The latter requires AR applications to maintain a detailed, three-dimensional model of the real world. AR is sometimes also called *hybrid reality*.

**Mixed Reality (MR)** is an umbrella term that encompasses the whole spectrum ranging from AR to VR. It stands for a digital, simulated world that can be merged with the real world to create an immersive, interactive experience.

Recently, the term **Extended Reality** (XR), which was not used by Milgram and Kishino, has emerged and is nowadays often used as an alternative umbrella term encompassing the whole spectrum from AR to VR. That is, XR can be considered to be equivalent to Milgram’s and Kishino’s definition of MR. This is partially to avoid confusion after Microsoft’s introduction of the *Windows Mixed Reality* ecosystem, which adopts the term MR for this specific context. However, there is also no clear consensus on the term XR, as it is sometimes used synonymously with the abovementioned definition of AR. Therefore, we decided to follow Milgram and Kishinos’ terminology and use the terms MR and *immersive environments* [DMI*18; FP21; LBZ*21] throughout this work to describe the settings on which we focus. We use these two terms interchangeably and mean them to encompass both AR and VR environments, i.e., the whole XR spectrum, as defined above.

### 2.2. Molecular Visualization

Molecular Visualization or Molecular Graphics—more specifically, the graphical depiction of three-dimensional molecular structures—has been a very active area of research for more than five decades [Lev66]. Therefore, a plethora of molecular representations or models has been proposed in that time [KKF*17]. On the one hand, this is since molecular structures are available on different scales, ranging from all-atom or even quantum mechanics data to coarse-grained data, where whole molecular complexes are represented as single data points. On the other hand, especially for atomistic data, there are many representations that visualize different properties of the molecular structure. For example, the ball-and-stick model shows the bonds between atoms, molecular surface models highlight the interface between the molecule and
its environment, and there are specific abstractions like the common twisted-ladder representation commonly used for DNA. Due to advances in data acquisition and simulation methods, data set sizes are continuously growing, which poses a challenge for interactive molecular visualization and drives the development of new, accelerated rendering methods. Time-dependent data that capture a molecular process and ensembles of data sets further add to the complexity. That is, suitable molecular representations used in interactive molecular visualization have to be chosen based on the data, the desired task, and the available processing power. For more details on these various aspects we refer the reader to Kozlíková, Krone et al.'s [KKL*15; KKF*17] surveys of common methods and representations in molecular visualization.

Especially the widespread availability of capable graphics hardware on almost all modern commodity computing devices—ranging from smartphones and tablets to desktop PCs—has led to considerable advances in interactive molecular visualization, not only concerning the visual quality but also regarding the size of the structures that can be displayed [CLK*11; MKA*19]. Modern molecular visualization research, however, does not only focus on scaling up existing visualization methods: to understand and explore huge, more complex data sets, traditional molecular graphics is not sufficient anymore and the development of advanced visual analysis methods is necessary. Examples include the interactive extraction and visualization of cavities in proteins [KKL*16], novel representations that support summarizing and comparing data sets [KFS*17], and visual analytics approaches for molecular dynamics simulations [BTM*19; SFS*21] or large-scale chemical compound screening [SUS*21].

Generally speaking, molecular visualization uses complex 3D models to represent rather abstract data that is inaccessible to a human observer in the physical world. Making these depicted structures easy to understand is thus an important task. Much research has focused on advanced rendering and illumination to improve perception [SVGR16] or on making tangible models. The latter, however, is not feasible for dynamic data and the use of this physicalization is mostly limited to educational scenarios [GO16]. To make such complex data and the spatial relations between entities easier to understand, the use of stereoscopic displays has a long tradition in molecular visualization, going back to the very early VR HMDs [Ihl97] or the original CAVE [CSD*92]. It is thus not surprising that molecular visualization nowadays often relies on the increasingly available, affordable, and user-friendly consumer-grade stereoscopic HMDs [Mor16]. As these devices have to process the scene separately for each eye, to achieve the stereoscopic effect and keep high refresh rates to avoid motion sickness, the computer graphics research focusing on improving the performance when rendering molecular data remains a vital endeavor along the visualization advancements [HL00; SLM*02; RKN*13; SSS16; MKH*18].

## 2.3. Hardware

Since different MR technologies require specific hardware, we discuss in the following sections the available display technologies for MR, associated interaction devices, and their advantages and shortcomings. Our goal is to provide the reader with the basic overview and terminology, whereas, for more in-depth tour, we would like to refer to some of the other existing reviews, such as those by

Hu et al. [HLC*21], Cárdenas-Robledo et al. [AHRA22], and Dincelli et al. [DY22].

### 2.3.1. Consumer-grade Immersive Hardware

From the proposed technologies that have dominated the market for the past 50+ years, HMDs are the only form factor that seems suitable for the public. Consequently, although in the past some molecular visualization systems were developed for CAVE-like systems [HDS96; YPHS04], most of the research and development in visualization and interaction in MR environments is nowadays commonly developed using HMDs [HP17; GBS*18; MKH*18; RBDR18; GK19; MBE19; PTA*20].

Though the first HMDs date from the 1960s [Sut68], it was not until 2012 that the first successful consumer-oriented device, Oculus DK1, was put on the market. There have been many previous attempts (almost 150 devices developed until 2007 are listed by Bungert [Bun]), but Oculus' products really caught the attention of the general public: probably thanks to the improvements in some key aspects of the technology and the right content. The year 2012 was also important for AR headsets, particularly thanks to the Google Glass presented at Google's I/O event that year. It was released to the public in 2013, but discontinued shortly after and re-released again in 2017 as an enterprise-only product. Microsoft took over the hype with their HoloLens, an AR HMD released for developers in 2016. Together with its successor, the HoloLens 2, it is currently the most common AR device in research laboratories.

In general, there are three types of dedicated immersive MR devices, though the first entry in the list cannot be considered consumer-grade due to the investment required:

**CAVE-like systems** [CSD*92] that consist of an enclosed room, where the user is surrounded by screens showing the virtual environment and can move freely within the room.

**3D screens** of different sizes and form factors [CPS*97; MRE13] with users in front of the screen; e. g., PowerWalls or 3D TVs.

**Head-mounted displays** (HMD) [Sut68], where a head-worn display is placed right in front (and around) the user's eyes.

Besides these three types of specialized MR hardware, handheld devices with front-facing cameras, like smartphones or tablets, can be used for simple AR applications.

### 2.3.2. HMD Features

There are two main families of HMDs: the ones oriented towards VR experience that block out other visual stimuli from the environment, and AR headsets, intended to produce synthetic images on top of the real information. The former usually consist of a flat panel worn close to the eye, with a pair of lenses, one for each eye, to increase the field of view and achieve the proper focal length. AR HMDs usually feature a see-through display and are commonly built using a light engine and an optical combiner. The light engine is responsible for generating the synthetic information, while the optical combiner delivers the images to the eye, while also transmitting the environment light [XHH*21]. To track users' position in space, HMD devices use integrated cameras or external sensors, assisted by inertial measurement units, such as accelerometers and gyroscopes, integrated into the HMD.
As for the visual quality, two important aspects of HMDs are
resolution and field of view (FOV). Early VR HMD devices had
a low resolution, and even the Oculus DK1's resolution was only
1200×1080 pixels (resulting in 640×800 pixels per eye) with a
FOV of 110°. Wider FOVs improve the sense of immersion and
have positive effects on distance judgment and motion sickness
[LDP*02]. However, popular VR HMDs still are limited in this as-
pect (e.g., HTC Vive Pro 2's approximately 116°×96°). Though the
resolution of commonly available devices also has not seen a major
leap, there are products that are promising, such as the Pimax 8K,
that already offers an exceptional resolution (3840×2160 per eye),
and human-like FOV (≈ 160°×115°) [HR*95; Str20]. As for the
world of AR HMDs, the field of view of the Microsoft HoloLens 2
is significantly lower ≈ 30–43°×29° than of its VR counterparts.
Latency is another important aspect in MR devices, since it may
cause cybersickness, especially in VR. Related characteristic is the
refresh rate [CGGG21], nowadays expected to be 90 Hz at minimum
for a comfortable experience. For more information about the MR
display technology, we would like to refer the reader to the review
by Zhan et al. [ZYX*20].

## 2.4. Hardware-related Specifics of MR Devices

One of the often-mentioned key advantages of MR is the ability
of the users to observe the complexity and internal relationships
of the data. This is quite true for 3D, but in visualization, we com-
monly use a set of 3D and 2D views in so-called Multiple Coordi-
nated View layouts. These configurations have only been studied
in very few cases—and using older technology—for MR environ-
ments [MCH*18] and best practices for embodied interactions and
embodied user interfaces are still not properly developed [Gra18]. A
second advantage of MR is that some aspects of the interaction, such
as changing the viewpoint, are achieved with natural movements.
The increased field of view is also important so that the application
can fit more data, although, as mentioned, this can be challenged
by the limited resolution of the displays. Furthermore, modern MR
hardware often also enables quite intuitive two-handed manipulation,
which increases the potential interaction abilities. Finally, HMDs
equipped with eye tracking sensors have become recently available
(e. g., HoloLens 2, HTC Vive Pro Eye, or Varjo XR-3) and other de-
vices can be retrofitted with eye tracking kits. This setup facilitates
novel interaction concepts and input mechanisms (see, e. g., Groß et
al. [GBR*19]).

However, MR environments are not short of disadvantages either.
In the rendering side, current untethered stand-alone hardware de-
vices like HoloLens or Oculus/Meta Quest 2 are significantly less
powerful than a desktop [MKH*18; MBE19]. This limits the com-
plexity of the 3D scenes that can be dealt with (and the framerates
that can be achieved). Wired devices, tethered to a PC exhibit better
behavior (the PC is the one rendering the scenes), but the connec-
tions can be annoying for the users as it restricts free movement.
Foremost, some tasks that are easy to achieve using mouse and
keyboard, such as data introduction, are difficult to achieve through
3D menus and virtual keyboards or voice commands. Goddard et
al. [GBS*18] state that “a virtual keyboard floating in the VR scene
is feasible but tedious.” Many modern HMDs like the HoloLens 2,
the Meta Quest 2, or the HTC Vive Focus 3 offer hand tracking

using multiple cameras integrated into the HMD. While this technol-
ogy is convenient and user-friendly as it does not require additional
hardware, the accuracy is still limited and the hands of the user have
to be within the field of view of the cameras. Furthermore, no haptic
feedback—that is, letting the user feel the interaction with virtual
objects—is available, which lessens the user experience. To obtain
a fully immersive experience (that is, reacting to users’ senses),
haptic feedback is desired. Operations such as palpably touching
or grasping greatly improve the sense of embodiment. A possible
solution is haptic gloves, which have been used already in early
VR systems [AF98; KPL*04]. However, in addition to the incon-
venience of wearing gloves while using the MR system, modern
haptic gloves like the Noitom Hi5 or the Manus VR haptic that are
commercially available are mostly not in a typical consumer price
range. Therefore, especially in VR, the most prevalent interaction
devices are tracked controllers tailored to gaming, which are usually
included with consumer-grade VR HMDs. Although the only hap-
tic signal provided by most of these common controllers is just a
small vibration (vibrotactile feedback), it already helps to improve
the sense of embodiment. MR, on the other hand, mixes physical
objects with virtual ones and therefore allows for some degree of
physicality [NPD*21]. Another important aspect is collaboration.
AR devices allow users to communicate naturally because they can
see each other. However, this is not true for VR HMDs, where
only virtual representations—“’avatars”—of the other participants
are typically available. Though high-bandwidth connections may
allow for remote collaboration setups, there are still numerous open
challenges in this area. Furthermore, movements in the virtual world
that are not aligned with the real world have to be designed with
great care as otherwise motion sickness can occur [KDSH22].

## 2.5. Software-related Specifics of MR Devices

Software development for MR needs to take the sensor data and
input signals described in Section 2.3 into account to deliver a seam-
less and sophisticated user experience. Usually, it needs to generate
highly interactive 3D graphics for a stereoscopic display. The Soft-
ware Development Kits (SDK) for modern HMDs usually include
functionality for automatic tracking and localization, as well as
stereo rendering. As an alternative, SDKs like Vuforia [PTC] offer
image recognition to create AR applications (mainly for hand-held
AR using smartphones or tablets). Developing an MR visualization
application from scratch that handles all necessary input and output,
however, is still a challenging task. Most visualization researchers
and developers thus use the existing frameworks. Especially game
engines often provide support for various HMDs. Unity [Tec] and
Unreal [Gam] Engine are commonly used, because they provide
instant functionality for developing applications with advanced in-
teractions and optimized rendering. An often-cited advantage is also
the large, powerful community, customer support, and asset stores.
The game engines do not come without limitations. While they are
easy to use and their scripting features encourage rapid prototyping,
they offer less control over rendering and the general tool func-
tions. Greater flexibility is possible at the cost of more technical
effort if the prototype is built from less-restrictive frameworks, or
even from scratch. In case of using existing engines, costs may also
play a role. While Unreal Engine is free if no revenue is generated,
Unity requires paid license for institutions whose annual revenue
is more than $100,000. However, academic institutions can apply
for a free education grant. Apart from these, free open-source game
engine Godot also offers MR support. Finally, many web-based
frameworks have MR support through the WebXR standard (e. g.,
three.js, A-Frame, or Babylon.js).

In summary, the technological stack for the development of molec-
ular visualization MR applications can take various forms, and many
approaches have been indeed used. As a by-product of our paper
search procedure, we collected list of existing ready-to-use MR
applications offering molecular visualization features. As this con-
tent is not the main focus of this report, we list these application in
*Supplementary Material S2*.

3. Typology

After describing the necessary background and providing the readers
with basics about the current status of MR hardware and software,
we dive into the approach we used when compiling this publication.

**3.1. Paper Collection Methodology**

For our paper discovery process, we opted for a systematic proce-
dure, combining database queries and manual searches, that went as
follows:

1. We agreed on a list of keywords that describe the intended area of focus. These keywords were chosen from the three relevant areas: mixed reality, (molecular) biology, and visualization (see point 4 for the list of keywords colored by area).

2. Then, we collected a list of databases and search engines poten-
tially containing relevant papers (listed also in point 4).

3. Based on the keywords, we assembled various search queries and tested their compatibility and outcomes on the selected databases.

4. Next, we performed several iterations of the first three steps. During these iterations, we filtered the keywords and databases, and optimized the query to provide more relevant results. In the end, we searched five databases (IEEE Xplore, Web of Science, ACM Digital Library, EG Digital Library, PubMed) and used the query ("virtual reality" OR "virtual-reality" OR "virtual environment" OR CVE OR HMD OR head-mounted OR "virtual collaborative environment" OR "mixed reality" OR "augmented reality" OR "extended reality") AND (molecular OR molecule OR biological OR biomolecular OR biochemical OR chemical OR bioinformatics OR cell OR cellular OR microbiology OR pharmacology OR protein OR DNA OR lipids OR membranes OR ligands OR QM OR RNA OR docking OR biomaterials OR drug design OR atom OR atomic OR crystal OR membrane) AND (visualization OR visualisation OR "visual analytics" OR "visual representation").

5. We created a shared Zotero (www.zotero.org) library for the
storage of all discovered publications.

6. We performed the first iteration over the query results to collect an initial set of papers.

7. Furthermore, after this step, we defined several categories which were used for the classification of papers based on their application area, used technology, publication type or other characteristics (see *Supplementary Material S1*). Assignment of categories to the individual papers was realized via corresponding Zotero tags.

8. After the first iteration, we decided to narrow down the search results to papers published in the year 2010 or later, as we focus primarily on recent technologies. We then went back through the selected databases and reviewed all returned results to ensure that we had included all relevant publications.

9. In the end, our Zotero library contained about 250 selected articles after this step. This also included publications that were not discovered by the query, but by manual search in the references of discovered papers or using other means.

10. We re-processed all the papers to assign the categories more precisely. During this step, we also filtered out irrelevant publications, as well as duplicates.

11. After narrowing down the paper count to 199 articles, we moved the Zotero library to a shared Google Docs table. This table allowed for more easy-to-use addition of textual comments. We divided the publications included in this table based on their core categories, which were used as a basis for the final classification of publications presented in this report. When processing the assigned papers, we further excluded the less relevant ones. In the end, we ended up with 127 publications, forming the core of the paper (see Table 1).

**3.2. Review Structure**

Our report consists of two main parts that arose from the categorization performed during the paper collection. In the first part, *Application Domains (Section 4)*, we focus on areas where the presented publications are used. This part is further split into two distinct categories—Education (Section 4.1) and Research (Section 4.2)—dividing the approaches between learning-oriented and experts-focused ones. In the second part, *Tasks (Section 5)*, we discuss papers from the perspective of the different goals that can be achieved using the presented approaches. We further divide this part into two self-explanatory sub-parts—Visualization (Section 5.1) and Collaboration (Section 5.2).

4. Application Domains

We begin our discussion of contributions by looking at domains to
which the papers contribute most.

4.1. Education Use Cases

Among all application domains, education is a major application
area for molecular visualization in immersive virtual environments,
receiving a lot of attention from the researchers over the years.

4.1.1. Target Audience

The goal of educative applications is usually straightforward—to deepen the target audience’s knowledge of the presented subject. This is the case in most approaches focused on students, being a prevalent target audience. Nevertheless, even the student-focused solutions can be further split into several categories based on the expected level of education. Molecular Zoo [GBS*18] and ProteinScannAR [NSM*12], for instance, focus on high-school students, while BiochemAR [SWL*20] and Peppy [DDG*20] aim for undergraduates. A distinction can also be drawn based on the target audience of
user studies—most researchers verify their approaches directly with students [SD18; BWH20; CGY20; PTA*20], while some perform an evaluation with teachers instead [NSM*12; SCM18]. As noted by Johnston et al. [JRA*18] in their paper on the mixed reality exploration of a cell, the attractiveness of the immersive virtual environments can also make it easier to engage the public. Thus, MR molecular visualization can target not only experts but also members of the public by providing a highly immersive introduction to the core concepts of molecular and structural biology. In consequence, this may open doors to an upcoming generation of biologists.

## 4.1.2. Learning Challenges and Outcomes

Generally speaking, students are tackling several challenges when learning about the structure and function of molecules. These challenges are not related only to the subject itself, but also to the techniques and approaches that are used to facilitate the study process—such as molecular visualization. The report by Jones et al. [JJS05] divides the students’ difficulties when dealing with molecular visualization into four areas, namely visual subtlety, complexity, abstractness, and conceptual depth. The first mentioned area—visual subtlety—is particularly interesting in the context of immersive environments, since it describes difficulties in interpreting spatial relationships in molecular visualization. Since MR is supposed to improve the spatial understanding of the visualized content [She03; DSKG06; KPB22], it has the potential to tackle the issue of visual subtlety naturally. Indeed, in the comparison of an AR application and a traditional desktop molecular visualization tool performed by Sung et al. [SWL*20], the students showed increased spatial awareness when using the AR solution. Moreover, it was generally accepted as being easier to use. Similarly, in the work of Peterson et al. [PTA*20] and Safadel et al. [SW19], students exhibited increased satisfaction with the spatial aspects of the AR molecular visualization. In the user study performed by Qin et al. [QCC21] based on the Nanome VR application [KBL*19], the spatial characteristics of the VR solution—influencing both the visualization of structures, and manipulation with them—were also strongly appreciated by the participants. This result aligns with the discoveries of Coan et al. [CGY20], who provided strong evidence that the students considered VR as a helpful tool when trying to better understand specific structural aspects of the presented molecules. Furthermore, the two VR labs conducted over a duration of a single semester by Coan et al. also exhibited additional positive results. In the first place, the learning outcomes of the labs were met. Therefore, the positive aspects of employing VR were not at the expense of educational goals. Then, Coan et al. discovered that the students’ responses to the VR were even more positive when they were introduced to the second VR lab. This observation suggests that the initial enthusiasm was not purely due to the novelty of the technology but due to its actual benefits and strengths.

An additional important—and often mentioned—outcome of user studies performed in the area of MR molecular visualization is increased user engagement: the participants generally seem to enjoy the ability to see and interact with the structures in the immersive environment [SJPG18; FCM*19; WMT*19; QCC21; SF21]. This can make lectures potentially more enjoyable. In addition, increasing the students’ engagement may have positive effects on the overall learning outcomes and interest in the study subject [CKK06;

<figure><img src="image_4.png" /><figcaption>**Figure 2:** Screenshot captured in the VR application Peppy [DDG*20] focused on teaching of principles of polypeptide structure. The students can interact both directly with the structure, as well as with the two-dimensional interaction dashboard.</figcaption></figure>

LR19]. As for the learning outcomes, Fujiwara et al. [FKH*20] developed an immersive virtual environment for the teaching of valence shell electron pair repulsion theory, a model for prediction of molecular geometry. In their user study, participants were split into two groups—one used the virtual environment while the other completed a traditional learning procedure—with the former group achieving better learning outcomes. Finally, an interesting case for employment of MR devices in the education scenario was presented by Lu et al. [LXZ21], proposing a virtual environment—focused on students with limited mobility—allowing them to perform chemical laboratory experiments by remotely controlling a specialized robot.

## 4.1.3. Open Questions

Despite the achievements mentioned above, there remain challenges and open questions with respect to the use of MR molecular visualization in education. Brown et al. [BWH20], e. g., compared virtual reality to traditional modeling and computer simulation, with students performing tasks such as the building of small molecules. Despite the researchers’ expectations, the study outcomes did not show any significant differences between the performance of students using the three methods. Brown et al. conclude that VR might be more suitable for complex processes than those presented in the study—therefore, this technology could not fully utilize its strengths in this case. Contrary to their results, in the studies performed by Brůža et al. [BBMK21] and Bennie et al. [BRD*19], VR produced better educational outcomes than corresponding desktop approaches. In the virtual reality study performed by Won et al. [WMT*19], it was observed that the students did not sufficiently consider the 3D structural aspects of the presented enzymes, despite the fact they are naturally available thanks to the immersive visualization, and instead focused mainly on color-coded electron density map when performing the given tasks. Patterson et al. [PLAM19] faced additional MR challenges in their gamified cell exploration. They discovered during the design process that the immersive environment could lead to an overly high cognitive load since it places the user in an unknown environment, simultaneously offering multiple points of interest. Therefore, the depth of scientific information and structural detail should be adequately tailored.
#### 4.1.4. Educational Goals

Educational approaches can be further divided based on the educational goal. In some cases, the focus is on broadening the understanding of structural aspects of molecules—focusing on their overall shape, symmetries, or other spatial characteristics [NSM*12; BJ14; CŠR*20; LT20; PTA*20; SWL*20]. Therefore, the user mostly perceives and explores the presented structure as is. On the contrary, in other applications the student is expected to be more involved with the visualized content by designing new parts or interacting with the data to better understand their interactions and dynamic behavior [RSH18; FCM*19; ABT*20; DDG*20; FKH*20; SWF*21; vWGK22]. For example, in NuPov [ABT*20], students are simulating a nucleophilic attack on a molecule by firing the nucleophile using their fingers. In InteraChem [SWF*21], students are performing various interactions, such as distorting molecules or trying to move hydrogen atoms through a benzene ring. All actions are supported by interactive molecular dynamics simulation, providing immediate feedback. InteraChem is also related to the topic of valence shell electron pair repulsion theory, which is the main focus of the VR tool presented by Fujiwara et al. [FKH*20]. In this application, students try to predict the geometry of the given molecule, while being guided by the real-time simulation of atomic forces. Simulation-backed approach is also chosen by the authors of MOF-VR [vWGK22], an application focused on construction, simulation, and visualization of metal-organic frameworks. In VR application Peppy [DDG*20] (see Figure 2), the students modify various properties of amino acid residues, such as torsion angles and strength of hydrogen bonds, while also being able to mutate residues, to better understand the effects of these changes on the final polypeptide structure. The effects themselves are computed by underlying molecular simulation approach to provide as real behavior as possible. While Peppy is developed primarily for VR, it also works on a desktop without any headset or stereo vision. This is an interesting addition, allowing one to experience the application, albeit less immersive, also in cases when the desired hardware is not readily available. This may be especially useful for students who want to examine the content individually outside of school.

#### 4.1.5. Gamification

A special category can be defined by approaches that build on the concept of gamification. Despite the proven educational benefits of this technique [BHH20], however, the number of applications using this approach in combination with MR molecular visualization is rather low. One of the most recent examples of gamification seems to be Pepblock Builder VR by Yallapragada et al. [YXW*21]. This application teaches the concepts of protein design, while employing a post-apocalyptic narrative and LEGO-style concepts to make the educative aspects more approachable. Another recent gamification approach has been presented by Patterson et al. [PLAM19] focusing on cell exploration. In this case, the gamification aspects are influencing both the visual side of the application and the interactions. For example, the students board a virtual platform onto which they descend into the cell nucleus. In the nucleus, they can perform various interactions, such as grabbing an RNA polymerase molecule and placing it onto the DNA strand to initiate transcription. A focus on cell exploration is present also in the work of Johnston et al. [JRA*18]. The colorful art style chosen for the visual

side of their application also suggests inspiration by gamified approaches. Moreover, they also show the user a mini map of the cell, simplifying navigation in the environment. Application of gamification in medicinal chemistry has also been extensively studied by Falah et al. [FWA*21], identifying additional positive aspects of gamified MR approaches. Finally, gamification was also explored in the early 2000s by Lu et al. [LLZC], who proposed an application providing a steering wheel support, enabling to use it to control a virtual racing car driving along an HIV secondary structure, for example.

<figure><img src="image_5.png" /><figcaption>**Figure 3:** *DNA-protein and protein-protein interactions being examined in AR application MoleculARweb [RFK*21].*</figcaption></figure>

#### 4.1.6. Technologies

Used technology is an additional factor distinguishing the educational approaches. In this area, current research covers both virtual reality [dSNGF17; SJPG18; FCM*19; BBMK21; MB21] and augmented reality [NSM*12; SW19; AD20; PTA*20; FCS21]. The chosen technology and hardware play an important role for educational institutions, being an environment that often works with more limited resources and less flexibility than individuals or small laboratories. The choice of technology thus does not only influence the students' experience but also additional factors—ease-of-use, portability of the application (Section 2.3.2), the maximum number of students using it simultaneously (Section 2.4), as well as overall cost. In this regard, web-based solutions appear as a promising approach, to reduce some burden [NSM*12; Abr20; CŠR*20; RFK*21; FCS21]. As demonstrated by MoleculARweb platform [RFK*21] (see Figure 3), web applications can offer a wide range of readily available study content, being a huge advantage for education. In any case, the accessibility of the chosen technology also plays an important role in the content creation process. While most applications provide content prepared by the developers [SJPG18; FCM*19; SWL*20], some researchers also focus on means for customized content creation, aimed at teachers [CGY20; BBMK21; RFK*21; CDA22; BF17] or students [AD20]. Some researchers
thus published guides describing a procedure for creation of MR
molecular content [GJK19; Abr20; ENP20]. Eriksen et al. [ENP20],
e. g., present a step-by-step guide on the development of a molecular
AR visualization application and its deployment to the Google Play
store, all without requiring any programming skills.

4.1.7. Rendering Techniques

Since molecular visualization is inherently coupled with computer graphics, educational approaches can be further distinguished based on their approach to the rendering of structures. In some applications, the visualized structures are pre-generated in some existing molecular visualization tool and then imported as a common three-dimensional model [BJ14; SCM18; SW19; ENP20; SWL*20; SF21; FCS21]. In other approaches, the structures are rendered in real-time based on underlying data, as common in regular molecular visualization tools [MKH*18; KBL*19; CŠR*20; LT20]. Each of these approaches comes with its strong and weak points. For example, with the former approach, the creator can create precisely crafted structural visualization decoupled from a specific application, making it thus possible to open the molecule in a variety of programs. On the other hand, real-time rendered molecules give users a possibility to render any of the many available molecules, achieving higher flexibility in the pipeline together with an option to dynamically change the properties of visualizations based on the current needs. The choice thus depends on a variety of factors, ranging from the needs of the course curriculum, through capabilities of the expected hardware, up to the skills and time constraints of the development team.

## 4.2. Research Use Cases

Apart from education, another important domain is biology re-
search, offering a variety of focus topics. For example, the study
of the constituents and behavior of molecules is of great impor-
tance in fields such as drug design, analysis of molecular dynamics
simulations, and structural analysis. Immersive applications have
emerged naturally, from traditional molecular modeling software
packages [SSS16; GBS*18] to newly developed use cases and ap-
plications [KBL*19; OBD*19]. While much of the relevant work
in the literature focuses on direct translation to the virtual environ-
ment [GBS*18; DWH*20; KSB*22], a small category exploits the
additional degrees of freedom and modalities that MR offers to
represent non-3D intrinsic molecular data, such as genome-wide
association data [WNM20]. In the end, MR-related research use
cases are broad with many application areas, as introduced in the
following subsections.

4.2.1. Common Activities in Use Cases

Based on our analysis of the most common activities performed
on molecular structures, we found four recurring basic activities
that a user performs in the virtual environment. The first activity,
molecular *exploration*, typically focuses on the perception of a
larger-scale molecular data [XLX*21; CŠR*20]. For the *analysis*
of molecular data, being a second activity, we note that it often
involves experts who, with specific domain goals in mind, take
advantage of immersion and the natural interactions of MR to an-
alyze structure and dynamics [GBS*18; MB21; RBDR18]. The

<figure><img src="image_6.png" /><figcaption>**Figure 4:** *The example on the top shows a common intrinsic 3D use case where users take advantage of an immersive environment for drug design [DWH*20]. The example on the bottom shows a typical non-3D intrinsic use case where the three dimensions in an immersive environment are used to lay out non-spatial data [PMI*21]. Permission for re-use obtained.*</figcaption></figure>

third activity is *manipulation*, where the user is interested in con-
structing and modifying a molecular structure to create new molec-
ular conformations [NGEB15; GAB*18; DSJ19]. The last activity
comprises the *presentation* of molecular knowledge, where a user
takes a more passive role with less prior knowledge, usually for
educational purposes [NGEB15] and often in a collaborative envi-
ronment [KBL*19]. In the following subsections we largely omit
the respective use cases in education that relate to these four activi-
ties, since we have already discussed them in detail in Section 4.1.
Many ideas covered by these educational applications could give
rise to interesting use cases. Surprisingly, this does not seem to have
happened yet, as the academic and industrial use cases described in
the literature so far only reflect a limited subset of features compared
to educational applications.

4.2.2. Intrinsic and Non 3D-intrinsic Data

Given the intuitive spatial perception in an immersive environment and the inherent 3D structure of molecules, traditional use cases in the molecular domain can be directly transferred from desktop-based approaches. Therefore the three-dimensional spatial awareness that the immersive devices intuitively add to any molecular structure dataset naturally leads to use cases that explore these spatial features and are very often cited as a major advantage. Natural candidates that rely on intuitive perception of spatial structures include interactive drug design [AW99; NGEB15], molecular dynamics simulation
analysis [AF98; NMB*16; GK19; DWH*20], and protein structure
analysis [RBDR18; XLX*21]. Interestingly, several works use the
added dimension in the immersive environment and the natural way
of interacting to visualize molecular data to add elements that are
not intrinsically 3D as depicted in Figure 4. The added dimension
in virtual space is used to lay out the data, which can be directly
related to the object under study or to generally useful information.
Todd and Emsley [TE21], e. g., allow users to place a widget into the
scene to show the current time. Similarly, research data that is not
naturally mapped to three dimensions can also be enhanced. Probst
and Raymond [PR18], for instance, represent the interrelationships
of a molecular data set in a chemical space as a cloud of points, in
which each molecule is represented as a point when viewed from a
distance. The molecular structure is only displayed when the user
gets closer. As another example, for genome-wide association data,
this approach could consist in representing allele frequency, *p*-value,
and chromosomal position in cylindrical coordinates to arrange the
data around the user. Relationships between biological elements,
often expressed as networks [SAKW02; LKF*17; PMI*21], are
visualized in 3D to allow the user to explore and analyze big data
without visual occlusion. These are examples of data without spa-
tial attributes, where the immersive environment is used only for
layout purposes. However, there are also hybrids that lie between
intrinsic and non-3D intrinsic data, such as Hi-C data that indirectly
provide spatial information through 2D interaction frequency views
[ZSW*15]. In this case, the 3D spatial positions must first be recon-
structed and then visualized in an immersive environment to enhance
the representation of chromosomal interactions and relationships,
highlighting another advantage of using a 3D environment: the direct
representation of positions in 3D instead of indirect 2D views.

4.2.3. Structural Analysis

For structural analysis tasks, the stereoscopic view provided by
immersive devices is often cited as an advantage, allowing for iden-
tification of features in the dynamic 3D environment that would oth-
erwise remain hidden in a 2D display. Moritz and Meyer [MM04]
present a case study of structural exploration of pyruvate kinase
where they also note that certain perspectives and representations in
VR may hinder the recognition of some structural motives. On the
other hand, VR provides clear advantages, one of which is described
in [CŠR*20]: clipping of surfaces is more easily avoided due to fine
camera position control and a wider field of view. The authors illus-
trate this observation by comparing a 2D ligand-protein view with a
VR view in their ProteinVR tool [CŠR*20]. ProteinVR specifically
addresses hypothesis generation for research purposes based on im-
mersive visual analysis (in addition to classroom use) of modeling
results from docking and molecular dynamics calculations. Such
structural analysis may also include the calculation of specific prop-
erties of the molecular structure under study, such as the electrostatic
potential, which can be visually analyzed in VR once the calcula-
tion is complete as described by Laureanti et al. [LBO*20]. Their
VR version also interfaces with the Adaptive Poisson-Boltzmann
Solver that can be configured in the VR environment without a
command line interface by allowing users to prepare input files for
the solver by selecting and grouping atoms. Similar concepts are
used in ChimeraX in VR [GHM*18] that connects the VR environ-
ment to several analysis and modeling components. With ChimeraX,

Goddard et al. provide a generic VR-enabled toolbox for structural
work as well as a way to share such experiences in collaborative VR
sessions for analysis that can be implemented in either ChimeraX or
AltPDB [GBS*18]. The presentation and sharing of such molecular
visualization experiences, particularly structural analysis, connects
to the sharing process described by Martinez et al. [MB21] follow-
ing the principles of FAIR. Sharing also finds echo in approaches
that allow users to create guided tours, as was proposed by Alharbi
et al. [ASL*23]. Similarly, VRdeo provides the means to create in-
teractive experiences by recording user actions that can be replayed
[BBMK21]. Instead of painting the molecular scene, their approach
incorporates visibility management and a camera to create trips
through two mesoscale biomolecular models. An important aspect
for all such use cases is the consideration of hardware limitations.
In the context of AR, such limitations for developing use cases are
discussed by Müller et al. [MKH*18] for the HoloLens device. In
[CPWG20] it is argued that AR has been somewhat overlooked so
far, but provides enticing opportunities for research as is illustrated
with the use case of antibody interaction analysis. Details on vi-
sual tasks related to structural analysis use cases can be found in
Section 5.1.5.

4.2.4. Interactive Drug Design

We have found that interactive drug design is one of the most
prominent use cases described in the literature, going back decades
[AW99]. Unsurprisingly, the most common reasons for using VR are
the added spatial awareness [NGEB15] and the natural interactions
that support particularly the modeling tasks [TLL*11]. In drug de-
sign, visually exploring the spatial conformation of molecules such
as ligands and proteins [TLL*11; DWH*20] is essential, and the
ability to select a molecule and view it from different angles while
changing its representation is key. Drug development teams often
bring together scientists from diverse backgrounds, for whom the
intuitive VR approach opens a natural window for sharing insights
about 3D molecular data, even for researchers who do not routinely
work with the 3D shapes of molecules. An example is the ability
to go inside molecules and look around, just like buildings, which
is a completely unique way of looking at these objects [NGEB15].
For drug development, and in particular docking, it is beneficial to
manually adjust the molecules in VR [LWLW18]. This approach,
guided by the user's intuition, demonstrates that, in VR, manual
adjustments are no longer limited the same way they are on the
desktop. Automated approaches can thus be supported in VR by
user interventions using 3D interactions that are otherwise difficult
to achieve on the desktop [XLX*21; KSB*22]. A key challenge is
that the manipulation combines two purposes: either customizing
the representation or actually modifying the molecular structure
through modeling. For example, one can change the position of
parts of a molecule merely to better view the interior, but the same
interaction could also be interpreted as a change in the actual spatial
conformation.

4.2.5. Molecular Dynamics Simulation

Immersive manipulation allows the user to go beyond simply exam-
ining a fixed model [DWH*20; LFB22]. The idea, which is common
in the literature, it to provide the user with a tool to quickly under-
stand the response of a molecular system to a given user input. Such
manipulations may be done by a single user or when multiple users coexist in the virtual environment. Narupa [OBD*19] allows such multi-user manipulation of MD simulations in real-time. The way in which the user interacts, e.g. using haptic and or visual feedback or not, is an important element. Koutek et al. [KHPB02] describe an immersive control environment for positioning particles for MD simulations in real-time. They use a spring manipulator that allows the user to navigate the particles through a molecular system and provide visual force feedback of physical plausibility. Because the interactive manipulation may cause changes to the molecular system, dynamic visual cues are also important. An example concerns encoding bond saturation when changing molecular conformation in VR and could be implemented in a ludic way using emojis [SWF*21]. Visual cues may also be adapted to specific types of data handled in a given MD simulation, as in Nakano et al. [NMB*16]'s use case of visualizing and analyzing electron transfer data from MD simulations in virtual reality. They use an HMD and describe how the immersive environment allows the user to walk through the data, and provide the user with means to identify trends in dynamic systems that might otherwise be missed. In a similar spirit of modeling data exploration, Salvadori et al. [SFP*16] present a toolbox and illustrate use cases for docking, molecular dynamics analysis, and exploration of modeling results at the QM and MM levels (see Figure 5). The authors explore both CAVE and HMD implementations. Ratamero et al. [RBDR18] focus more specifically on the utility of HMDs for consumers and propose a system that allows users to interact with static displays but also to consider protein dynamics. They conclude that immersion helps inexperienced users analyze protein-ligand interactions and understand the conformational changes in protein dynamics. NOMADVR [GK19] targets VR environments at different price points, including HMDs but also cardboard-type devices and is focused on materials systems. In another focus on materials' science, Hagita et al. [HMO19] describe molecular dynamics simulations of phase separation of ABA block copolymers and experimental observations of filler morphologies in rubber. They present a methodology and benchmark measurements to evaluate such experiments, which is a welcome step forward in assessing such experiences. Molecular dynamics often is perfomed subsequent to model building activities. MOF-VR [vWGK22] specifically targets an integrated package that goes from model building to molecular dynamics of guest molecules in Metal-Organic Frameworks (MOFs). Model building use cases are described in more detail in the next section.

### 4.2.6. Modeling and Model Building Approaches

The key challenge in molecular modeling is to generate 3D models based on experimental data. The examples discussed in the literature include building atomic models from X-ray crystallography or electron cryo-microscopy (CootVR [TE21]), modeling DNA shape from Hi-C data (CSynth [TTM*21]) or even directly manipulating atoms via hand-controlled scanning probe microscopy manipulation of single molecules [LGE*15]. The authors of CootVR [TE21] report an order of magnitude speedup when modeling is performed in VR instead of using the desktop. Their work offers insight on the challenges of VR molecular modeling with 6 DOF controllers. First, the arms must rest on a surface, as the user can quickly become fatigued from the elevated arm position (i. e., gorilla arm

such experiences[tb] or in assessing

<figure><img src="image_7.png" /><figcaption>**Figure 5:** *The combination of 2D chart being embedded directly into the 3D scene in Caffeine [SFP*16]. The two linked views allow for a better understanding of the data. The image was adapted. Permission for re-use obtained.*</figcaption></figure>

syndrome). Second, modeling sessions should be less than 25 minutes to limit eye and neck strain. The biggest challenge of working in a virtual environment using HMDs is isolation: actions like drinking coffee, using other software, or even taking notes become very tedious as the headset must be removed each time. While authors of CSynth [TTM*21] focus primarily on the computational aspects of the modeling problem, they also remark that VR helps with the exploration of the 3D models during the modeling process and can be useful for education. VR visual control can also be used to enhance direct experimental manipulation of single molecules as described in [LGE*15]. The authors of this paper add 3D visual feedback relevant for scanning probe microscopy (SPM) by displaying the currently executed trajectory and the position of the SPM tip during manipulation in real-time. This information is combined with additional control data to assist the experimentalist.

The construction of novel biomolecular structures in VR, such as DNA, was demonstrated by Schkolne et al. [SIS04] and Kut'ák et al. [KSB*22]. Modeling of DNA molecules is different from other biomolecular structures because it consists of a chain of repeating units that behave predictably. For DNA nanotechnology, an interesting observation in [KSB*22] is that the exact alignment of structures is difficult as the controller has to be held in 3D space. However, the authors deal with this accuracy problem by adding constraints to the modeling process that are dictated by biology, e. g., the user can only place the DNA at a specific spatial position with a given rotation, so it remains biologically relevant. Schkolne et al. [SIS04] go beyond the software and propose peripherals (tongs, ray gun, handle) specifically designed to build DNA molecules. For collaborative modeling, Grebner et al. [GNE*16] present a user-friendly web-based platform for 3D modeling of molecules, with applications such as ligand alignments and molecular docking. Similarly, the work of O'Connor et al. [OBD*19] also provides multiple users with the ability to edit MD simulations in real-time. Modeling oper-
ations can be entered without peripherals but via gestures, such as
selecting and placing molecular fragments [APM13]. Finally, while
the main contribution of Abriata et al. [Abr20] is based on education,
it also gives a perspective on interesting AR-based applications in
modeling biological macromolecules. These examples indicate that
some fine-tuning is needed to make 3D modeling tasks in VR really
efficient, be it related to the specific nature of the molecules, to the
interaction modalities or to the hardware used.

4.2.7. Illustration

Illustrative visualization of molecular structures for the desktop has recently gained popularity (e. g., [KAK*18; HMK*20; KIK*21]). Although generic 3D painting has been explored as a powerful method for creating art in VR, we have found only few approaches that focus on molecular illustration in VR, despite the positive benefits of this technique, for example for research dissemination. Cellpaint-VR [Cen12] is the 3D version of Gardner et al.'s CellPAINT [GAB*18], which allows non-artistic users to create molecular scenes. Their approach uses stereoscopic rendering and includes isomorphic interactions that mimic painting, erasing, spraying, and coloring. The approach by Johnston et al. [JRA*18] also starts from the cellular level to create a VR model based on experimental data that serves as a template, with the future idea of further populating this vivid model up to the molecular level. LifeBrush [DSJ19] allows users to paint molecular agents that can be arranged to form a molecular simulation for illustrative purposes. The authors use the controllers to enable various interactions, such as sculpting, path line and event trace visualizations, to create complex molecular scenes for educational purposes. While MR illustrative visualization has been explored by the visualization community to some degree, there are still many open questions for the generation of molecular illustrations, given their importance for education and science communication. We have found few examples that incorporate many biological models to facilitate the creation of static and dynamic illustrations. An exception to this rule is the work of Alharbi et al. [ASL*23], focusing on VR guided-tours in dense molecular environments. A fundamental question in this context is how much biological context is required to make complex molecular illustrations, i. e., to what extent does biological information need to be included, compared to an approach that uses general 3D modeling and painting? We believe that incorporating the repetitive nature of biomolecular structures, as done in the work described above, should be included in illustration-based approaches, even though they may also limit the flexibility of illustrators.

4.2.8. Genomics

We found several papers that focus on visualizing genomics data
in VR. Genomics data is often large and has complex relationships
that take advantage of the generous space in a virtual environment
for visualization. Pirch et al. [PMI*21] allow users to explore large
genomic networks in VR. The authors argue that VR offers new
approaches to combine human cognition with advanced data science
methods. Zhu et al. [ZSW*15] argue that the 3D view facilitates the
discovery of patterns that are difficult to see on the desktop. They
demonstrate visualization for spatial reconstruction of the human
genome in VR. A different type of spatial reconstruction is used by
Zhang et al. [ZPCC19], who connect sequence data and structural

spaces using proof-of-concept software to integrate protein and
nucleic acid data, using Gria2 and its gene product as an example.
Each amino acid or nucleotide of interest in the sequence is linked to
the corresponding site in the protein structure. Although genomics is
not intrinsically 3D, Stolk et al. [SAKW02] demonstrate a creative
use of 3D space in VR and report that bioinformaticians can identify
new relationships between genes that may have otherwise remained
hidden. Although most work focused on 3D data, the work described
above demonstrated the potential of VR for non-3D data. We believe
there is even more potential here. The key, however, is to find a good
method for layouting or mapping to 3D in VR.

4.2.9. Potential For Visualization Research

Much of the work covered in the use cases we described addressed the possibility of visualizing and interacting with molecular data in immersive environments. These were mostly research prototypes, except for a few established molecular visualization systems [SSS16; GBS*18]. Many of the papers were published in the domain itself rather than in journals, such as IEEE TVCG. While many papers report the benefits of spatial perception and natural interactions, some also point to limitations, indicating the need for research to reach a pivot point where these specific use cases are primarily facilitated in an immersive environment. We believe that additional research should focus less on translating a desktop approach to VR and more on evaluating the effectiveness of specific visual encoding and interaction relating to the use case. A hybrid setup that combines desktop and immersive environments as it is being investigated for other application domains [WBR*20] may also be possible. Evaluation between modalities (e. g., VR vs. desktop) may not always be feasible due to the novelty of immersive devices or the lack of a comparable desktop tool, but some generic comparisons of specific aspects such as input and output devices [WBAI22] are possible. Evaluation within a modality through a quantifiable measurement of task completion time, however, could advance the field of molecular visualization in immersive environments.

5. Tasks

After having discussed education and research use cases, we now
turn our attention to specific tasks supported by immersive tools.

5.1. Visualization Tasks

In particular, we have analyzed the papers regarding the basic vi-
sualization tasks needed for understanding any kind of data as de-
scribed by Shneiderman [Shn96]: *Overview, Zoom, Filter, Details-
on-demand, Relate, History, and Extract*. Because most of the papers
we surveyed do not explicitly refer to these tasks in any way, we pro-
vide a summary based on our observations. In addition, we analyze
how these traditional visualization tasks align with the immersive
tools that often rely on isomorphic interactions [FCW19].

5.1.1. Task 1: Overview

The vast majority of techniques described in the surveyed papers
provide the readers with an overview of the data by showing the
whole molecule in 3D using common spatial representations, such
<figure><img src="image_8.png" /><figcaption>**Figure 6:** *The interface of Eukaryo [YSCJ16] with a minimap (top-right corner) that indicates the relative location of the user inside the cell. Permission for re-use obtained.*</figcaption></figure>

as molecular surface or ball-and-stick. The reason is that, in the im-
mersive environment, zooming out is a rapid way to get an overview
of the entire data by reducing the overall size of all molecules.
Nevertheless, one could still argue that some representations are
better suited for overview purposes (e. g., ribbon representation),
while others give more details (e. g., ball-and-stick or van der Waals
representations). Only a few surveyed papers provide representa-
tions tailored explicitly for overview purposes. Probst and Ray-
mond [PR18] represent every molecule in the dataset as a point
in a point cloud, and only if the users get closer, the actual spatial
representation of a molecule is used. An overlayed minimap, such as
in Eukaryo [YCJ16] (see Figure 6), can also aid users to understand
the location and spatial relationships of molecular structures and
often acts as a navigational device. Alternatively, focus and context
techniques [CYB*05; KSB*22] can be used to define a region of
interest. Doutreligne et al. [DGC*15] also suggested a method of
constraining the camera trajectory to minimize the occlusion.

In addition, derived quantitative measures often provide a good overview of specific structural properties. Numerous quantitative measures depicted as 2D plots (e. g., [ZPCC19; SFP*16; DDG*20]), however, are challenging to display in immersive environments and are only rarely used. Using semantics to link them to the underlying 3D objects seems to have great potential, but has been explored only little so far [TFF*18]. Such representations, therefore, remain a good opportunity for future research.

**5.1.2. Task 2: Zoom**

Depending on the type of MR devices, the users can use a pair of
6 DOF controllers, special gloves [KPL*04], or even fingers to per-
form gestures (e. g., moving the controllers closer to each other) that
will zoom in or out on items of interest [LLZC; KSB*22; DDG*20;
TE21]. The MR zoom in and out operations are equivalent to the
desktop-based zoom operations, where users increase or decrease
the size of the structure to see more or fewer details, respectively.
The users are also often allowed to move the molecules closer to
them (which makes them naturally larger due to the perspective
camera). This operation is achieved by grabbing the molecules with
controllers or using remote interactions via ray casting to select
and pull distant objects. We can achieve a similar effect by letting
users walk or move the device depicting the scene (e. g., cellphone

or tablet) closer to the molecule of interest. Solutions such as tele-
porting within the VR scene are often used to cross larger distances
and reach a more advantageous viewpoint. A unique solution was
suggested by Patterson et al. [PLAM19] who use a slowly moving
platform (similar to an elevator) that carries the user through the
scene. Zoom does not only provide a better view of details, but also
accounts for the lower pointer accuracy of VR controllers that cannot
be stabilized in the air by making the selection easier through more
prominent visual elements [KSB*22]. Zooming into molecules can
also be achieved using voice commands, as described by Goddard
et al. [GBS*18].

**5.1.3. Task 3: Filter**

Tasks such as selection and filtering are inherently coupled with
visualization due to the various structural properties that can be cal-
culated from the atoms and bonds models. Nevertheless, in the pa-
pers we surveyed, especially in early approaches, we often only saw
uses of VR/AR technology to display molecular data, without more
advanced interaction techniques such as filtering or selecting data.
Yet, several researchers recently recognized the challenges posed by
cluttered molecular environments and allow users to hide molecules
or their parts on demand [ZSW*15; dCN17; GBS*18; CŠR*20;
DDG*20] or use clipping [RBDR18; TE21]. While most of these
techniques completely remove the filtered data, some solutions only
suppress unimportant data by modulating their opacity [CYB*05].

**5.1.4. Task 4: Details-on-demand**

The more recent molecular viewers (e. g., [CGY20; MB21]) allow users to change between multiple molecular representations, such as ball-and-stick, ribbon, or surface, each providing a different level of detail. While means of smoothly transitioning between such different visual abstraction [VI18; VCI20] stages exist in the non-immersive literature (e. g., [vdZLBI11; MDS*18; HMK*20]), we did not find any that would offer similar means in MR environments.

Additional detail is often provided by labels that appear when
users touch, hover over, or point towards a part of a molecule with a
controller. A useful functionality is measuring distances and angles
between selected atoms [ZW17; KBL*19; RB20]. Selecting the
correct atoms of interest, however, may prove challenging due to the
visual clutter caused by the complex molecules. Kazatzis [Kaz20]
thus suggested an alternative context-aware selection technique
based on selecting the relevant atoms using 2D directional input.

A related challenge is controlling the amount of actual detail
shown to users to avoid cluttered views. We saw two main ap-
proaches to this challenge—either the control is left to the user or
an appropriate representation is selected automatically. For manual
approaches, users may be able to adjust the level of detail globally
by moving a slider [YCJ16] or, as shown by Kut'ák et al. [KSB*22],
use more complex lenses, allowing them to set a local level of detail.
Some systems also handle the visual clutter by providing parts of the
user interface (e. g., axes, view panels, interactive buttons, etc.) only
when requested by the user [TE21]. In automatic approaches, the
distance from the camera is often used as the main criterion [PR18].

In desktop applications, details are often retrieved via brush-
ing and linked views. But we have not yet seen a fully-fledged
solution that would use a similar approach in immersive environments (at least not for molecular data). The most common examples include static 2D charts showing results of various computations [TLN17]. More interactive solutions, however, are starting to appear as well [SFP*16]. A middle-ground option is to combine the mixed reality with a desktop-like interface in multi-window applications that often utilize web technologies [TFBB16; MMD*18; JTO*20]. While the spatial aspects are investigated using, e. g., head-mounted displays, the standard 2D desktop interface provides non-spatial views. By using semantic approaches, the details can be accessed with very little disruption [TFF*18].

An interesting take on a details-on-demand task was provided by Martinez et al. [MB21] who consider immersive environments only as one of the modalities that can be used to explore data. In their FAIR sharing model, the overview may be provided with one modality, while details can be provided with another.

### 5.1.5. Task 5: Relate

The relate task pertains to the understanding the relationship between the visual elements in the virtual scene. These visual elements can either encode the same data, different data, or the relationship itself. Within the papers, we found that a quite commonly represented task was to relate various parts of datasets to each other. We found that there are four distinct ways how this visual relationship can be formed, which we use to structure our following discussion: *colocation, spatial, dynamic, and explicit visual elements*.

**Colocation** is used when visual elements appear at the same location and the user can switch between them. Because they appear at the same location the user will know that they refer to the same data. Tools such as UnityMol [LTD*13], ChimeraX [GHM*18], or custom visual representation designs [FKH*20; KSB*22; MB21] thus enable users to switch between different representations of a molecule or its constituents, depending on the derived molecular property of interest (e. g., all-atom representation vs. secondary structures). As the representations occupy the same space, it indicates to the user that the same data or its derivatives are depicted. However, in some cases, the elements do not have to align exactly on top of each other. This is useful when comparing multiple similar structures where the users are interested primarily in the differences. For example, Todd et al. [TTM*21] proposed a method where multiple results of DNA structure modeling can be shown and the corresponding areas are connected by semitransparent polygons. The technique immediately highlights regions with low and high variation (see Figure 7). Another special form of collocation is the possibility to share viewpoints in collaborative environments, as proposed by Chastine et al. [CYB*05]. We discuss this topic further in Section 5.2.

The **spatial** relation is used when, due to the proximity and geometric features of visual elements, users can derive interaction between these two visual elements. Particularly in molecular visualization the key challenge is understanding the spatial relationships between or within molecules, such as the distance of fragments or their geometric fit. This concerns two spatial properties: position and geometry of objects. In VR, users can relate these two properties by “grabbing” two objects and positioning them side by side. Two structural conformations after MD simulation, e. g., could be compared in

<figure><img src="image_9.png" /><figcaption>**Figure 7:** *The history trace view in CSynth [TTM*21] showing the differences between several predicted DNA models. Permission for re-use obtained.*</figcaption></figure>

this way or ligand-protein interaction through user intuition-guided docking [XLX*21]. Understanding distance and sizes is crucial for modeling use cases, such as interactive drug design [TLL*11; NGEB15; DWB*20] as we described in Section 4.2.4. Similarly, the modeling of nanostructures relies on fitting and aligning several molecules [KSB*22]. In addition, some methods map non-spatial information to the position of data items in the environment to relate their properties [WNM20]. Some tools also utilize illustrative representations such as toon shading and contours to simplify the scene and produce non-photorealistic renderings [OBD*19].

The **dynamic** relation refers to the case when the same visual element appears and can be recognized at different time steps. It appears quite often since many of the papers we analyzed were concerned with dynamic data, such as the MD simulations we described in Section 4.2.5. Here, the task is often to set up the computation parameters [GJB*20; JTO*20] or understand action-reaction relationships when exploring the results of the MD simulation (e. g., [FZG11; HKM*19; DWB*20]). Among the physically inaccurate solutions we can list Molecular Zoo [GBS*18], which simulates interactions between molecules that are freely moving in the space around the user. The notion of random movement within the crowded molecular environments can be also achieved using particle effects [YCJ16]. Finally, users are sometimes encouraged to interact with the environment directly to influence the simulations of dynamic processes [OBD*19; PLAM19; VWGK22].

The most common method to encode relationships and properties is via **explicit visual elements** such as lines, highlights, and glyphs. While multiple coordinated views are employed on desktop applications, this task is slightly more challenging in an immersive environment as these views coexist in the same space, hence becoming spatial-visual objects themselves. The typical task using this approach is to explore the properties of individual atoms, e. g., their charges [LBO*20], by placing additional visual cues, such as lines depicting the electrostatic field next to them. Another variant of the same task is the exploration of bonds. For instance, Seritan et al. [SWF*21] use emoji placed on top of atoms to show the bond saturation (see Figure 8).

Slightly more challenging are cases where there are one-to-many or many-to-many relationships between the spatial elements (e.g.,
<figure><img src="image_10.png" /><figcaption>**Figure 8:** The emojis are used in InteraChem [SWF*21] to label happy/angry atoms based on a bond saturation. The image was adapted. Permission for re-use obtained.</figcaption></figure>

atoms) and other non-spatial data. A typical solution to this challenge is to use 2D planes embedded within the 3D environment, where the additional non-spatial information is depicted [KHPB02; TLN17; CGY20]. For AR, the markers can serve as the mapping planes [RSH18]. Similar to linked desktop views, the additional non-spatial information can be used as an interactive tool for selecting various parts of molecular structure [ZPCC19; DDG*20].

While some solutions abstract from the spatial representation of molecules and use node-link diagrams to depict relationships [SAKW02; LKF*17; AGM*18; PMI*21], others only “flatten” the spatial representations and, while technically still being 3D, the representations appear two-dimensional [SJPG18]. For direct fully-fledged spatial representations we observed that, especially when using AR technology, multiple approaches depict possible interactions between molecules when a user places the molecules in proximity (e. g., by moving AR markers representing the molecules close to each other) [RFK*21; Abr20; ABT*20].

### 5.1.6. Task 6: History

The vast majority of papers we surveyed describe research prototypes that only rarely keep track of user actions. The few exceptions include the ability to reset the application to one of the few preset states, such as in the case of Rodríguez-Sotres et al. [RRG*09], or tracking the user actions for undo functionality that can be found in more complex tools. As for matured desktop applications such as ChimeraX [GBS*18] that got extended with VR viewing capabilities, undo/redo functions are already built-in. In our opinion, there is a clear opportunity to explore provenance visualization when it comes to immersive environments. The tracking of the user's head and controllers provides valuable information on their behavior, which can also be analyzed in experiments and used for further navigation as shown by Leinen et al. [LGE*15].

### 5.1.7. Task 7: Extract

Extraction of subsets to another file for sharing, further analysis, and presentation is one of the last tasks in visualization. None of the papers and techniques we surveyed explicitly discusses possibilities for extracting or exporting subsets of the data for further

<figure><img src="image_11.png" /><figcaption>**Figure 9:** Screenshot from the VRdeo tool, showing a user setting up a camera trajectory before exporting the video recording [BBMK21]. Permission for re-use obtained.</figcaption></figure>

use and analysis. Although crucial for an effective workflow, these tasks are often not considered as a key paper contribution, hence are rarely implemented or described. Furthermore, many publications focus on demonstrating the system using prototypes rather than such saving/export features. Some systems that focus on the integration into the workflow of domain scientists provide export functionality, such as DNA sequence export into FASTA format after modeling in VR [KSB*22]. The purpose of exporting functions can be grouped into two categories. The first is to export data for further analysis in different tools. Defining these interfaces through data formats is essential, as VR often does not provide a full feature set but only deals with certain subsets, such as visualization and modeling. In this category, Martinez et al.'s [MB21] extract task relies on the FAIR model of sharing the data among users, while Goddard et al. [GBS*18] describe ChimeraX' capabilities for exporting data to AltPDB. The second export use case is to create images and videos for presentation and publication, as described by Spark et al. [SKE*20]. Alternatively, the immersive environments can be used to enable the user to create biomolecular scenes for educational purposes (see Figure 9) and save and export the results [GAB*18; DSJ19; BBMK21]. While the extraction task is often neglected in research prototypes, we believe that more systems will need this built-in functionality as molecular visualization in MR environments matures, providing new research opportunities. While seemingly easy on the desktop, this task involves navigation through a file structure and text input, which is inherently challenging in VR.

## 5.2. Collaboration Tasks

Visualization plays an important role in the success of researchers' efforts. Similarly, the means of collaboration may significantly determine what can be accomplished. In this regard, MR interfaces are providing users with extensive options for collaborative work, namely when comparing them with desktop solutions. Especially in biology and biochemistry, where we face very complex phenomena, collaborative efforts and joining forces and experience of researchers worldwide are necessary for future progress in these fields. For that, MR represents an ideal environment and opens completely new possibilities.

In general, a meaningful collaboration in virtual environments
can foster research, dissemination, and educational activities. Billinghurst et al. [BCBM18] introduced and discussed the general concept of *collaborative immersive analytics*. They viewed the topic from different perspectives: spanning from the different types of possible collaboration to diverse users, their roles, and related interaction possibilities. Although they outline many insightful general aspects, in this section we address the topic of collaborative VR from the perspective of molecular visualization, where several very interesting approaches were presented throughout the years.

### 5.2.1. Asynchronous Collaboration

In 2009, Lee et al. [LQK*09] developed VR system for remote collaboration, with particular application in molecular docking and crystallography. They focused primarily on asynchronous collaboration, solving the problem of different time zones of users. One of their solutions is inspired by the version control concepts for collaborative work. The system records the session of the user as a set of files that are stored in the collaboration server and can be later downloaded, reviewed, and further enhanced. Asynchronous communication is supported also by the 3D-Lab web-based platform introduced by Grebner et al. [GNE*16], which aimed to provide the users with a collaborative and user-friendly interface for 3D modeling of molecules, with particular applications in conformer generation, ligand alignments, or molecular docking. The platform has a modular architecture and its ultimate goal was to promote interactions between drug designers. Another example of asynchronous communication was recently presented by Brůža et al. [BBMK21]. Here, the tutor prepares a VR scene with educational content, which can be later entered and interactively explored by students.

### 5.2.2. Synchronous Collaboration

One of the earlier synchronous collaborative multi-view virtual environments for molecular visualization and modeling was proposed in 2005 by Chastine et al. [CYB*05]. They focused on one of the most challenging tasks in collaborative VR: how to handle situations when different users manipulate the same part of a molecular structure. In their case, each user determines an “area of interest” with a bounding box. Objects outside this area are de-emphasized, which also improves the clarity of the user’s view. This area of interest is visible to the other users as well for location and focus awareness. They represented the user in the environment by 3D hand models and they experimented with the size of the models when the user is working. This concept supports the three main issues that must be addressed in such collaborative environments: awareness of the presence, attention awareness, and action awareness.

Synchronous communication between users was also addressed in research efforts by Corrêa et al. [FSTdP10], who targeted multi-projection and collaboration capabilities between geographically dispersed research groups. The interaction mode they implemented is based on handing the token of the active user, as only a single user at a given time can interact with the system. Other users are queuing requests for the control, and negotiations between users are realized through text and voice communication. Similarly, Corrêa et al. [FTG11] focused on the integration of two or more geographically separated research groups working on the same molecular visualization. They decided to utilize one of the standard applications for molecular visualization, JViewer, operating in VR.

Maes et al. [MMD*18] introduced MinOmics, a visualization framework dedicated to multi-omics interactive visual analysis. By utilizing UnityMol WebGL, it supports stereoscopic representations, and WebVR is used for integrating the framework into VR. They proposed four different scenarios for possible interaction between users. The first one utilizes the wall-sized display, showing a single monoscopic instance of the scene to all users in the room. Only one user can interact with such a setup. The second scenario combines the wall-sized display with VR, using a VR headset. In their arrangement, only a single user can use the VR setup, other users only see a restricted 2D view of the 3D scene the VR user is controlling. In a third scenario, they experimented with stereoscopic projection on wall-sized displays, while a fourth scenario finally offers a fully immersive experience using only VR. At the time of publication, however, this last setup was still restricted to only a single person.

In 2018, Goddard et al. [GBS*18] released three VR applications: ChimeraX for analyzing molecular structures and electron and light microscopy data, AltPDB for collaborative discussions about atomic models, and Molecular Zoo for teaching young students characteristics of biomolecules. In ChimeraX, two or more users can join the same VR session. Each participant is represented by simple cones for hands and a postage stamp, where the user can upload a photo. Any user can then manipulate the molecular structure and point at features of interest, as a part of collaborative discussions. In AltPDB, the social VR site AltspaceVR handles the technical complexity of initiating the multi-person VR session, maintaining the synchronized views, audio connection, and customizable avatars. This environment as well as the users' avatars are more elaborated than in ChimeraX, leading to a richer overall user experience.

Probably one of the most elaborate VR applications to date for collaborative viewing, manipulating, and modifying chemical and macromolecular structures is Nanome [KBL*19]. Kingsley et al. present the architecture of the tool, three proposed collaborative modes, and a demonstration of its usage in designing a small molecular structure. The goal of Nanome is to minimize the need for the technical know-how, and thus reduce the communication barrier between structural biologists and other disciplines to enhance the idea flow and collaboration within drug-discovery teams. Nanome can host more than 10 users at the same time. A user is represented by an avatar with a head and hands (see Figure 10). Furthermore, the user can act in one of three possible user roles: virtual participant, 2D viewer, and ghost mode. The first role offers the full VR experience and the user is represented by an avatar in the scene. At a given time, only a single user (presenter) can manipulate the loaded structures, to avoid clashes with the other virtual participants. These can still move autonomously in space and freely explore the structures. In addition, the presenter can teleport all other avatars to his or her location, giving all participants the same view onto the scene. The 2D viewer is visible in the virtual scene as a video camera that is visible to other participants. Such users can freely move in the space and speak to other participants, but cannot become a presenter and are thus prevented from manipulating the scene. The ghost mode provides the users with a passive viewing experience from a fixed position and is intended for streaming the scene to a large audience.

Narupa [ODD*18; OBD*19; DWH*20; JOB*20] is an open-source software package where multiple users can cohabit in the
<figure><img src="image_12.png" /><figcaption>**Figure 10:** Nanome environment with two collaborators sharing the view of the protein and its binding pocket. Image taken from [KBL*19]. Permission for re-use obtained.</figcaption></figure>

same VR space and interact with real-time molecular simulations. It is based on the iMD-VR software framework and extends its functionality by enabling a multi-person VR experience, setting up and customizing interactive user manipulation with MD simulations, and running the application on a local network. Via controlled studies in the laboratory, they demonstrated that Narupa and iMD-VR tools enable researchers to complete molecular modeling tasks more quickly than using conventional interfaces, such as mouse or touchscreen.

Gauthier et al. [GMS19] developed Dynamic Virtual Proteins, a methodology of interaction design that combines human interactions and virtual agents that are assisting humans with the manipulation of a virtual protein. The methodology operates with three levels of user interface: (a) manual, (b) collaboration with a virtual agent, where some actions are taken by the agent, and (c) automatic, where all actions are performed by the agent. They are targeting mixed audiences—experts and non-experts—, operating in the shared space. They focus specifically on collaborative tasks from engineering and decision-making about new drugs and treatments.

Collaborative VR was recently thoroughly studied by Martinez and Baaden [MB21]. They examined options for sharing curated visualizations of structural biology, modeling, and bioinformatics datasets for interactive and collaborative exploration. Their goal is to enable easy sharing of the visual experience with others. In their setup, each participant is represented by an avatar and has their point of view, separated from the others. However, participants may choose to group to adopt a similar viewing angle.

One of the latest approaches that was motivated by the communication issues caused due to the COVID-19 pandemic, is ProMVR by Xu et al. [XYTT21]. This tool aims to support the collaborative tasks of protein designers and provide them with a virtual environment containing the protein 3D model and intuitive interactions with peers. Although the work on this tool is still in progress, it already shows interesting results.

### 5.2.3. Collaboration in Augmented Reality

Augmented reality opens another perspective on collaboration inside complex molecular scenes. An indisputable benefit is that users can walk around the molecular model and choose their views without

influencing others. Moreover, AR allows users to point out interesting locations in the molecule to each other naturally. Moreover, the communication between collaborators is natural, as there is no need for avatars. However, a downside is that such a solution can only support collaboration between co-located users.

Very recently, Noizet et al. [NPD*21] published their application framework for augmented 3D printing for molecular modeling. The molecule of interest is first 3D-printed, and this physical model is subsequently used as a support for the visual augmentation by the superimposition of additional visual representations. This is realized using a HoloLens. The results of their experimental testing showed that such a mode facilitates collaboration and dissemination.

### 5.2.4. Concluding Remarks on Collaborative Environments

Although the importance of communication and collaboration is evident and indisputable, only the recent pandemic showed us that the usage of collaborative virtual environments provides an excellent option in cases when the physical presence is not possible. The above-mentioned examples of already existing approaches and tools that support the collaborative aspects are first pioneers in these efforts. As a consequence of the recent dramatic events that influenced lives of all of us, we believe that this domain of building collaborative virtual environments will become one of the leading research fields in the near future.

## 6. Evaluating MR Research

As the domain of MR molecular visualization is still rather new, there are no long-term established procedures for the evaluation of the presented results. Nevertheless, as presented in Table 1, many of the authors use some kind of validation of their concepts. Five basic levels of evaluation seem to be currently employed in general, similarly to common non-MR visualization research [LBI*12; IIC*13]:

* *No evaluation.* Some authors completely omit evaluation or, at least, do not mention it in their publication.

* *Informal evaluation.* In some cases, the evaluation is informal, i. e., it is mentioned that some people tried the presented work and shared their impressions but no specific procedure is presented.

* *Software-based evaluation.* In some publications, the evaluation is performed by comparing the properties of achieved results (e. g., novel algorithms) with other existing solutions.

* *Qualitative evaluation.* This type of evaluation usually involves smaller number of users, often experts in particular domain, providing often subjective feedback on the given work.

* *Quantitative evaluation.* This type involves more participants to evaluate a given application, while the specific procedure is often designed to allow authors to statistically analyze results.

In the case of our collection of papers, the quantitative approach seems to be the most commonly used one. However, the choice is also affected by the domain and target audience of the application. For example, for applications aimed at education it is often natural choice to use quantitative evaluation with a group of students (as outlined in Section 4.1.1). On the other hand, researcher-focused approaches may be more suited for qualitative evaluation, as the
<table><caption>**Table 1:** The list of surveyed papers with the dots indicating the topics covered by the given paper: education (✉), collaboration (✉), web applications (✉), HMDs (), AR/MR (), and interaction (■). Furthermore, the table also shows if the given paper was evaluated and if so, in what way: no evaluation (X), informal evaluation (≈), quantitative evaluation (△), qualitative evaluation (○), and software-based evaluation (S).</caption><thead><tr><th>Reference</th><th>✉</th><th>✉</th><th>✉</th><th>✉</th><th>✉</th><th>Evaluation</th><th>Reference</th><th>✉</th><th>✉</th><th>✉</th><th>✉</th><th>Evaluation</th><th>Reference</th><th>✉</th><th>✉</th><th>✉</th><th>✉</th><th>Evaluation</th></tr></thead><tbody><tr><td>[Abr20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[GHM*18]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>S</td><td>[PTA*20]</td><td>●</td><td>●</td><td>●</td><td></td><td>△</td></tr><tr><td>[AF98]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[GNE*16]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td><td>[PMI*21]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[APM13]</td><td></td><td></td><td></td><td></td><td></td><td>≈</td><td>[GBR*19]</td><td></td><td></td><td></td><td></td><td>≈</td><td>[PR18]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[AW99]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[HMO19]</td><td></td><td></td><td></td><td></td><td>■</td><td>[PB20]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[AGM*18]</td><td></td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[HP17]</td><td></td><td>●</td><td>●</td><td></td><td>■</td><td>[QCC21]</td><td>●</td><td>●</td><td>●</td><td></td><td>△</td></tr><tr><td>[AD20]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[HL00]</td><td></td><td></td><td></td><td></td><td>■</td><td>[RTH16]</td><td></td><td></td><td></td><td></td><td>≈</td></tr><tr><td>[ABT*20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>▲</td><td>[JOB*20]</td><td></td><td></td><td></td><td></td><td>■</td><td>[RB20]</td><td></td><td></td><td></td><td></td><td>≈</td></tr><tr><td>[BWE17]</td><td></td><td></td><td>●</td><td></td><td></td><td>■</td><td>[JRA*18]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td><td>[RBDR18]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[BRD*19]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>○</td><td>[JJS05]</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[RFK*15]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[BJ14]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[JTO*20]</td><td></td><td></td><td></td><td></td><td>■</td><td>[RKN*13]</td><td>●</td><td></td><td></td><td></td><td>≈</td></tr><tr><td>[BF17]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[Kaz20]</td><td></td><td></td><td></td><td></td><td>■</td><td>[RRG*09]</td><td>●</td><td>●</td><td></td><td></td><td>△</td></tr><tr><td>[BM20]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[KPL*04]</td><td></td><td></td><td></td><td></td><td>■</td><td>[RFK*21]</td><td></td><td></td><td></td><td></td><td>△</td></tr><tr><td>[BWH20]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[KBL*19]</td><td></td><td></td><td></td><td></td><td>○</td><td>[RSH18]</td><td>●</td><td>●</td><td>●</td><td></td><td>≈</td></tr><tr><td>[BBMK21]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>▲</td><td>[SD18]</td><td>●</td><td>●</td><td>●</td><td></td><td>○</td><td>[SW19]</td><td>●</td><td>●</td><td></td><td></td><td>△</td></tr><tr><td>[TW13]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[KHPB02]</td><td></td><td></td><td></td><td></td><td>≈</td><td>[SFP*16]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[CŠR*20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[KSB*22]</td><td></td><td></td><td></td><td></td><td>○</td><td>[SIS04]</td><td></td><td></td><td></td><td></td><td>△</td></tr><tr><td>[CPWG20]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[LNQ*19]</td><td>●</td><td>●</td><td>●</td><td></td><td>≈</td><td>[SWF*21]</td><td>●</td><td>●</td><td>●</td><td></td><td>△</td></tr><tr><td>[CYB*05]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[LBO*20]</td><td></td><td></td><td></td><td></td><td>■</td><td>[SLM*02]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[CGY20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>▲</td><td>[LT20]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td><td>[SJPG18]</td><td>●</td><td>●</td><td></td><td></td><td>△</td></tr><tr><td>[FSTdP10]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[LKKK11]</td><td>●</td><td>●</td><td>●</td><td></td><td>S</td><td>[SF21]</td><td></td><td></td><td></td><td></td><td>≈</td></tr><tr><td>[CDA22]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[LQK*09]</td><td></td><td></td><td></td><td></td><td>■</td><td>[SBKW18]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[dCN17]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[LGE*15]</td><td></td><td></td><td></td><td></td><td>■</td><td>[SKE*20]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[DSJ19]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[LLNW14]</td><td></td><td></td><td></td><td></td><td>■</td><td>[SAKW02]</td><td></td><td></td><td></td><td></td><td>≈</td></tr><tr><td>[DWH*20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>○</td><td>[LMKT05]</td><td></td><td></td><td></td><td></td><td>▲</td><td>[SSS16]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[DWB*20]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[LKF*17]</td><td></td><td></td><td></td><td></td><td>■</td><td>[SWL*20]</td><td>●</td><td>●</td><td></td><td></td><td>△</td></tr><tr><td>[DDG*20]</td><td>●</td><td></td><td></td><td></td><td></td><td>▲</td><td>[LXZ21]</td><td></td><td></td><td></td><td></td><td>▲</td><td>[SCM18]</td><td>●</td><td>●</td><td></td><td></td><td>○</td></tr><tr><td>[dSNGF17]</td><td></td><td></td><td></td><td></td><td></td><td>S</td><td>[LLZC]</td><td>●</td><td>●</td><td>●</td><td></td><td>▲</td><td>[TLN17]</td><td></td><td></td><td></td><td></td><td>△</td></tr><tr><td>[DGC*15]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>≈</td><td>[MMD*18]</td><td></td><td></td><td></td><td></td><td>S</td><td>[TTM*21]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[DPSR98]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[MTK*10]</td><td></td><td></td><td></td><td></td><td>▲</td><td>[TE21]</td><td></td><td></td><td></td><td></td><td>S</td></tr><tr><td>[EDC*19]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[MB21]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td><td>[TFBB16]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[ENP20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>▲</td><td>[MFVB15]</td><td></td><td></td><td></td><td></td><td>▲</td><td>[vWGK22]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[FWA*21]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[MGK*13]</td><td>●</td><td>●</td><td>●</td><td></td><td>▲</td><td>[WNM20]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[FCS21]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[MM04]</td><td></td><td></td><td></td><td></td><td>○</td><td>[WMW*18]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[FCM*19]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[Mor16]</td><td></td><td></td><td></td><td></td><td>■</td><td>[WMT*19]</td><td>●</td><td>●</td><td></td><td></td><td>○</td></tr><tr><td>[FTG11]</td><td></td><td></td><td></td><td></td><td></td><td>■</td><td>[MKH*18]</td><td>●</td><td>●</td><td>●</td><td></td><td>S</td><td>[XYTT21]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[FKH*20]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[NMB*16]</td><td></td><td></td><td></td><td></td><td>■</td><td>[XLX*21]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[GJB*20]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>▲</td><td>[NSM*12]</td><td>●</td><td>●</td><td>●</td><td></td><td>○</td><td>[YXW*21]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[GJK19]</td><td></td><td></td><td></td><td></td><td></td><td>▲</td><td>[NPD*21]</td><td></td><td></td><td></td><td></td><td>≈</td><td>[YPHS04]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[GK19]</td><td></td><td></td><td></td><td></td><td></td><td>≈</td><td>[NGEB15]</td><td>●</td><td>●</td><td>●</td><td></td><td>○</td><td>[YCJ16]</td><td>●</td><td>●</td><td>●</td><td></td><td>■</td></tr><tr><td>[GAB*18]</td><td></td><td></td><td></td><td></td><td></td><td>≈</td><td>[OBD*19]</td><td>●</td><td>●</td><td>●</td><td></td><td>▲</td><td>[ZPCC19]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[GMS19]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>■</td><td>[ODD*18]</td><td>●</td><td>●</td><td>●</td><td></td><td>▲</td><td>[ZW17]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[GSS*04]</td><td></td><td></td><td></td><td></td><td></td><td>○</td><td>[PLAM19]</td><td>●</td><td>●</td><td>●</td><td></td><td>≈</td><td>[ZSW*15]</td><td></td><td></td><td></td><td></td><td>■</td></tr><tr><td>[GBS*18]</td><td>●</td><td>●</td><td>●</td><td></td><td></td><td>≈</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

potential target audience is rather small and qualitative feedback may turn out to be more valuable.

Examples of well-thought quantitative evaluations can be found in works of O'Connor et al. [OBD*19; ODD*18] describing a series of studies measuring the performance of users throughout various tasks. Similarly, an interesting qualitative study was performed by the authors of StereoChem tool [SD18], evaluated with a diverse set of users, having a different level of chemistry knowledge. This allowed for better examination of the reasoning process and, in consequence, also more precise identification of drawbacks of the chosen approach.

## 7. Conclusion and Future Challenges

For this state-of-the-art report, we systematically surveyed, analyzed and classified papers that report on molecular visualization in MR

environments. We found, for example, that many developments in the field somehow relate to education. In addition, a particular feature of the papers we surveyed for this STAR is the fact that many research projects come from the domain experts themselves, rather than from the field of visualization. This observation shows that there is a dedicated need within the molecular science community for MR solutions. However, it also means that we need to communicate the research that has been carried out within the application domain to our visualization community. Furthermore, to keep this work up to date with developments in the field, we also created an online curated list of Molecular Visualization MR software, available through our GitHub repository†.

Currently, much research is carried out in domains that may pass

<footnote>† github.com/davous267/molecular-visualization-in-virtual-environments</footnote>
undetected by the visualization community, and both fields can cross-
fertilize each other. Moreover, molecular structures exhibit a high
degree of complexity that can be better understood in specialized
MR setups (e. g., the volume of pockets [CŠR*20]), requiring a
better integration of molecular visualization with MR research. Also,
as already said, the adoption of MR for molecular visualization is
driven by the widespread adoption of MR devices at a consumer
level, with the affordable hardware making it increasingly easy to
be used in domain labs in practice.

While a large amount of research has been already done, we
found that much of it targets the creation of prototypes for using MR
for viewing and interacting with molecules for a specific use case,
most commonly within drug design, structural analysis, and MD
simulations. While this is a valid first step, future research should
also focus on experiments to understand which interactions and
representations are better suited for given tasks in MR. We envision
that the established paradigms, rooted in mouse and keyboard-based
interactions, are simply not valid anymore. Moreover, there is a set
of tools, such as 2D views for charts, that are not easily ported to
MR, and the number of studies regarding the perception of such
views is small. This does not mean that we need to mimic every
desktop task or tool in an MR application. Researchers rather have
to start thinking about MR applications from scratch, take advantage
of MR, and design with MR (instead of only the desktop) in mind.

Therefore, some areas that still present important challenges are:

* *Enhanced interactions with molecular models.* 3D interaction is a particular challenge in immersive environments. Molecular visualization research cannot be done in isolation, but instead interaction design needs to be considered at the same time, taking an inspiration in the field of human-computer interaction. Often, the natural ego-centric perspective through HMDs is seen as a key advantage of MR. One of the key challenges with 6DOF controllers, however, is that are held in unsteady hands, which inevitably decreases precision when interacting with the virtual objects. As discussed in Section 4, one of the key use cases is molecular modeling and drug design, which would benefit from dedicated attention on interaction research with stable input or control configurations (e. g., [LODI16]).

* *Improved rendering times for complex molecules.* Molecular landscapes are often large-scale containing millions or even billions of atoms. Rendering imposes a particular challenge due to basic requirement of VR (90Hz refresh rate + stereoscopic rendering). Therefore, the opportunities lie in accelerating and adopting the existing rendering techniques for VR applications.

* *Incorporation of 2D charts into molecular visualization immersive applications.* The majority of the papers we surveyed focused on understanding the spatial structure and relationships. We found a few papers that have been using VR to layout and view abstract data [WNM20; PMI*21]. The sense of depth and immersive nature were mentioned as the major benefits of VR. Hence, we believe that other 2D-centric visualization could similarly benefit.

* *Evaluation of perceptual issues in Immersive Analytics.* Based on the surveyed papers, we conclude that the process of validation of outcomes is still in a rather early stage and some standards for evaluation of MR molecular approaches might be necessary. Ultimately, we need to have a better understanding of perceptual issues that may arise when large-scale molecular landscapes are

viewed. While a large body of research in VR perception ex-
ists in general, the question is how applicable it is to molecular
visualization.

The papers we found are also distributed along the MR continuum,
but with a larger portion of them lying on the side of the VR end.
This may be attributed to the fact that molecular structures already
exist on a different scale, hence having no connection to the world
the human eye perceives. The AR work we found mostly focused
on augmenting molecular visualizations for educational purposes
that use markers such as QR codes.

Despite its advantages, MR devices are not free from issues. For example, HMDs can isolate the users, especially those that are entirely blocking out the view of the real world. This creates several challenges that make simple tasks, such as taking notes, inputting text, or even drinking coffee, quite cumbersome. Fully virtual settings, however, also have advantages since remote users can connect to such environments, as we commented in Section 5.2. Consequently, this is an area that needs to be explored further.

We also want to point out possible further extensions that have
not yet received much attention. One example is making deeper use
of touch. By using haptic devices in VR or physicalization in AR,
user experiences can be improved and become more engaging by
increasing immersion. Other researchers also suggested the use of
sonification, to extend the communication features [OBD*19].

Acknowledgments

This work was supported by the US DOE LLNL-LDRD 22-FS-026 and was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 (LLNL-CONF-831245) and by projects TIN2017-88515-C2-1-R(GEN3DLIVE), from the Spanish Ministerio de Economía y Competitividad, by 839 FEDER (EU) funds, and by Ministerio de Ciencia e Innovación/AEI (PID2021-122136OB-C21 by 10.13039/501100011033/FEDER, UE). The work received further support by the French National Research agency (Grant "PIRATE", ANR-21-CE45-0014) and by the "Initiative d'Excellence" program from the French State (Grant "DY-NAMO", ANR-11-LABX-0011 and grant "CACSICE", ANR-11-EQPX-0008), and from Carl Zeiss Stiftung.

Figures and Tables License Copyright

With the exception of those images from external authors whose
licenses/copyrights we have noted in the respective figure captions
(labeled as *Permission for re-use obtained*), we as authors state that
all of our own figures and data tables in this article (i.e., Table 1,
Figure 2, and Figure 3) are and remain under our own personal
copyright, with the permission to be used here. We also make them
available under the [Creative Commons Attribution 4.0 International
](https://creativecommons.org/licenses/by/4.0/) (©① CC BY 4.0) license and share them at [osf.io/5yjku](https://osf.io/5yjku).

References

[Abr20] ABRIATA, L. A.: Building blocks for commodity augmented reality-based molecular visualization and modeling in web browsers. *PeerJ Comput Sci* 6 (2020), art. no. e260, 23 pages. DOI: 10.7717/peerj-cs.260 6, 7, 10, 13, 16 ⚙ ⚪ 📥.

<figure><img src="image_13.png" /></figure>
[ABT*20] AW, J. K., BOELLAARD, K. C., TAN, T. K., YAP, J., LOH, Y. P., COLASSON, B., BLANC, E., LAM, Y., and FUNG, F. M.: Interacting with three-dimensional molecular structures using an augmented reality mobile app. *J Chem Educ* 97, 10 (2020), 3877–3881. DOI: 10.1021/acs.jchemed.0c00387 6, 13, 16 🌐.

[AD20] ARGÜELLO, J. M. and DEMPSKI, R. E.: Fast, simple, student generated augmented reality approach for protein visualization in the classroom and home study. *J Chem Educ* 97, 8 (2020), 2327–2331. DOI: 10.1021/acs.jchemed.0c00323 6, 16 🌐.

[AEFQ96] AKKIRAJU, N., EDELSBRUNNER, H., FU, P., and QIAN, J.: Viewing geometric protein structures from inside a CAVE. *IEEE CG & A* 16, 4 (1996), 58–61. DOI: 10.1109/38.511855 1.

[AF98] AI, Z. and FRÖHLICH, T.: Molecular dynamics simulation in virtual environments. *Comput Graph Forum* 17, 3 (1998), 267–273. DOI: 10.1111/1467-8659.00273 3, 8, 16.

[AGM*18] AOUAA, N., GHERBI, R., MEZIANE, A., HADJAR, H., and SETITRA, I.: Visual data mining by virtual reality for protein-protein interaction networks. In: *Proc. AICCSA (2018)*. IEEE, Piscataway, 8 pages. DOI: 10.1109/AICCSA.2018.8612849 13, 16 🌐.

[AHRA22] ADRIANA CÁRDENAS-ROBLEDO, L., HERNÁNDEZ-URIBE, Ó., RETA, C., and ANTONIO CANTORAL-CEBALLOS, J.: Extended reality applications in industry 4.0. – A systematic literature review. *Telem Inf* 73 (2022), art. no. 101863, 39 pages. DOI: 10.1016/j.tele.2022.101863 2.

[APM13] ALSAYEGH, R., PARAMONOV, L., and MAKATSORIS, C.: A novel virtual environment for molecular system design. In: *Proc. CIVEMSA (2013)*. IEEE, Piscataway, 37–42. DOI: 10.1109/CIVEMSA.2013.6617392 10, 16 🌐.

[ASL*23] ALHARBI, R., STRNAD, O., LUIDOLT, L. R., WALDNER, M., KOU'RIL, D., BOHAK, C., KLEIN, T., GRÖLLER, E., and VIOLA, I.: Nanotilus: Generator of immersive guided-tours in crowded 3D environments. *IEEE Trans Vis Comput Graph* (2023). To appear. DOI: 10.1109/TVCG.2021.3133592 8, 10.

[AW99] ANDERSON, A. and WENG, Z.: VRDD: Applying virtual reality visualization to protein docking and design. *J Mol Graph Modell* 17, 3–4 (1999), 180–186. DOI: 10.1016/S1093-3263(99)00029-7 7, 8, 16.

[BBMK21] BRŮŽA, V., BYŠKA, J., MIČAN, J., and KOZLÍKOVÁ, B.: VRdeo: Creating engaging educational material for asynchronous student-teacher exchange using virtual reality. *Comput Graph* 98 (2021), 280–292. DOI: 10.1016/j.cag.2021.06.009 5, 6, 8, 13, 14, 16 🌐.

[BCBM18] BILLINGHURST, M., CORDEIL, M., BEZERIANOS, A., and MARGOLIS, T.: Collaborative immersive analytics. In: *Immersive Analytics*. Ed. by MARRIOTT, K., SCHREIBER, F., DWYER, T., KLEIN, K., RICHE, N. H., ITOH, T., STUERZLINGER, W., and THOMAS, B. H. Springer, Cham, 2018. Chap. 8, 221–257. DOI: 10.1007/978-3-030-01388-2 8 14.

[BF17] BORREL, A. and FOURCHES, D.: RealityConvert: A tool for preparing 3D models of biochemical structures for augmented and virtual reality. *Bioinf* 33, 23 (2017), 3816–3818. DOI: 10.1093/bioinformatics/btx485 6, 16 🌐.

[BHH20] BAI, S., HEW, K. F., and HUANG, B.: Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. *Educ Res Rev* 30 (2020), art. no. 100322, 20 pages. DOI: 10.1016/j.edurev.2020.100322 6.

[BJ14] BERRY, C. and JASON, B.: A protein in the palm of your hand through augmented reality. *Biochem Mol Biol Educ* 42, 5 (2014), 446–449. DOI: 10.1002/bmb.20805 6, 7, 16 🌐.

[BM20] BOUCHARA, T. and MONTÈS, M.: Immersive sonification of protein surface. In: *Proc. VRW (2020)*. IEEE CS, Los Alamitos, 380–383. DOI: 10.1109/VRW50115.2020.00082 16 🌐.

[BRD*19] BENNIE, S. J., RANAGHAN, K. E., DEEKS, H., GOLDSMITH, H. E., O'CONNOR, M. B., MULHOLLAND, A. J., and GLOWACKI, D. R.: Teaching enzyme catalysis using interactive molecular dynamics in virtual reality. *J Chem Educ* 96, 11 (2019), 2488–2496. DOI: 10.1021/acs.jchemed.9b00181 5, 16 🌐.

[Bry93] BRYSON, S.: Virtual reality in scientific visualization. *Comput Graph* 17, 6 (1993), 679–685. DOI: 10.1016/0097-8493(93)00117-R0.

[BTM*19] BYŠKA, J., TRAUTNER, T., MARQUES, S., DAMBORSKÝ, J., KOZLÍKOVÁ, B., and WALDNER, M.: Analysis of long molecular dynamics simulations using interactive focus+context visualization. *Comput Graph Forum* 38, 3 (2019), 441–453. DOI: 10.1111/cgf.13701 2.

[Bun] BUNGERT, C.: *HMD/headset/VR-helmet comparison chart*. Web site. Last accessed January 2022. URL: http://www.stereo3d.com/hmd.htm 2.

[BWE17] BALO, A. R., WANG, M., and ERNST, O. P.: Accessible virtual reality of biomolecular structural models using the Autodesk Molecule Viewer. *Nat Methods* 14, 12 (2017), 1122–1123. DOI: 10.1038/nmeth.4506 16 🌐.

[BWH20] BROWN, C. E., WHALEY, B., and HYSLOP, R. M.: Visualizing molecular structures and shapes: A comparison of virtual reality, computer simulation, and traditional modeling. *Chem Teach Int* 3, 1 (2020), 69–80. DOI: 10.1515/cti-2019-0009 1, 5, 16 🌐.

[CDA22] CORTÉS RODRÍGUEZ, F., DAL PERARO, M., and ABRIATA, L. A.: Online tools to easily build virtual molecular models for display in augmented and virtual reality on the web. *J Mol Graphics Modell* 114 (2022), art. no. 108164, 8 pages. DOI: 10.1016/j.jmgm.2022.108164 6, 16 🌐.

[Cen12] CENTER FOR COMPUTATIONAL STRUCTURAL BIOLOGY: *Cell-PaintVR*. Web site. Last accessed Jan. 2022. 2012. URL: https://ccsb.scripps.edu/cellpaint/cellpaint-vr/ 10.

[CGGG21] CASERMAN, P., GARCIA-AGUNDEZ, A., GÁMEZ ZERBAN, A., and GÖBEL, S.: Cybersickness in current-generation virtual reality head-mounted displays: systematic review and outlook. *Virtual Reality* 25, 4 (2021), 1153–1170 3.

[CGY20] COAN, H. A., GOEHLE, G., and YOUKER, R. T.: Teaching biochemistry and molecular biology with virtual reality – Lesson creation and student response. *J Teach Learn* 14, 1 (2020), 71–92. DOI: 10.22329/jtl.v14i1.6234 5, 6, 11, 13, 16 🌐.

[CKK06] CARINI, R. M., KUH, G. D., and KLEIN, S. P.: Student engagement and student learning: Testing the linkages*. *Res Higher Educ* 47, 1 (2006), 1–32. DOI: 10.1007/s11162-005-8150-9 5.

[CLK*11] CHAVENT, M., LÉVY, B., KRONE, M., BIDMON, K., NOMINÉ, J.-P., ERTL, T., and BAADEN, M.: GPU-powered tools boost molecular visualization. *Briefings Bioinf* 12, 6 (2011), 689–701. DOI: 10.1093/bib/bbq089 2.

[CPG20] CALVELO, M., PIÑEIRO, Á., and GARCIA-FANDINO, R.: An immersive journey to the molecular structure of SARS-CoV-2: Virtual reality in COVID-19. *Comput Struct Biotechnol J* 18 (2020), 2621–2628. DOI: 10.1016/j.csbj.2020.09.018 1.

[CPS*97] CZERNUSZENKO, M., PAPE, D., SANDIN, D., DEFANTI, T., DAWE, G. L., and BROWN, M. D.: The ImmersaDesk and Infinity Wall projection-based virtual reality displays. *ACM SIGGRAPH Comput Graph* 31, 2 (1997), 46–49. DOI: 10.1145/271283.271303 2.

[CPWG20] CHAN, K.-F., POH, J.-J., WU, W.-L., and GAN, S. K.-E.: Augmented reality in scientific visualization and communications: A new dawn of looking at antibody interactions. *Antibody Ther* 3, 3 (2020), 221–226. DOI: 10.1093/abt/tbaa021 8, 16 🌐.

[CSD*92] CRUZ-NEIRA, C., SANDIN, D. J., DEFANTI, T. A., KENYON, R. V., and HART, J. C.: The CAVE: Audio visual experience automatic virtual environment. *Commun ACM* 35, 6 (1992), 64–72. DOI: 10.1145/129888.129892 1, 2.

[ČSR*20] CASSIDY, K. C., ŠEFČÍK, J., RAGHAV, Y., CHANG, A., and DURRANT, J. D.: ProteinVR: Web-based molecular visualization in virtual reality. *PLoS Comput Biol* 16, 3 (2020), art. no. e1007747, 17 pages. DOI: 10.1371/journal.pcbi.1007747 0, 1, 6–8, 11, 16, 17 🌐.
[CYB*05] CHASTINE, J., YING ZHU, BROOKS, J., OWEN, G., HARRISON, R., and WEBER, I.: A collaborative multi-view virtual environment for molecular visualization and modeling. In: *Proc. CMV (2005)*. IEEE, Piscataway, 77–84. DOI: 10.1109/CMV.2005.111, 12, 14, 16 ⚙️.

[dCN17] Da COSTA, L. A. L. and NEDEL, L. P.: An immersive visualization study on molecules manipulation. In: *Proc. SVR (2017)*. IEEE, Piscataway, 169–177. DOI: 10.1109/SVR.2017.30116, 16 ⚙️.

[DDG*20] DOAK, D. G., DENYER, G. S., GERRARD, J. A., MACKAY, J. P., and ALLISON, J. R.: Peppy: A virtual reality environment for exploring the principles of polypeptide structure. *Protein Sci* 29, 1 (2020), 157–168. DOI: 10.1002/pro.3752 4–6, 11, 13, 16 ⚙️.

[DGC*15] DOUTRELIGNE, S., GAGEAT, C., CRAGNOLINI, T., TALY, A., PASQUALI, S., DERREUMAUX, P., and BAADEN, M.: UnityMol: Interactive and ludic visual manipulation of coarse-grained RNA and other biomolecules. In: *Proc. VARMS (2015)*. IEEE, Piscataway, 1–6. DOI: 10.1109/VARMS.2015.7151718 11, 16 ⚙️.

[DMI*18] DWYER, T., MARRIOTT, K., ISENBERG, T., KLEIN, K., RICHE, N., SCHREIBER, F., STUERZLINGER, W., and THOMAS, B.: Immersive analytics: An introduction. In: *Immersive Analytics. Ed. by MARRIOTT, K., SCHREIBER, F., DWYER, T., KLEIN, K., RICHE, N. H., ITOH, T., STUERZLINGER, W., and THOMAS, B. H.* Springer, Cham, 2018. Chap. 1, 1–23. DOI: 10.1007/978-3-030-01388-2 1 0, 1 ⚙️.

[DPSR98] DREES, R. C., PLEISS, J., SCHMID, R. D., and ROLLER, D.: Integrating molecular modeling tools and virtual reality engines: An architecture for a highly immersive molecular modeling (HIMM) environment. In: *Proc. CGI (1998)*. IEEE CS, Los Alamitos, 391–392. DOI: 10.1109/CGI.1998.694289 16.

[DSJ19] DAVISON, T., SAMAVATI, F., and JACOB, C.: LifeBrush: Painting, simulating, and visualizing dense biomolecular environments. *Comput Graph* 82 (2019), 232–242. DOI: 10.1016/j.cag.2019.05.006 7, 10, 13, 16.

[DSKG06] DÜNSER, A., STEINBÜGL, K., KAUFMANN, H., and GLÜCK, J.: Virtual and augmented reality as spatial ability training tools. In: *Proc. CHINZ (2006)*. ACM, New York, 125–132. DOI: 10.1145/1152760.1152776 5.

[dSNGF17] Dos SANTOS BAGLIE, L. S., NETO, M. P., GUIMARÃES, M. D. P., and FERREIRA BREGA, J. R.: Distributed, immersive and multi-platform molecular visualization for chemistry learning. In: *Proc. ICCSA (2017)*. Springer, Cham, 569–584. DOI: 10.1007/978-3-319-62392-4 41 6, 16 ⚙️.

[DWB*20] DEEKS, H. M., WALTERS, R. K., BARNOUD, J., GLOWACKI, D. R., and MULHOLLAND, A. J.: Interactive molecular dynamics in virtual reality is an effective tool for flexible substrate and inhibitor docking to the SARS-CoV-2 main protease. *J Chem Inf Model* 60, 12 (2020), 5803–5814. DOI: 10.1021/acs.jcim.0c01030 12, 16 ⚙️.

[DWH*20] DEEKS, H. M., WALTERS, R. K., HARE, S. R., O’CONNOR, M. B., MULHOLLAND, A. J., and GLOWACKI, D. R.: Interactive molecular dynamics in virtual reality for accurate flexible protein-ligand docking. *PLoS ONE* 15, 3 (2020), art. no. e0228461, 21 pages. DOI: 10.1371/journal.pone.0228461 7, 8, 14, 16 ⚙️.

[DY22] DINCELLI, E. and YAYLA, A.: Immersive virtual reality in the age of the metaverse: A hybrid-narrative review based on the technology affordance perspective. *J Strategic Inf Syst* 31, 2 (2022), art. no. 101717, 22 pages. DOI: 10.1016/j.jsis.2022.101717 2.

[EDC*19] EL BEHEIRY, M., DOUTRELIGNE, S., CAPORAL, C., OTERTAG, C., DAHAN, M., and MASSON, J.-B.: Virtual reality: Beyond visualization. *J Mol Biol* 431, 7 (2019), 1315–1321. DOI: 10.1016/j.jmb.2019.01.033 16.

[ENP20] ERIKSEN, K., NIELSEN, B. E., and PITTELKOW, M.: Visualizing 3D molecular structures using an augmented reality app. *J Chem Educ* 97, 5 (2020), 1487–1490. DOI: 10.1021/acs.jchemed.9b01033 7, 16 ⚙️.

[FCM*19] FERRELL, J. B., CAMPBELL, J. P., MCCARTHY, D. R., MCKAY, K. T., HENSINGER, M., SRINIVASAN, R., ZHAO, X., WURTHMANN, A., LI, J., and SCHNEEBELI, S. T.: Chemical exploration with virtual reality in organic teaching laboratories. *J Chem Educ* 96, 9 (2019), 1961–1966. DOI: 10.1021/acs.jchemed.9b00036 5, 6, 16 ⚙️.

[FCS21] FERNANDES, H. S., CERQUEIRA, N. M. F. S. A., and SOUSA, S. F.: Developing and using BioSIM(AR), an augmented reality program to visualize and learn about chemical structures in a virtual environment on any internet-connected device. *J Chem Educ* 98, 5 (2021), 1789–1794. DOI: 10.1021/acs.jchemed.0c01317 6, 7, 16 ⚙️.

[FCW19] FRUTOS-PASCUAL, M., CREED, C., and WILLIAMS, I.: Head mounted display interaction evaluation: manipulating virtual objects in augmented reality. In: *Proc. INTERACT (2019)*. Springer International, Cham, 287–308. DOI: 10.1007/978-3-030-29390-1 16 10.

[FFV22] FOMBONA-PASCUAL, A., FOMBONA, J., and VAZQUEZ-CANO, E.: VR in chemistry, a review of scientific research on advanced atomic/molecular visualization. *Chem. Educ. Res. Pract* 23, 2 (2022), 300–312. DOI: 10.1039/d1rp00317h 1.

[FKH*20] FUJIWARA, D., KELLAR, K., HUMER, I., PIETROSZEK, K., and ECKHARDT, C.: VSEPR theory, an interactive and immersive virtual reality. In: *Proc. ILRN (2020)*. IEEE, Piscataway, 140–146. DOI: 10.23919/iLRN47897.2020.9155185 5, 6, 12, 16 ⚙️.

[FP21] FONNET, A. and PRIÉ, Y.: Survey of immersive analytics. *IEEE Trans Vis Comput Graph* 27, 3 (2021), 2101–2122. DOI: 10.1109/TVCG.2019.2929033 0, 1.

[FSTdP10] FRANCISCHETTI-CORRÉA, M., SCHPECTOR, J. Z., TREVELIN, L. C., and de PAIVA GUIMARÃES, M.: Immersive environment for molecular visualization to interaction between research groups geographically dispersed. In: *Proc. ISABEL (2010)*. IEEE, Piscataway, 5 pages. DOI: 10.1109/ISABEL.2010.5702889 14, 16 ⚙️.

[FTG11] FRANCISCHETTI-CORRÉA, M., TREVELIN, L. C., and GUIMARÃES, M. D. P.: Molecular visualization with supports of interaction, immersion and collaboration among geographically-separated research groups. In: *Proc. CENTERIS*. Springer, Berlin, 2011, 128–135. DOI: 10.1007/978-3-642-24352-3 14 14, 16 ⚙️.

[FWA*21] FALAH, J., WEDYAN, M., ALFALAH, S. F. M., ABUTARBOUSH, M., AL-JAKHEEM, A., AL-FARANEH, M., ABUHAMMAD, A., and CHARISSIS, V.: Identifying the characteristics of virtual reality gamification for complex educational topics. *Multimodal Technol Interact* 5, 9 (2021), art. no. 53, 27 pages. DOI: 10.3390/mti5090053 6, 16 ⚙️.

[FZG11] FLIKKEMA, E., ZHOU, Z., and GREAVES, G. N.: Using visualization techniques and molecular dynamics to study atoms diffusing in glass. In: *Eurographics Posters (2011)*. Eurographics Assoc, Goslar, 47–48. DOI: 10.2312/EG2011/posters/047–048 12.

[GAB*18] GARDNER, A., AUTIN, L., BARBARO, B., OLSON, A. J., and GOODSELL, D. S.: CellPAINT: Interactive illustration of dynamic mesoscale cellular environments. *IEEE Comput Graph Appl* 38, 6 (2018), 51–66. DOI: 10.1109/MCG.2018.2877076 7, 10, 13, 16.

[Gam] GAMES, E.: *The most powerful real-time 3D creation tool – Unreal Engine*. URL: https://www.unrealengine.com/en-US (visited on 08/24/2022) 3.

[GBR*19] GROSS, A., BECHER, M., REINA, G., ERTL, T., and KRONE, M.: A user interaction design for object manipulation via eye tracking in virtual reality. In: *Proc. NIDIT (2019)*. IEEE CS, Los Alamitos, 1699–1704 3, 16 ⚙️.

[GBS*18] GODDARD, T. D., BRILLIANT, A. A., SKILLMAN, T. L., VERGENZ, S., TYRWHITT-DRAKE, J., MENG, E. C., and FERRIN, T. E.: Molecular visualization on the Holodeck. *J Mol Biol* 430, 21 (2018), 3982–3996. DOI: 10.1016/j.jmb.2018.06.040 1–4, 7, 8, 10–14, 16 ⚙️.

[GHM*18] GODDARD, T. D., HUANG, C. C., MENG, E. C., PETTERSEN, E. F., COUCH, G. S., MORRIS, J. H., and FERRIN, T. E.: UCSF ChimeraX: Meeting modern challenges in visualization and analysis. *Protein Sci* 27, 1 (2018), 14–25. DOI: 10.1002/pro.3235 8, 12, 16 ⚙️.
[GJB*20] GANDHI, H. A., JAKYMIW, S., BARRETT, R., MAHASETH, H., and WHITE, A. D.: Real-time interactive simulation and visualization of organic molecules. *J Chem Educ* 97, 11 (2020), 4189–4195. DOI: 10.1021/acs.jchem.ed.9b01161 0, 12 🌐.

[GJK19] GARCIA-BONETE, M.-J., JENSEN, M., and KATONA, G.: A practical guide to developing virtual and augmented reality exercises for teaching structural biology: VR and AR for teaching structural biology. *Biochem Mol Biol Educ* 47, 1 (2019), 16–24. DOI: 10.1002/bmb. 21188 7, 16 🌐.

[GK19] GARCÍA-HERNÁNDEZ, R. J. and KRANZLMÜLLER, D.: NOMAD VR: Multiplatform virtual reality viewer for chemistry simulations. *Com- put Phys Commun* 237 (2019), 230–237. DOI: 10.1016/j.cpc.2018. 11.013 2, 8, 9, 16.

[GMS19] GAUTHIER, J.-M., McGrath, I., and SURESH PATEL, J.: Dy- namic virtual proteins: Visualization, interaction and collaboration in virtual reality. In: *Proc. VRST (2019)*. ACM, New York, art. no. 107, 2 pages. DOI: 10.1145/3359996.3365050 15, 16 🌐.

[GNE*16] GREBNER, C., NORRBY, M., ENSTRÖM, J., NILSSON, I., HOGNER, A., HENRIKSSON, J., WESTIN, J., FARAMARZI, F., WERNER, P., and BOSTRÖM, J.: 3D-Lab: A collaborative web-based platform for molecular modeling. *Future Med Chem* 8, 14 (2016), 1739–1752. DOI: 10.4155/fmc-2016-0081 9, 14, 16 🌐.

[GO16] GARDNER, A. and OLSON, A.: 3D printing of molecular models. *J Biocommun* 40, 1 (2016), 15–21. DOI: 10.5210/jbc.v40i1.6626 2.

[Gra18] GRAČANIN, D.: Immersion versus embodiment: Embodied cogni- tion for immersive analytics in mixed reality environments. In: *Proc. AC* (2018). Springer, Cham, 355–368. DOI: 10.1007/978-3-319-91470- 1_29 3.

[GSS*04] GILLET, A., SANNER, M., STOFFLER, D., GOODSELL, D., and OLSON, A.: Augmented reality with tangible auto-fabricated models for molecular biology applications. In: *Proc. Visualization (2004)*. IEEE CS, Los Alamitos, 235–241. DOI: 10.1109/VISUAL.2004.7 16 🌐.

[Haa96] HAASE, H.: Symbiosis of virtual reality and scientific visualization system. *Comput Graph Forum* 15, 3 (1996), 443–451. DOI: 10.1111/ 1467-8659.1530443 0.

[HDS96] HUMPHREY, W., DALKE, A., and SCHULTEN, K.: VMD: Visual molecular dynamics. *J Mol Graph* 14, 1 (1996), 33–38. DOI: 10.1016/ 0263-7855(96)00018-5 2.

[HKM*19] HORTON, B. K., KALIA, R. K., MOEN, E., NAKANO, A., No- MURA, K.-I., QIAN, M., VASHISHTA, P., and HAFREAGER, A.: Game- engine-assisted research platform for scientific computing (GEARS) in virtual reality. *SoftwareX* 9 (2019), 112–116. DOI: 10.1016/j.softx. 2019.01.009 12.

[HL00] HUIITEMA, H. and LIERE, R. V.: Time critical computing and rendering of molecular surfaces using a zonal map. In: *Proc. EGVE (2000)*. Eurographics Assoc, Goslar, 115–124. DOI: 10.2312/EGVE/ EGVE00/115-124 2, 16.

[HLC*21] HU, M., LUO, X., CHEN, J., LEE, Y. C., ZHOU, Y., and WU, D.: Virtual reality: A survey of enabling technologies and its applications in IoT. *J Network Comput Appl* 178 (2021), art. no. 102970, 18 pages. DOI: 10.1016/j.jnca.2020.102970 2.

[HMK*20] HALLADJIAN, S., MIAO, H., KOUŘIL, D., GRÖLLER, M. E., VIOLA, I., and ISENBERG, T.: ScaleTrotter: Illustrative visual travels across negative scales. *IEEE Trans Vis Comput Graph* 26, 1 (2020), 654– 664. DOI: 10.1109/TVCG.2019.2934334 10, 11.

[HMO19] HAGITA, K., MATSUMOTO, S., and OTA, K.: Study of com- modity VR for computational material sciences. *ACS Omega* 4, 2 (2019), 3990–3999. DOI: 10.1021/acsomega.8b03483 9, 16 🌐.

[HP17] HOFFMAN, M. A. and PROVANCE, J. B.: Visualization of molecu- lar structures using HoloLens-based augmented reality. *AMIA Jt Summits Transl Sci Proc.* 2017 (2017), 68–74. URL: https://pubmed.ncbi. nlm.nih.gov/28815109/ 2, 16 🌐.

[HR*95] HOWARD, I. P., ROGERS, B. J., et al.: *Binocular vision and stereopsis*. Oxford University Press, USA, 1995 3.

[Ihl97] IHLENFELDT, W.-D.: Virtual reality in chemistry. *J Mol Model* 3, 9 (1997), 386–402. DOI: 10.1007/s008940050056 2.

[IIC*13] ISENBERG, T., ISENBERG, P., CHEN, J., SEDLMAIR, M., and MÖLLER, T.: A systematic review on the practice of evaluating visual- ization. *IEEE Trans Vis Comput Graph* 19, 12 (2013), 2818–2827. DOI: 10.1109/TVCG.2013.126 15.

[JJS05] JONES, L. L., JORDAN, K. D., and STILLINGS, N. A.: Molecular visualization in chemistry education: The role of multidisciplinary col- laboration. *Chem Educ Res Pract* 6, 3 (2005), 136–149. DOI: 10.1039/ B5RP90005K 5, 16 🌐.

[JOB*20] JAMIESON-BINNIE, A. D., O’CONNOR, M. B., BARNOUD, J., WONNACOTT, M. D., BENNIE, S. J., and GLOWACKI, D. R.: Narupa iMD: A VR-enabled multiplayer framework for streaming interactive molecular simulations. In: *ACM SIGGRAPH Immersive Pavilion (2020)*. ACM, New York, art. no. 13, 2 pages. DOI: 10.1145/3388536. 3407891 14, 16 🌐.

[JRA*18] JOHNSTON, A. P., RAE, J., ARIOTTI, N., BAILEY, B., LILJA, A., WEBB, R., FERGUSON, C., MAHER, S., DAVIS, T. P., WEBB, R. I., McGHEE, J., and PARTON, R. G.: Journey to the centre of the cell: Virtual reality immersion into scientific data. *Traffic* 19, 2 (2018), 105– 110. DOI: 10.1111/tra.12538 5, 6, 10, 16 🌐.

[JTO*20] JUÁREZ-JIMÉNEZ, J., TEW, P., O’CONNOR, M., LLABRÉS, S., SAGE, R., GLOWACKI, D., and MICHEL, J.: Combining virtual reality visualization with ensemble molecular dynamics to study complex protein conformational changes. *J Chem Inf Model* 60, 12 (2020), 6344–6354. DOI: 10.1021/acs.jcim.0c00221 12, 16 🌐.

[KAK*18] KLEIN, T., AUTIN, L., KOZLIKOVÁ, B., GOODSELL, D., OL- SON, A., GRÖLLER, M. E., and VIOLA, I.: Instant construction and visu- alization of crowded biological environments. *IEEE Trans Vis Comput Graph* 24, 1 (2018), 862–872. DOI: 10.1109/TVCG.2017.2744258 10.

[Kaz20] KAZATZIS, K.: Molecular Visualization and 2D Interaction in Vir- tual Reality. MA thesis. Spain: Universitat Politècnica de Catalunya, 2020. URL: https://upcommon s.upc.edu/handle/2117/336962 11, 16 🌐.

[KBL*19] KINGSLEY, L. J., BRUNET, V., LELAIS, G., MCCLOSKEY, S., MILLIKEN, K., LEIJA, E., FUHS, S. R., WANG, K., ZHOU, E., and SPRAGGON, G.: Development of a virtual reality platform for effective communication of structural data in drug discovery. *J Mol Graph Modell* 89 (2019), 234–241. DOI: 10.1016/j.jmgm.2019.03.010 5, 7, 11, 14–16 🌐.

[KDSH22] KAUFELD, M., DE CONINCK, K., SCHMIDT, J., and HECHT, H.: Chewing gum reduces visually induced motion sickness. *Exp Brain Res* 240, 2 (2022), 651–663. DOI: 10.1007/s00221-021-06303- 5 3 🌐.

[KFS*17] KRONE, M., FRIESS, F., SCHARNOWSKI, K., REINA, G., Fademrecht, S., Kulschevski, T., Pleiss, J., and Ertl, T.: Molec- ular surface maps. *IEEE Trans Vis Comput Graph* 23, 1 (2017), 701–710. DOI: 10.1109/TVCG.2016.2598824 2.

[KHPB02] KOUTEK, M., HEES, J. V., POST, F. H., and BAKKER, A. F.: Virtual spring manipulators for particle steering in molecular dynamics on the ResponsiveWorkbench. In: *Proc. EGVE (2002)*. Eurographics Assoc, Goslar, 53–62. DOI: 10.2312/EGVE/EGVE02/053-062 9, 13, 16 🌐.

[KIK*21] KOUŘIL, D., ISENBERG, T., KOZLÍKOVÁ, B., MEYER, M., GRÖLLER, M. E., and VIOLA, I.: HyperLabels—Browsing of dense and hierarchical molecular 3D models. *IEEE Trans Vis Comput Graph* 27, 8 (2021), 3493–3504. DOI: 10.1109/TVCG.2020.2975583 10.

[KKF*17] KOZLÍKOVÁ, B., KRONE, M., FALK, M., LINDOW, N., BAADEN, M., BAUM, D., VIOLA, I., PARULEK, J., and HEGE, H.-C.: Visualization of biomolecular structures: State of the art revisited. *Comput Graph Forum* 36, 8 (2017), 178–204. DOI: 10.1111/cgf.13072 1, 2.

[KKL*15] KOZLÍKOVÁ, B., KRONE, M., LINDOW, N., FALK, M., BAADEN, M., BAUM, D., VIOLA, I., PARULEK, J., and HEGE, H.-C.: Visualization of biomolecular structures: State of the art. In: *EuroVis – STARs (2015)*. Eurographics Assoc, Goslar, 61–81. DOI: 10.2312/ eurovisstar.20151112 1, 2.
[KKL*16] KRONE, M., KOZLÍKOVÁ, B., LINDOW, N., BAADEN, M., BAUM, D., PARULEK, J., HEGE, H.-C., and VIOLA, I.: Visual analysis of biomolecular cavities: State of the art. *Comput Graph Forum* 35, 3 (2016), 527–551. DOI: 10.1111/cgf.12928 2.

[KPB22] KRÜGER, J. M., PALZER, K., and BODEMER, D.: Learning with augmented reality: Impact of dimensionality and spatial abilities. *Comput Educ Open* 3 (2022), art. no. 100065, 20 pages. DOI: 10.1016/j.caeo.2021.100065 5.

[KPL*04] KIM, J.-I., PARK, S.-J., LEE, J., CHOI, Y.-J., and JUNG, S.-H.: Development of a gesture-based molecular visualization tool based on virtual reality for molecular docking. *Bull Korean Chem Soc* 25, 10 (2004), 1571–1574. DOI: 10.5012/BKCS.2004.25.10.1571 3, 11, 16 🟧.

[KSB*22] KUŤÁK, D., SELZER, M. N., BYŠKA, J., GANUZA, M. L., BARIŠIĆ, I., KOZLÍKOVÁ, B., and MIAO, H.: Vivern – A virtual environment for multiscale visualization and modeling of DNA nanostructures. *IEEE Trans Vis Comput Graph* 28, 12 (2022), 4825–4838. DOI: 10.1109/TVCG.2021.3106328 7–9, 11–13, 16 🟧.

[LBI*12] LAM, H., BERTINI, E., ISENBERG, P., PLAISANT, C., and CARPENDALE, S.: Empirical studies in information visualization: seven scenarios. *IEEE Trans Vis Comput Graph* 18, 9 (2012), 1520–1536. DOI: 10.1109/TVCG.2011.279 15.

[LBO*20] LAUREANTI, J., BRANDI, J., OFFOR, E., ENGEL, D., RALLO, R., GINOVSKA, B., MARTINEZ, X., BAADEN, M., and BAKER, N. A.: Visualizing biomolecular electrostatics in virtual reality with UnityMOL-APBS. *Protein Sci* 29, 1 (2020), 237–246. DOI: 10.1002/pro.3773 8, 12, 16 🟧.

[LBZ*21] LEE, L.-H., BRAUD, T., ZHOU, P., WANG, L., XU, D., LIN, Z., KUMAR, A., BERMEJO, C., and HUI, P.: *All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda*. arXiv preprint 2110.05352. 2021. DOI: 10.48550/arXiv.2110.05352 1.

[LDP*02] LIN, J.-W., DUH, H. B.-L., PARKER, D. E., ABI-RACHED, H., and FURNESS, T. A.: Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. In: *Proc. VR (2002)*. IEEE CS, Los Alamitos, 164–171. DOI: 10.1109/VR.2002.996519 3.

[Lev66] LEVINTHAL, C.: Molecular model-building by computer. *Sci Am* 214, 6 (1966), 42–52. DOI: 10.1038/scientificamerican0666-42 1.

[LFB22] LANREZAC, A., FÉREY, N., and BAADEN, M.: Wielding the power of interactive molecular simulations. *WIREs Computational Molecular Science* 12, 4 (2022), art. no. e1594, 32 pages. DOI: 10.1002/wcms.1594 8.

[LGE*15] LEINEN, P., GREEN, M. F. B., ESAT, T., WAGNER, C., TAUTZ, F. S., and TEMIROV, R.: Virtual reality visual feedback for hand-controlled scanning probe microscopy manipulation of single molecules. *Beilstein J Nanotechnol* 6 (2015), 2148–2153. DOI: 10.3762/bjnano.6.220 9, 13, 16 🟧.

[LKF*17] LILUASHVILI, V., KALAYCI, S., FLUDER, E., WILSON, M., GABOW, A., and GÜMÜŞ, Z. H.: iCAVE: An open source tool for visualizing biomolecular networks in 3D, stereoscopic 3D and immersive 3D. *GigaSci* 6, 8 (2017), art. no. gix054, 13 pages. DOI: 10.1093/gigascience/gix054 8, 13, 16.

[LKKK11] LEE, J., KANG, L.-W., KIM, H., and KIM, J.-I.: Co-Coot: A real-time collaborative tool for bio-molecular modeling and visualization. In: *Proc. ISVRI (2011)*. IEEE CS, Los Alamitos, 281–286. DOI: 10.1109/ISVRI.2011.5759651 16 🟧.

[LLNW14] LI, H., LEUNG, K.-S., NAKANE, T., and WONG, M.-H.: Iview: An interactive WebGL visualizer for protein-ligand complex. *BMC Bioinf* 15, 1 (2014), art. no. 56, 6 pages. DOI: 10.1186/1471-2105-15-56 16 🟧.

[LLZC] LU, B. F., LIM, K. T., ZHENG, J. M., and CAI, Y. Y.: Learning molecular biology by VR playing. In: *Proc. VRCAI*. ACM, New York, 168–172. DOI: 10.1145/1044588.1044622 6, 11, 16 🟧.

[LMKT05] LIERE, R. V., MARTENS, J.-B. O. S., KOK, A. J. F., and TIENEN, M. H. A. V. v.: Interacting with molecular structures: User performance versus system complexity. In: *Proc. EGVE (2005)*. Eurographics Assoc, Goslar, 147–156. DOI: 10.2312/EGVE/IPT_EGVE2005/147-156 16 🟧.

[LNQ*19] LAU, C. W., NGUYEN, Q. V., QU, Z., SIMOFF, S., and CATCHPOOLE, D.: Immersive intelligence genomic data visualisation. In: *Proc. ACSW (2019)*. ACM, New York, art. no. 18, 10 pages. DOI: 10.1145/3290688.3290722 16 🟧.

[LODI16] LÓPEZ, D., OEHLBERG, L., DOGER, C., and ISENBERG, T.: Towards an understanding of mobile touch navigation in a stereoscopic viewing environment for 3D data exploration. *IEEE Trans Vis Comput Graph* 22, 5 (2016), 1616–1629. DOI: 10.1109/TVCG.2015.2440233 17.

[LQK*09] LEE, J., QUY, P., KIM, J., KANG, L., SEO, A., and KIM, H.: A collaborative virtual reality environment for molecular biology. In: *Proc. ISUVR (2009)*. IEEE CS, Los Alamitos, 68–71. DOI: 10.1109/ISUVR.2009.14 14, 16 🟧.

[LR19] LOVEYS, B. R. and RIGGS, K. M.: Flipping the laboratory: Improving student engagement and learning outcomes in second year science courses. *Int J Sci Educ* 41, 1 (2019), 64–79. DOI: 10.1080/09500693.2018.1533663 5.

[LSSB12] LAHA, B., SENSHARMA, K., SCHIFFBAUER, J. D., and BOWMAN, D. A.: Effects of immersion on visual analysis of volume data. *IEEE Trans Vis Comput Graph* 18, 4 (2012), 597–606. DOI: 10.1109/TVCG.2012.42 0 🟧.

[LT20] LEE, N. Y. and TUCKER-KELLOGG, G.: An accessible, open-source mobile application for macromolecular visualization using augmented reality. *Biochem Mol Biol Educ* 48, 3 (2020), 297–303. DOI: 10.1002/bmb.21335 6, 7, 16 🟧.

[LTD*13] LV, Z., TEK, A., DA SILVA, F., EMPEREUR-MOT, C., CHAVENT, M., and BAADEN, M.: Game on, science – How video game technology may help biologists tackle visualization challenges. *PloS ONE* 8, 3 (2013), art. no. e57990, 13 pages. DOI: 10.1371/journal.pone.0057990 0, 12 🟧.

[LWLW18] LIU, X.-H., WANG, T., LIN, J.-P., and WU, M.-B.: Using virtual reality for drug discovery: A promising new outlet for novel leads. *Expert Opinion on Drug Discovery* 13, 12 (2018), 1103–1114. DOI: 10.1080/17460441.2018.1546286 8 🟧.

[LXZ11] LU, Y., XU, Y., and ZHU, X.: Designing and implementing VR²E²C, a virtual reality remote education for experimental chemistry system. *J Chem Educ* 98, 8 (2021), 2720–2725. DOI: 10.1021/acs.jchemed.1c00439 5, 16 🟧.

[MB21] MARTINEZ, X. and BAADEN, M.: UnityMol prototype for FAIR sharing of molecular-visualization experiences: From pictures in the cloud to collaborative virtual reality exploration in immersive 3D environments. *Acta Crystallogr Sect D: Struct Biol* 77 (2021), 746–754. DOI: 10.1107/S2059798321002941 0, 6–8, 11–13, 15, 16 🟧.

[MBE19] MÜLLER, C., BRAUN, M., and ERTL, T.: Optimised molecular graphics on the HoloLens. In: *Proc. VR (2019)*. IEEE CS, Los Alamitos, 97–102. DOI: 10.1109/VR.2019.8798111 2, 3 🟧.

[MCH*18] MARRIOTT, K., CHEN, J., HLAWATSCH, M., ITOH, T., NACENTA, M. A., REINA, G., and STUERZLINGER, W.: Immersive analytics: Time to reconsider the value of 3D for information visualisation. In: *Immersive Analytics*. Ed. by MARRIOTT, K., SCHREIBER, F., DWYER, T., KLEIN, K., RICHE, N. H., ITOH, T., STUERZLINGER, W., and THOMAS, B. H. Springer, Cham, 2018. Chap. 2, 25–55. DOI: 10.1007/978-3-030-01388-2_2 3.

[MDS*18] MIAO, H., DE LLANO, E., SORGER, J., AHMADI, Y., KÉKIC, T., ISENBERG, T., GRÖLLER, M. E., BARIŠIĆ, I., and VIOLA, I.: Multiscale visualization and scale-adaptive modification of DNA nanostructures. *IEEE Trans Vis Comput Graph* 24, 1 (2018), 1014–1024. DOI: 10.1109/TVCG.2017.2743981 11.

[MFVB15] MARTINEZ, X., FÉREY, N., VÉZIEN, J.-M., and BOURDOT, P.: Virtual structure reconstruction and energy estimation of a peptide from a physical tangible interface. In: *Proc. VARMS (2015)*. IEEE CS, Los Alamitos, 41–42. DOI: 10.1109/VARMS.2015.7151724 16 🟧.
[MGK*13] MERCHANT, Z., GOETZ, E., KEENEY-KENNICUTT, W., CIFIUENTES, L., KWOK, O., and DAVIS, T.: Exploring 3-D virtual reality technology for spatial ability and chemistry achievement. *J Comput Assisted Learn* 29, 6 (2013), 579–590. DOI: 10.1111/jcal.12018 0, 16 ⚙️.

[MJK18] MILLAIS, P., JONES, S. L., and KELLY, R.: Exploring data in virtual reality: Comparisons with 2D data visualizations. In: *CHI EA (2018)*. ACM, New York, art. no. LBW007, 6 pages. DOI: 10.1145/3170427.3188537 1.

[MK94] MILGRAM, P. and KISHINO, F.: A taxonomy of mixed reality visual displays. *IEICE Trans Inf Syst* 77, 12 (1994), 1321–1329. URL: https://search.ieice.org/bin/summary.php?id=e77-d_12_1321 1.

[MKA*19] MARTINEZ, X., KRONE, M., ALHARBI, N., ROSE, A. S., LARAMEE, R. S., O'DONOGHUE, S., BAADEN, M., and CHAVENT, M.: Molecular graphics: Bridging structural biologists and computer scientists. *Struct* 27, 11 (2019), 1617–1623. DOI: 10.1016/j.str.2019.09.001 2.

[MKH*18] MÜLLER, C., KRONE, M., HUBER, M., BIENER, V., HERR, D., KOCH, S., REINA, G., WEISKOPF, D., and ERTL, T.: Interactive molecular graphics for augmented reality using HoloLens. *J Integr Bioinf* 15, 2 (2018), art. no. 20180005, 13 pages. DOI: 10.1515/jib-2018-0005 2, 3, 7, 8, 16 ⚙️.

[MM04] MORITZ, E. and MEYER, J.: Interactive 3D protein structure visualization using virtual reality. In: *Proc. BIBE (2004)*. IEEE CS, Los Alamitos, 503–507. DOI: 10.1109/BIBE.2004.1317384 8, 16 ⚙️.

[MMD*18] MAES, A., MARTINEZ, X., DRUART, K., LAURENT, B., GUEGANG, S., MARCHAND, C. H., LEMAIRE, S. D., and BAADEN, M.: MinOmics, an integrative and immersive tool for multi-omics analysis. *J Integr Bioinf* 15, 2 (2018), art. no. 20180006, 20 pages. DOI: 10.1515/jib-2018-0006 12, 14, 16 ⚙️.

[Mor16] MORRISON, J.: Will chemists tilt their heads for virtual reality?: *Chem Eng News* 94, 14 (2016), 22–23. URL: https://cen.acs.org/articles/94/i14/chemists-tilt-heads-virtual-reality.html 0, 2, 16 ⚙️.

[MRE13] MÜLLER, C., REINA, G., and ERTL, T.: The VVand: A two-tier system design for high-resolution stereo rendering. In: *Proc. CHI POWERWALL Workshop (2013)* 2.

[MTK*10] MAIER, P., TÖNNIS, M., KLINKER, G., RAITH, A., DREES, M., and KÜHN, F.: What do you do when two hands are not enough? Interactive selection of bonds between pairs of tangible molecules. In: *Proc. 3DUI (2010)*. IEEE CS, Los Alamitos, 83–90. DOI: 10.1109/3DUI.2010.5444716 16 ⚙️.

[NGEB15] NORRBY, M., GREBNER, C., ERIKSSON, J., and BOSTRÖM, J.: Molecular Rift: Virtual reality for drug designers. *J Chem Inf Model* 55, 11 (2015), 2475–2484. DOI: 10.1021/acs.jcim.5b00544 7, 8, 12, 16 ⚙️.

[NHG19] NUSRAT, S., HARBIG, T., and GEHLENBORG, N.: Tasks, techniques, and tools for genomic data visualization. *Comput Graph Forum* 38, 3 (2019), 781–805. DOI: 10.1111/cgf.13727 1.

[NMB*16] NAKANO, C. M., MOEN, E., BYUN, H. S., MA, H., NEWMAN, B., McDOWELL, A., WEI, T., and EL-NAGGAR, M. Y.: iBET: Immersive visualization of biological electron-transfer dynamics. *J Mol Graph Modell* 65 (2016), 94–99. DOI: 10.1016/j.jmgm.2016.02.009 8, 9, 16 ⚙️.

[NPD*21] NOIZET, M., PELTIER, V., DELEAU, H., DAUCHEZ, M., PRÉVOST, S., and JONQUET-PREVOTEAU, J.: A collaborative molecular graphics tool for knowledge dissemination with augmented reality and 3D printing. In: *Proc. MolVA (2021)*. Eurographics Assoc, Goslar, 1–5. DOI: 10.2312/molva.20211071 3, 15, 16 ⚙️.

[NSM*12] NICKELS, S., SMINIA, H., MUELLER, S. C., KOOLS, B., DEHOF, A. K., LENHOF, H.-P., and HILDEBRANDT, A.: ProteinScanAR – An augmented reality web application for high school education in biomolecular life sciences. In: *Proc. IV (2012)*. IEEE CS, Los Alamitos, 578–583. DOI: 10.1109/IV.2012.97 4–6, 16 ⚙️.

[OBD*19] O'CONNOR, M. B., BENNIE, S. J., DEEKS, H. M., JAMIESON-BINNIE, A., JONES, A. J., SHANNON, R. J., WALTERS, R., MITCHELL, T. J., MULHOLLAND, A. J., and GLOWACKI, D. R.: Interactive molecular dynamics in virtual reality from quantum chemistry to drug binding: An open-source multi-person framework. *J Chem Phys* 150, 22 (2019), art. no. 220901, 22 pages. DOI: 10.1063/1.5092590 0, 7, 9, 12, 14, 16, 17 ⚙️.

[ODD*18] O'CONNOR, M., DEEKS, H. M., DAWN, E., METATLA, O., ROUDAUT, A., SUTTON, M., THOMAS, L. M., GLOWACKI, B. R., SAGE, R., TEW, P., WONNACOTT, M., BATES, P., MULHOLLAND, A. J., and GLOWACKI, D. R.: Sampling molecular conformations and dynamics in a multiuser virtual reality framework. *Sci Adv* 4, 6 (2018), art. no. eaat2731, 10 pages. DOI: 10.1126/sciadv.aat2731 14, 16 ⚙️.

[OOKO15] OLSHANNIKOVA, E., OMETOV, A., KOUCHERYAVY, Y., and OLSSON, T.: Visualizing big data with augmented and virtual reality: Challenges and research agenda. *J Big Data* 2 (2015), art. no. 22, 27 pages. DOI: 10.1186/s40537-015-0031-2 1.

[PB20] PUZZARINI, C. and BARONE, V.: A never-ending story in the sky: The secrets of chemical evolution. *Phys Life Rev* 32 (2020), 59–94. DOI: 10.1016/j.plrev.2019.07.001 16.

[PLAM19] PATTERSON, K., LILJA, A., ARREBOLA, M., and MCGHEE, J.: Molecular genomics education through gamified cell exploration in virtual reality. In: *Proc. VRCAI (2019)*. ACM, New York, art. no. 64, 2 pages. DOI: 10.1145/3359997.3365724 5, 6, 11, 12, 16 ⚙️.

[PMI*21] PIRCH, S., MÜLLER, F., IOFINOVA, E., PAZMANDI, J., HÜTTER, C. V. R., CHIETTINI, M., SIN, C., BOZTUG, K., PODKOSOVA, I., KAUFMANN, H., and MENCHE, J.: The VRNetzer platform enables interactive network analysis in virtual reality. *Nat Commun* 12, 1 (2021), art. no. 2432, 14 pages. DOI: 10.1038/s41467-021-22570-w 7, 8, 10, 13, 16, 17 ⚙️.

[PR18] PROBST, D. and REYMOND, J.-L.: Exploring DrugBank in virtual reality chemical space. *J Chem Inf Model* 58, 9 (2018), 1731–1735. DOI: 10.1021/acs.jcim.8b00402 8, 11, 16 ⚙️.

[PTA*20] PETERSON, C. N., TAVANA, S. Z., AKINLEYE, O. P., JOHNSON, W. H., and BERKMEN, M. B.: An idea to explore: Use of augmented reality for teaching three-dimensional biomolecular structures. *Biochem Mol Biol Educ* 48, 3 (2020), 276–282. DOI: 10.1002/bmb.21341 2, 5, 6, 16 ⚙️.

[PTC] PTC: *Vuforia Engine | Create AR Apps and AR Experiences | PTC*. URL: https://www.ptc.com/en/products/vuforia/vuforia-engine (visited on 08/24/2022) 3.

[QCC21] QIN, T., COOK, M., and COURTNEY, M.: Exploring chemistry with wireless, PC-less portable virtual reality laboratories. *J Chem Educ* 98, 2 (2021), 521–529. DOI: 10.1021/acs.jchemed.0c00954 5, 16 ⚙️.

[RB20] RAMÍREZ, J. Á. and BUENO, A. M. V.: Learning organic chemistry with virtual reality. In: *Proc. ICEV (2020)*. IEEE, Piscataway, art. no. 280610, 4 pages. DOI: 10.1109/ICEV50249.2020.9289672 11, 16 ⚙️.

[RBDR18] RATAMERO, E. M., BELLINI, D., DOWSON, C. G., and RÖMER, R. A.: Touching proteins with virtual bare hands: Visualizing protein-drug complexes and their dynamics in self-made virtual reality using gaming hardware. *J Comput-Aided Mol Des* 32, 6 (2018), 703–709. DOI: 10.1007/s10822-018-0123-0 2, 7–9, 11, 16 ⚙️.

[RFK*15] RAU, B., FRIESS, F., KRONE, M., MULLER, C., and ERTL, T.: Enhancing visualization of molecular simulations using sonification. In: *Proc. VARMS (2015)*. IEEE CS, Los Alamitos, 25–30. DOI: 10.1109/VARMS.2015.7151725 16 ⚙️.

[RFK*21] RODRIGUEZ, F. C., FRATTINI, G., KRAPP, L. F., MARTINEZ-HUNG, H., MORENO, D. M., ROLDAN, M., SALOMON, J., STEMKOSKI, L., TRAEGER, S., DAL PERARO, M., and ABRIATA, L. A.: MolecuAR-web: A web site for chemistry and structural biology education through interactive augmented realityout of the box in commodity devices. *J Chem Educ* 98, 7 (2021), 2243–2255. DOI: 10.1021/acs.jchemed.1c00179 0, 6, 13, 16 ⚙️.
[RKN*13] REDA, K., KNOLL, A., NOMURA, K.-I., PAPKA, M. E., JOHN-SON, A. E., and LEIGH, J.: Visualizing large-scale atomistic simulations in ultra-resolution immersive environments. In: *Proc. LDAV* (2013). IEEE CS, Los Alamitos, 59–65. DOI: 10.1109/LDAV.2013.6675159 2, 16.

[RRG*09] RODRÍGUEZ-SOTRES, R., RODRÍGUEZ-PENAGOS, M., GONZÁLEZ-CRUZ, J., ROSALES-LEÓN, L., and MARTÍNEZ-CASTILLA, L. P.: Simulated site-directed mutations in a virtual reality environment as a powerful aid for teaching the three-dimensional structure of proteins. *Educación Química* 20, 4 (2009), 461–465. DOI: 10.1016/S0187-893X(18)30051-X 13, 16.

[RSH18] ROSTIANINGSIH, S., SETIAWAN, A., and HALIM, C. I.: Ionic and metallic bonding visualization using augmented reality. In: *Proc. TIMES-iCON* (2018). IEEE, Piscataway, 4 pages. DOI: 10.1109/TIMES-iCON.2018.8621665 6, 13, 16.

[RTH16] RAKKOLAINEN, I., TURK, M., and HÖLLERER, T.: A superwide-FOV optical design for head-mounted displays. In: *Proc. ICAT-EGVE* (2016). Eurographics Assoc, Goslar, 45–48. DOI: 10.2312/egve.20161433 16.

[SAKW02] STOLK, B., ABDOELRAHMAN, F., KONING, A., and WIELINGA, P.: Mining the human genome using virtual reality. In: *Proc. EGPGV* (2002). Eurographics Assoc, Goslar, 17–22. DOI: 10.2312/EGPGV/EGPGV02/017-022 8, 10, 13, 16.

[SBKW18] SOMMER, B., BAADEN, M., KRONE, M., and WOODS, A.: From virtual reality to immersive analytics in bioinformatics. *J Integr Bioinf* 15, 2 (2018), art. no. 20180043, 6 pages. DOI: 10.1515/jib-2018-0043 16.

[SCM18] SWAMY K L, N., CHAVAN, P. S., and MURTHY, S.: StereoChem: Augmented reality 3D molecular model visualization app for teaching and learning stereochemistry. In: *Proc. ICALT* (2018). IEEE, Piscataway, 252–256. DOI: 10.1109/ICALT.2018.00065 5, 7, 16.

[SD18] SWAMY, N. and DASGUPTA, C.: Investigating the nature of students’ reasoning in connecting molecular structures of stereoisomers with their physical properties using an AR app. In: *Proc. T4E* (2018). IEEE, Piscataway, 53–60. DOI: 10.1109/T4E.2018.00018 5, 16.

[SF21] SMITH, C. and FRIEL, C. J.: Development and use of augmented reality models to teach medicinal chemistry. *Currents in Pharmacy Teaching and Learning* 13, 8 (2021), 1010–1017. DOI: 10.1016/j.cptl.2021.06.008 5, 7, 16.

[SFP*16] SALVADORI, A., FRATE, G. D., PAGLIAI, M., MANCINI, G., and BARONE, V.: Immersive virtual reality in computational chemistry: Applications to the analysis of QM and MM data. *Int J Quantum Chem* 116, 22 (2016), 1731–1746. DOI: 10.1002/qua.25207 9, 11, 12, 16.

[SFS*21] SCHATZ, K., FRANCO MORENO, J. J., SCHÄFER, M., ROSE, A. S., FERRARIO, V., PLEISS, J., VÁZQUEZ ALCOCER, P. P., ERTL, T., and KRONE, M.: Visual analysis of large-scale protein-ligand interaction data. *Comput Graph Forum* 40, 6 (2021), 394–408. DOI: 10.1111/cgf.14386 2.

[She03] SHELTON, B. E.: How Augmented Reality Helps Students Learn Dynamic Spatial Relationships. PhD thesis. USA: University of Washington, 2003. URL: https://www.learntechlib.org/p/116999 5.

[Shn96] SHNEIDERMAN, B.: The eyes have it: A task by data type taxonomy for information visualizations. In: *Proc. VL* (1996). IEEE CS, Los Alamitos, 336–343. DOI: 10.1109/VL.1996.545307 10.

[SIS04] SCHKOLNE, S., ISHII, H., and SCHRODER, P.: Immersive design of DNA molecules with a tangible interface. In: *Proc. Visualization* (2004). IEEE CS, Los Alamitos, 227–234. DOI: 10.1109/VISUAL.2004.47 9, 16.

[SJPG18] SHARMA, L., JIN, R., PRABHAKARAN, B., and GANS, M.: LearnDNA: An interactive VR application for learning DNA structure. In: *Proc. IWICS* (2018). ACM, New York, 80–87. DOI: 10.1145/3191801.3191810 5, 6, 13, 16.

[SKE*20] SPARK, A., KITCHING, A., ESTEBAN-FERRER, D., HANDA, A., CARR, A. R., NEEDHAM, L.-M., PONJAVIC, A., SANTOS, A. M., MCCOLL, J., LETERRIER, C., DAVIS, S. J., HENRIQUES, R., and LEE, S. F.: vLUME: 3D virtual reality for single-molecule localization microscopy. *Nat Methods* 17, 11 (2020), 1097–1099. DOI: 10.1038/s41592-020-0962-1 13, 16.

[SLM*02] SHARMA, A., LIU, X. L., MILLER, P., NAKANO, A., KALIA, R. K., VASHISHTA, P., ZHAO, W., CAMPBELL, T. J., and HAAS, A.: Immersive and interactive exploration of billion-atom systems. In: *Proc. VR* (2002). IEEE CS, Los Alamitos, 217–223. DOI: 10.1109/VR.2002.996525 2, 16.

[SSS16] STONE, J. E., SHERMAN, W. R., and SCHULTEN, K.: Immersive molecular visualization with omnidirectional stereoscopic ray tracing and remote rendering. In: *Proc. ISVC* (2016). IEEE CS, Los Alamitos, 1048–1057. DOI: 10.1109/IPDPSW.2016.121 2, 7, 10, 16.

[Str20] STRASBURGER, H.: Seven myths on crowding and peripheral vision. *i-Perception* 11, 3 (2020), art. no. 2041669520913052, 46 pages. DOI: 10.1177/2041669520913052 3.

[SUS*21] SABANDO, M. V., ULBRICH, P., SELZER, M., BYŠKA, J., MIČAN, J., PONZONI, I., SOTO, A. J., GANUZA, M. L., and KOZLIKOVÁ, B.: ChemVA: Interactive visual analysis of chemical compound similarity in virtual screening. *IEEE Trans Vis Comput Graph* 27, 2 (2021), 891–901. DOI: 10.1109/TVCG.2020.3030438 2.

[Sut68] SUTHERLAND, I. E.: A head-mounted three dimensional display. In: *Proc. AFIPS* (1968). ACM, New York, 757–764. DOI: 10.1145/1476589.1476686 2.

[SVGR16] SKÁNBERG, R., VÁZQUEZ, P.-P., GUALLAR, V., and ROPINSKI, T.: Real-time molecular visualization supporting diffuse interreflections and ambient occlusion. *IEEE Trans Vis Comput Graph* 22, 1 (2016), 718–727. DOI: 10.1109/TVCG.2015.2467293 2.

[SW19] SAFADEL, P. and WHITE, D.: Facilitating molecular biology teaching by using augmented reality (AR) and protein data bank (PDB). *Techtrends* 63, 2 (2019), 188–193. DOI: 10.1007/s11528-018-0343-0 5-7, 16.

[SWF*22] SERITAN, S., WANG, X., BESANÇON, L., MCGUFFIN, M. J., and ISENBERG, T.: Collaborative work in augmented reality: A survey. *IEEE Trans Vis Comput Graph* 28, 6 (2022), 2530–2549. DOI: 10.1109/TVCG.2020.3032761 1.

[SWF*21] SERITAN, S., WANG, Y., FORD, J. E., VALENTINI, A., GOLD, T., and MARTÍNEZ, T. J.: InteraChem: Virtual reality visualizer for reactive interactive molecular dynamics. *J Chem Educ* 98, 11 (2021), 3486–3492. DOI: 10.1021/acs.jchemed.1c00654 6, 9, 12, 13, 16.

[SWL*20] SUNG, R.-J., WILSON, A. T., LO, S. M., CROWL, L. M., NARDI, J., ST. CLAIR, K., and LIU, J. M.: BiochemAR: An augmented reality educational tool for teaching macromolecular structure and function. *J Chem Educ* 97, 1 (2020), 147–153. DOI: 10.1021/acs.jchemed.8b00691 4-7, 16.

[TE21] TODD, H. and EMSLEY, P.: Development and assessment of CootVR, a virtual reality computer program for model building. *Acta Crystallogr Sect D: Struct Biol* 77, 1 (2021), 19–27. DOI: 10.1107/S2059798320013625 8, 9, 11, 16.

[Tec] TECHNOLOGIES, U.: *Unity Real-Time Development Platform* | 3D, 2D VR & AR Engine. URL: https://unity.com/ (visited on 08/24/2022) 3.

[TFBB16] TRELLET, M., FÉREY, N., BAADEN, M., and BOURDOT, P.: Interactive visual analytics of molecular data in immersive environments via a semantic definition of the content and the context. In: *Proc. IA* (2016). IEEE CS, Los Alamitos, 48–53. DOI: 10.1109/IMMERSIVE.2016.7932383 12, 16.

[TFF*18] TRELLET, M., FÉREY, N., FLOTYŃSKI, J., BAADEN, M., and BOURDOT, P.: Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data. *J Integr Bioinf* 15, 2 (2018), art. no. 20180004, 19 pages. DOI: 10.1515/jib-2018-0004 11, 12.
[TLL*11] TSE, C.-M., LI, H., LEUNG, K.-S., LEE, K.-H., and WONG, M.-H.: Interactive drug design in virtual reality. In: Proc. IV (2011). IEEE CS, Los Alamitos, 226–231. DOI: 10.1109/IV.2011.72 8, 12.

[TLN17] THEART, R. P., LOOS, B., and NIESLER, T. R.: Virtual reality assisted microscopy data visualization and colocalization analysis. BMC Bioinf 18, 2 (2017), art. no. 64, 16 pages. DOI: 10.1186/s12859-016-1446-2 12, 13, 16 🟧.

[TTM*21] TODD, S., TODD, P., MCGOWAN, S. J., HUGHES, J. R., KAKUI, Y., LEYMARIE, F. F., LATHAM, W., and TAYLOR, S.: CSynth: An interactive modelling and visualization tool for 3D chromatin structure. *Bioinf* 37, 7 (2021), 951–955. DOI: 10.1093/bioinformatics/btaa757 9, 12, 16 🟥 ⚙️.

[TW13] TAN, S. and WAUGH, R.: Use of virtual-reality in teaching and learning molecular biology. In: 3D Immersive and Interactive Learning. Ed. by CAI, Y. Springer, Singapore, 2013, 17–43. DOI: 10.1007/978-981-4021-90-6_2 16 🟧.

[VBB*04] VIDAL, F. P., BELLO, F., BRODLIE, K., JOHN, N. W., GOULD, D., PHILLIPS, R., and AVIS, N. J.: Principles and applications of medical virtual environments. In: Eurographics STARs (2004). Eurographics Assoc, Goslar, 35 pages. DOI: 10.2312/egst.20041024 1.

[VCI20] VIOLA, I., CHEN, M., and ISENBERG, T.: Visual abstraction. In: Foundations of Data Visualization. Ed. by CHEN, M., HAUSER, H., RHEINGANS, P., and SCHEUERMANN, G. Springer, Berlin/Heidelberg, 2020. Chap. 2, 15–37. DOI: 10.1007/978-3-030-34444-3_2 11.

[vDFL*00] Van DAM, A., FORSBERG, A. S., LAIDLAW, D. H., LAVIOLA, J. J., and SIMPSON, R. M.: Immersive VR for scientific visualization: A progress report. *IEEE Comput Graph Appl* 20, 6 (2000), 26–52. DOI: 10.1109/38.888006 0, 1.

[vDLS02] Van DAM, A., LAIDLAW, D. H., and SIMPSON, R. M.: Experiments in immersive virtual reality for scientific visualization. *Comput Graph* 26, 4 (2002), 535–555. DOI: 10.1016/S0097-8493(02)00113-9 1.

[vdZLBI11] Van der ZWAN, M., LUEKS, W., BEKKER, H., and ISENBERG, T.: Illustrative molecular visualization with continuous abstraction. *Comput Graph Forum* 30, 3 (2011), 683–690. DOI: 10.1111/j.1467-8659.2011.01917.x 11.

[VI18] VIOLA, I. and ISENBERG, T.: Pondering the concept of abstraction in (illustrative) visualization. *IEEE Trans Vis Comput Graph* 24, 9 (2018), 2573–2588. DOI: 10.1109/TVCG.2017.2747545 11.

[VMR*21] VENKATESAN, M., MOHAN, H., RYAN, J. R., SCHURCH, C. M., NOLAN, G. P., FRAKES, D. H., and COSKUN, A. F.: Virtual and augmented reality for biomedical applications. *Cell Rep Med* 2, 7 (2021), art. no. 100348, 13 pages. DOI: 10.1016/j.xcrm.2021.100348 1 🟧.

[vWGK22] Von WEDELSTEDT, A., GOEBEL, G., and KALIES, G.: MOEVR: A Virtual Reality Program for Performing and Visualizing Immersive Molecular Dynamics Simulations of Guest Molecules in Metal-Organic Frameworks. *Journal of Chemical Information and Modeling* 62, 5 (2022), 1154–1159. DOI: 10.1021/acs.jcim.2c00158 6, 9, 12, 16 🟧.

[WBAI22] WANG, X., BESANÇON, L., AMMI, M., and ISENBERG, T.: Understanding differences between combinations of 2D and 3D input and output devices for 3D data visualization. *Int J Hum-Comput Stud* 163 (2022), art. no. 102820, 16 pages. DOI: 10.1016/j.ijhcs.2022.102820 10.

[WBR*20] WANG, X., BESANÇON, L., ROUSSEAU, D., SERENO, M., AMMI, M., and ISENBERG, T.: Towards an understanding of augmented reality extensions for existing 3D data analysis tools. In: Proc. CHI (2020). ACM, New York, art. no. 528, 13 pages. DOI: 10.1145/3313831.3376657 10.

[WMT*19] WON, M., MOCERINO, M., TANG, K.-S., TREAGUST, D. F., and TASKER, R.: Interactive immersive virtual reality to enhance students’ visualisation of complex molecules. In: Proc. ICCE (2019). Springer, Singapore, 51–64. DOI: 10.1007/978-981-13-6998-8_4 5, 16 🟩 ⚙️.

[WMW*18] WIEBRANDS, M., MALAJCZUK, C. J., WOODS, A. J., ROHL, A. L., and MANCERA, R. L.: Molecular dynamics visualization (MDV): stereoscopic 3D display of biomolecular structure and interactions using the Unity game engine. *J Integr Bioinf* 15, 2 (2018), art. no. 20180010, 8 pages. DOI: 10.1515/jib-2018-0010 16 🟩.

[WNM20] WESTREICH, S. T., NATTESTAD, M., and MEYER, C.: BigTop: A three-dimensional virtual reality tool for GWAS visualization. BMC Bioinf 21, 1 (2020), art. no. 39, 8 pages. DOI: 10.1186/s12859-020-3373-5 7, 12, 16, 17 🟩 ⚙️.

[XHH*21] XIONG, J., HSIANG, E.-L., HE, Z., ZHAN, T., and WU, S.-T.: Augmented reality and virtual reality displays: Emerging technologies and future perspectives. *Light Sci Appl* 10, 1 (2021), art. no. 216, 30 pages. DOI: 10.1038/s41377-021-00658-8 2.

[XLX*21] XU, K., LIU, N., XU, J., GUO, C., ZHAO, L., WANG, H.-W., and ZHANG, Q. C.: VRmol: An integrative web-based virtual reality system to explore macromolecular structure. *Bioinformatics* 37, 7 (2021). Ed. by PONTY, Y., 1029–1031. DOI: 10.1093/bioinformatics/btaa696 7, 8, 12, 16 🟩 ⚙️.

[XYTT21] XU, T., YALLAPRAGADA, V. V., TANGNEY, M., and TABIRCA, S.: ProMVR – Protein multiplayer virtual reality tool. In: Proc. VRST (2021). ACM, New York, art. no. 91, 3 pages. DOI: 10.1145/3489849.3489935 15, 16 🟩 ⚙️.

[YCJ16] YUEN, D., CARTWRIGHT, S., and JACOB, C.: Eukaryo: Virtual reality simulation of a cell. In: Proc. VRIC (2016). ACM, New York, art. no. 3, 4 pages. DOI: 10.1145/2927929.2927931 11, 12, 16 🟩 ⚙️.

[YPHS04] YANLIN LUO, PING GUO, HASEGAWA, S., and SATO, M.: An interactive molecular visualization system for education in immersive multi-projection virtual environment. In: Proc. ICIG (2004). IEEE CS, Los Alamitos, 485–488. DOI: 10.1109/ICIG.2004.36 2, 16 🟩.

[YSCJ16] YUEN, D., SANTOSO, M., CARTWRIGHT, S., and JACOB, C.: Eukaryo: An AR and VR application for cell biology. *Int J Virtual Reality* 16, 1 (2016), 7–14. DOI: 10.20870/IJVR.2016.16.1.2877 11.

[YXW*21] YALLAPRAGADA, V. V. B., XU, T., WALKER, S. P., TABIRCA, S., and TANGNEY, M.: Pepblock Builder VR – An open-source tool for gaming-based bio-edutainment in interactive protein design. *Front Bioeng Biotechnol* 9 (2021), art. no. 674211, 11 pages. DOI: 10.3389/fbioe.2021.674211 6, 16 🟩 ⚙️.

[ZPCC19] ZHANG, J. F., PACIORKOWSKI, A. R., CRAIG, P. A., and CUI, F.: BioVR: A platform for virtual reality assisted biological data integration and visualization. BMC Bioinf 20, 1 (2019), art. no. 78, 10 pages. DOI: 10.1186/s12859-019-2666-z 10, 11, 13, 16 🟩 ⚙️.

[ZSW*15] ZHU, Y., SELVARAJ, S., WEBER, P., FANG, J., SCHULZE, J. P., and REN, B.: 3D chromosome rendering from Hi-C data using virtual reality. In: Proc. Visualization and Data Analysis (2015). Vol. 9397. SPIE, Bellingham, art. no. 93970Q, 8 pages. DOI: 10.1117/12.2083470 8, 10, 11, 16.

[ZW17] ZHENG, M. and WALLER, M. P.: ChemPreview: An augmented reality-based molecular interface. *J Mol Graph Modell* 73 (2017), 18–23. DOI: 10.1016/j.jmgm.2017.01.019 11, 16 🟩 ⚙️.

[ZYX*20] ZHAN, T., YIN, K., XIONG, J., HE, Z., and WU, S.-T.: Augmented reality and virtual reality displays: perspectives and challenges. *iScience* 23, 8 (2020), art. no. 101397, 13 pages. DOI: 10.1016/j.isci.2020.101397 3.