<figure><img src="image_1.png" /></figure>

HAL
open science

# Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing

Xuanyu Wang, Weizhan Zhang, Christian Sandor, Hongbo Fu

► To cite this version:

Xuanyu Wang, Weizhan Zhang, Christian Sandor, Hongbo Fu. Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing. IEEE Transactions on Visualization and Computer Graphics, 2025, 31 (9), pp.5626-5641. 10.1109/TVCG.2024.3466554. hal-05251854

HAL Id: hal-05251854

https://hal.science/hal-05251854v1

Submitted on 12 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing

Xuanyu Wang, Weizhan Zhang, Christian Sandor, and Hongbo Fu

**Abstract**—Augmented Reality (AR) teleconferencing allows spatially distributed users to interact with each other in 3D through agents in their own physical environments. Existing methods leveraging volumetric capturing and reconstruction can provide a high-fidelity experience but are often too complex and expensive for everyday use. Other solutions target mobile and effortless-to-setup teleconferencing on AR Head Mounted Displays (HMD). They directly transplant the conventional video conferencing onto an AR-HMD platform or use avatars to represent remote participants. However, they can only support either a high fidelity or a high level of co-presence. Moreover, the limited Field of View (FoV) of HMDs could further degrade users' immersive experience. To achieve a balance between fidelity and co-presence, we explore using life-size 2D video-based avatars (video avatars for short) in AR teleconferencing. Specifically, with the potential effect of FoV on users' perception of proximity, we first conducted a pilot study to explore the local-user-centered optimal placement of video avatars in small-group AR conversations. With the placement results, we then implement a proof-of-concept prototype of videoAvatar-based teleconferencing. We conduct user evaluations with our prototype to verify its effectiveness in balancing fidelity and co-presence. Following the indication in the pilot study, we further quantitatively explore the effect of FoV size on the video avatar's optimal placement through a user study involving more FoV conditions in a VR-simulated environment. We regress placement models to serve as references for computationally determining video avatar placements in such teleconferencing applications on various existing AR HMDs and future ones with bigger FoVs.

**Index Terms**—AR Teleconference, Video-Based Avatar, Field of View, Co-presence, Fidelity.

## 1 INTRODUCTION

AR teleconferencing is an essential ingredient of future teleconferencing. It symmetrically overlays agents of spatially separated users on each other's local physical environment, creating an illusion of human teleportation. With the potential to improve the experience of online interaction to get closer to physically being together, such techniques have emerged as trending topics in both industry and research.

In pursuit of a high-quality 3D experience, one group of existing solutions present impressive real-time visual-audio effects by thoroughly building media processing pipelines containing multiple RGB-D cameras, high-bandwidth networking systems, and sophisticatedly designed displays [1], [2], [3]. However, such high hardware requirements currently make these systems too expensive and complicated to set up in daily-use scenarios. The prevalence of AR HMDs in recent years brings the opportunity to make immersive experiences more accessible, which is why they are considered

the next-generation personal computing platform in place of smartphones. Therefore, it is critical to design teleconferencing applications on AR HMDs that, on the one hand, are mobile, affordable, and easy to access, and on the other hand, can truly provide an upgraded experience compared to video conferencing on PCs and mobile devices. To this end, existing solutions directly transplant video conferencing onto the AR HMD platform [4], [5] or use virtual avatars to represent the remote peers [6], [7], [8], [9], [10]. The direct transplant presents remote users' videos in mid-air. It preserves the screen-based video conference's advantage of being photorealistic, but fails to immerse the remote users into the local physical environment. Using 3D virtual avatars can create a more immersive experience well integrated with the environment, but significantly loses the fidelity of real users and tends to fall into the uncanny valley when increasing the realism of the avatar model. They can match the needs of certain task-oriented usage scenarios such as remote assisting or guidance, but are not enough for interaction-, social-, or user-oriented scenarios [11], [12], [13], [14] such as group conversations with friends, formal meetings, and everyday telepresence [9], [10], where users want the remote peers to feel both real and present.

To optimize these two objectives, we explore the use of life-size 2D video avatars (simply referred as video avatars) for AR teleconferencing applications on the HMD platform and examine whether and how they can help to achieve the balance between fidelity and co-presence. We focus on the small-group conversation scenario [15], [16], [17], [18] as a fundamental use case in AR teleconferencing. There

* Corresponding authors: Weizhan Zhang and Hongbo Fu.

* Xuanyu Wang is with the School of Computer Science and Technology, MOEKLINNS, Xi'an Jiaotong University, and the School of Creative Media, City University of Hong Kong. E-mail: xwang2247c@my.cityu.edu.hk.

* Weizhan Zhang is with the School of Computer Science and Technology, BDKE Lab, Xi'an Jiaotong University. E-mail: zhangwzh@xjtu.edu.cn.

* Christian Sandor is with Université Paris-Saclay / CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique (LISN). E-mail: chris.sandor@gmail.com.

* Hongbo Fu is with the Division of Arts and Machine Creativity, Hong Kong University of Science and Technology. E-mail: hongbofu@ust.hk.
<figure><img src="image_2.png" /><figcaption>Fig. 1. A local user (not shown in this figure) is having a small-group AR teleconference with four remote users through their video avatars. The left scene is seen from the local user's perspective, with the remote users' video avatars (denoted as "RU 1" to "RU 4") overlaid in the local physical space. It is a screenshot from our VR-simulated AR environment (in the FoV = 110° condition) with a post-added HoloLens on each RU's head to demonstrate that every user has the conversation through an AR HMD symmetrically in real-use scenarios. The right image illustrates the layout of the conversation group from the top-down view. The local user adjusts the RUs' placement on the circle using the Radian and Radius parameters.</figcaption></figure>

<figure><img src="image_3.png" /></figure>

are two types of video-based avatars: namely, 3D video avatars and 2D video avatars, in the literature. Volumetric 3D video avatars with both textures and geometries often require rather complex setups [2], [19], [20], [21]. In contrast, a 2D video avatar is a simpler format consisting of only pixels on a plane, which is also explored by previous works [13], [16], [22], [23], [24] and is used in our work. However, they rarely explore the HMD-based AR small-group conversation scenario with multiple remote users as life-size video avatars. In this scenario, although qualitatively predictable, quantifying proxemics in the immersive interaction with video avatars is important. Literature has found certain formations [25], [26], [27] and personal proxemics in both real [28], [29] and virtual [30], [31] worlds for one-to-one and one-to-many conversations. However, despite its importance, computationally determining the placement of video avatars in AR telepresence is unexplored. Moreover, users might perceive proxemics differently due to a potentially undermined immersive experience in the HMD caused by the limited FoV. This leads to our first question: **what is the user-centered optimal placement for life-size video avatars in an HMD-based limit-FoV AR small-group teleconference (RQ1)?** To answer this question, we conduct a pilot study in Sec. 3 to determine the video avatar's optimal placement regarding the local user's comfortableness and sense of co-presence. The results yield references for the video avatar placement in our subsequent evaluation in Sec. 4 and inform the further quantitative study in Sec. 5.

With the suggested placement from the pilot study, we then implement a proof-of-concept prototype of video avatar-based AR teleconferencing. Specifically, it enables remote teleconferencing through video avatars (Video Avatar mode), AR transplant of video conferencing (Video Grid mode), and virtual avatar representations (Avatar mode) using a commodity camera and an AR HMD (Microsoft HoloLens 2) for each user. Using our prototype, we compare the video avatar-based teleconferencing with the other two modes of teleconferencing mentioned above and conventional video conferencing in a 2-remote-user scenario to verify whether the use of video avatars outperforms other solutions and achieves the balance between fidelity and

**co-presence (RQ2).** The results show that the video avatar condition is the most preferred, presenting a sense of co-presence as high as the avatar condition while having a high fidelity similar to the video-conferencing condition.

While video avatars are proved to be the most feasible solution for current HMD-based AR teleconferencing in the evaluation, their social proxemics perceived by users (obtained from our pilot study) are significantly different from those of real humans. We consider the limited FoV of the AR HMD to be one of the most important causes. Existing works reveal the negative effect of a small FoV on the performance of visual tasks [32], [33], [34], [35] and a user's perception of virtual agents [36], [37]. This leads to our third question: **what is the effect of HMD's FoV on the optimal placement of video avatars in AR small-group teleconferencing (RQ3)?** To answer this question, we conduct our further quantitative study that involves a wide range of FoV conditions using a VR-simulated AR environment. It is a common practice to use VR simulation to explore problems from AR with some hardware constraints [38], [39], [40]. We adopt the VR simulation since VR HMDs have relatively larger FoVs than Optical See-Through (OST) AR HMDs, such as HoloLens 2 we used in the pilot study and the evaluation. We also compare the two most common aspect ratios for current AR HMDs (16:9 and 3:2)¹ in the study design. We first initially verify the validity of the VR simulation by comparing the data from the AR pilot study and the VR study. Then we examine the aspect ratio's effect and find no significant placement difference between the conditions. We find significant correlations between FoV and the video avatar placement and regress the data into placement models with FoVs ranging from 10° to 180°. They can give references for the video avatar placement on not only AR HMDs currently available but also future ones with larger FoVs up to nearly that of human eyes.

The main contributions of this work are threefold.

* Through a pilot study, we find the optimal user-centered video avatar placement in small-group AR teleconferencing scenarios with up to 4 remote users,
and identify the potential influence of the HMD's FoV on the optimal placement.

* We build a proof-of-concept prototype system using the results obtained from the pilot study. Through a user study based on this prototype, we conclude the video avatar's ability to achieve a better balance between fidelity and co-presence.

* We regress placement models from the data collected in a further experiment in a VR-simulated AR environment involving larger FoV conditions to provide quantitative references for the video avatar placement in AR teleconferencing applications on current and future AR HMDs.

# 2 RELATED WORK

## 2.1 Video-Based Avatars for AR Teleconferencing

For 3D video avatars, the media processing pipeline generally consists of capturing, transmitting, reconstructing, and rendering the volumetric data. The effect of systems using specially equipped installations to build the pipeline has seen a tremendous evolution from earlier endeavors [19], [20], [21], [41], [42], [43] to most recent works [1], [2], [3]. To capture and present 3D video avatars, existing systems rely on complex setups (e.g., fifteen surrounding cameras [41], CAVE-like installations [19], a multi-camera setup and a projection-based display [42], a set of RGB-D cameras and projectors [21], [43], two rooms equipped with RGB-D cameras and projectors [20]). State-of-the-art systems such as Holoportation [2], Project Starline from Google [1], and VirtualCube [3] also develop their own sophisticated installations to demonstrate teleconference experiences with unprecedented fidelity and 3D effect. However, the complicated and expensive configurations currently prevent these solutions with high-quality 3D video avatars from being applied in broader everyday scenarios. While using a single consumer-grade RGB-D camera such as Microsoft Kinect can also obtain a 3D or "2.5D" video avatar [14], [44], it still suffers from artifacts with the effect more similar to that of those methods at the earlier stage.

While it is hard for 3D video avatars to be both high-fidelity and easy to capture and reconstruct, real-time video streaming makes it very simple for 2D video avatars. It has been decades for researchers to incorporate 2D human videos into VR [22], [23], [24], [45], [46], [47] and AR [13], [16], [48] environments. For example, Insley et al. [22] pre-record humans from every angle into videos and select an appropriate image frame according to the relative position of a viewer to the video avatar to display in a CAVE VR environment. Yura et al. [45] embed a 2D video avatar into a collaborative virtual environment and change the video stream according to the perspective between the viewer and the video avatar as well. Prins et al. [47] use video avatars to represent remote users in social VR. Borst et al. [23] incorporate a teacher as a video avatar in VR education scenarios. Although they capture the teacher using an RGB-D camera and render him as a 3D mesh, showing majorly the frontal view with little movement leads to an effect with fewer artifacts and is similar to a 2D video avatar. Cui et al. [24], and Wang et al. [46] fuse video avatars into virtual scene models for surveillance use. Billinghurst

et al. [16], [48] overlay videos of users in the real world aligned with the corresponding objects through AR HMD using marker-based tracking. Simone et al. [13] study the use of a video avatar for watching TV together in a VR-simulated AR environment. They either do not use life-size portraits or concentrate on the system implementation and how to display the 2D video avatar from the right perspective to make the experience more 3D-like. We focus more on the life-size whole-body video avatar's placement, its performance regarding fidelity and presence, and how the placement is affected by the HMD's FoV in the user-centered group conversation scenario.

Telepresence robots such as Double² represent another form of using 2D video avatars in teleconference and telepresence scenarios. Such approaches mount a screen on a movable robot controlled by a remote user as his/her local representation. Otsuka [17] uses actuated screens to present small group-to-group teleconferences. Neustaedter et al. [49] explore the usage of telepresence robots to attend academic meetings. To improve the sense of co-presence of telepresence robots, Furuya and Takashio [50] propose to replace the background of the remote user's video on the screen with the local environment, blending the robot more into the surroundings. Chuah et al. [51] use props to increase the physicality of avatars on screens. While being a practical solution, the form of a robot with the extra hardware component makes them less similar to real humans than placing life-size video avatars in AR. Moreover, the AR solution is more scalable to multi-user scenarios.

We conclude that compared to existing solutions, using life-size 2D video avatars in small-group HMD-based AR teleconferencing is more feasible with its low cost and easy setup, and the ability to present a rather high-fidelity experience with fewer artifacts in this user-centered layout.

## 2.2 Content Placement in AR

Existing works exploring the placement of virtual agents in AR and VR mostly focus on the scene semantics. They statically place them on a sofa or a seat [20], [21] or retarget them according to the semantics of the remote user's position [9], [52]. For the video avatar placement in our scenario, the main considerations are the user's comfortableness and perceived co-presence and similarity to real-human scenarios, and the potential influence of the HMD's FoV. The works by Lang et al. [53] and Ye et al. [31] share a similar goal to ours by considering the agent-user interaction. But they focus on virtual avatars in assistant and navigation scenarios while we explore video avatars in the AR teleconferencing scenario.

From the perspective of User Interface (UI), researchers also explore the placement of UI components in AR and VR regarding user's social and physical comfortableness in public transit [39], the UI elements' relations with each other and the physical environment [40], and users' cognitive load [54]. With the video avatar being a special type of UI element, besides ergonomics, we also focus on interpersonal relations [55] and the user's feel of proxemics [30] when determining the placement of remote users' video avatars.
## 2.3 Limited Field of View in AR

One of the major restrictions of the current AR HMD is the limited FoV. Being not able to cover the full FoV of human eyes (around 200° horizontally and 150° vertically), the small FoV of HMDs can cause problems for users during the immersive experience. On the one hand, it can harm task performance. Czerwinski et al. [32] find it slows the navigation speed in virtual environments. Ragan et al. [34] find it worsens training effectiveness for a visual scanning task. Kim et al. [35] find it increases the targeting time in gaming. Ren et al. [38] find it increases the task completion time in an information-seeking scenario. On the other hand, it can affect the user's perception of virtual content. Lee et al. [36] find the mismatch between the FoVs of the HMD and human eyes affects the user's behavior related to the perception of the virtual agent in the proximity. Wang et al. [37] discuss the potential impact of small FoV on users' sense of presence or the agent's realism. Swan et al. [56] find the existence of an occluder (resulting in a decreased FoV) leads to more misjudgments on distance. With these previously-found various effects caused by FoV, we explore whether it can affect the optimal placement of video avatars in our group conversation scenario.

## 2.4 Perception of Remote Participant Representation

Researchers find video-based representations to be more preferred than virtual avatars in social-oriented scenarios involving more user-agent interactions such as conferencing for work [11], advice seeking [12], teleconsultation [14], education [57], and TV-watching [13]. Other than the video avatar perceived from the third-person perspective, existing works show an improved experience with self-embodied video avatars. Gonzalez et al. [58] find using a video avatar to present the self embodiment can lead to a high level of sense of presence and embodiment, and find it provides better visual quality than using virtual replicas [59]. Lee et al. [60] find incorporating the self-embodied video avatar can enhance the AR cinematic experience. Similar positive effects of video-based representation were found on size estimation, body ownership, and presence [61].

These works motivated us to further investigate the use of video avatars in teleconferencing. Our work explores the user's perception by comparing the video avatar representation with screen-based video conferencing, direct transplant of video conferencing, and avatar-mediated teleconferencing. We find video avatars can provide a high level of both fidelity and co-presence in this user-centered layout, with an experience surpassing those of the direct transplant and the virtual avatar-mediated approach in such teleconferencing applications on AR HMDs.

# 3 PILOT STUDY

In this section, we introduce our pilot study on the optimal placement of the remote participants' video avatars in the local space. As mentioned previously in this paper, users often place themselves following certain formations in real-world group conversations and tend to have different perceptions of proxemics and co-presence in the virtual world with the change in factors such as remote user representation and

FoV. Therefore, with life-size 2D video avatars in the small-group conversation scenario unexplored, we conduct our pilot study using a within-subjects design to find their optimal placements that serve the best for group conversations with different numbers of participants, and initially explore the effect of FoV (the independent variable) on the optimal placement on the widely adopted Microsoft HoloLens 2 OST AR HMD.

## 3.1 Participants

We recruited 18 graduate students (14 males and 4 females; average age: 23.6 (SD = 0.7)) from the local campus as the participants. The sample size (along with that in the subsequent user study in Sec. 4) is in line with the suggestion by relevant research in HCI studies [62], and is thus believed to be sufficient for drawing our conclusions. We recruited young college students as participants since they are sensitive to XR technology and are our main target users. They can represent the population who are willing to try and judge new AR/VR technologies. It is also a common sample composition in other AR/VR studies [9], [57], [63]. We mainly focused on the diversity of the participant's AR/VR experience since the novelty effect could influence the result of studies on new technologies. Among the participants, 5 had no prior AR/VR experience, 9 had experienced several times, and 4 had used HMDs extensively.

## 3.2 Study Scenarios

We conduct our pilot study in the small-group symmetric AR teleconference scenario. In this scenario, users are located in different spaces and join the teleconference using their AR HMDs as video avatars. Each user has the same setup and an equal role with no superior characters such as leaders, teachers, or interviewers. The relation between each user is acquaintances or friends, with no extra intimacy or hostility. The conversation topics are free. People can talk about serious topics such as work and research or casual topics such as dinner suggestions and games. We set four scenarios with the difference only in the conversation group size. We denote them as 1-RU, 2-RU, 3-RU, and 4-RU scenarios, including 1, 2, 3, and 4 remote users (a total of 2, 3, 4, and 5 participants, with the local user included), respectively.

## 3.3 Experiment Setup

As mentioned earlier, we use Microsoft HoloLens 2 to present the AR experience in our pilot study. It has a diagonal FoV of approximately 50° with the 3:2 aspect ratio. We pre-record the videos of every remote user greeting the local user and each other in all four scenarios, remove the background using a video matting method [64] to get human-only videos with transparent backgrounds, save them as local files, and place them in Unity as video avatars for HoloLens to render. To ensure the video avatars are life-size, it is important to calibrate their sizes to be the same as the corresponding users. So we (the experimenters) ask each user who is recorded in the corresponding video clip to stand next to his/her video avatar (observed through
HoloLens) as the reference and do the calibration to make sure they are of the same height and width.

Informed by previous studies on the F-Formation in group conversations [27] that people commonly tend to form a circle in conversations with multiple people, we set a circular layout for our AR small-group conversation scenario, as shown in Fig. 1 (Right). We pin the local user at the bottom of the circle and symmetrically place the RUs on the circle. The final placement is determined by two parameters: *Radius* and *Radian*. Radius defines the size of the circle. It is also perceived as the vertical distance from the local user to the group of remote users' video avatars. A large Radius puts the video avatars far away from the local user, while a small Radius pulls them near. Radian refers to the angle between the vectors formed from the user's position to the leftmost and rightmost video avatars. Since the video avatars distribute evenly in the Radian range, it is also perceived as the horizontal distance between adjacent video avatars. A large Radian separates the video avatars from each other and makes the ones on the leftmost and rightmost get closer to the local user, while a small Radian gathers the video avatars towards the top of the circle. We let the participants adjust the video avatars' placement using 'IJKL' keys on a keyboard. The initial Radius and Radian in each task are random values between 0.4m and 5m, and 10° and 180°, respectively. The frame rate is 30 fps, and when pressing the corresponding keys, the Radius and Radian increase/decrease by 0.01m and 0.3° per frame, respectively. All participants got used to the keyboard control quickly, and we did not receive any report of difficulties in controlling the video avatar placement.

We control the FoV by placing four black virtual occluders 0.3m away in front of the main camera and manipulating their positions accordingly in each FoV condition. With pure black being transparent in OST HMDs, the occluders shape the FoV by clipping certain parts of the virtual content on the top, bottom, left, and right, leaving the physical space unaltered.

We consider three FoV conditions, i.e., **30-FoV** (FoV = 30°), **40-FoV** (FoV = 40°), and **50-FoV** (FoV = 50°). They cover the FoV range of common current-available OST HMDs. Here in the pilot study, we use FoV as a simple reference to the diagonal FoV with an aspect ratio of 3:2. We annotate the coarse FoV boundaries under the three FoV conditions in Fig. 2. The images are screenshots from a participant's trial (p5) in the 3-RU scenario when the video avatars are placed at the distance the participant felt optimal.

## 3.4 Metrics

We ask the participants to decide the optimal distance by considering three metrics, namely comfortableness, sense of co-presence, and similarity to the real-world scenario. A high level of comfortableness means the participant would not feel too much fatigue during the conversation and feel everyone's placement is socially appropriate. According to previous measurements [65], we describe the sense of co-presence to the participants as the feeling that the video avatars "create a sense of distance between you", "they actually take up a spot in the physical space", and "you have

<figure><img src="image_4.png" /><figcaption>Fig. 2. An illustration of the (a) 30-, (b) 40-, and (c) 50-FoV conditions in the pilot study. Here in the images, boundaries of FOV are indicated with white rectangles and are not visible to participants. The virtual content outside of the FoV is clipped by the transparent occluders, whose exact positions for each FoV condition are calculated and defined in the controller script. Each image shows the view after users adjust the video avatars' placement under the corresponding FoV condition. The change in the user's perceived optimal *Radius* leads to different sizes of video avatars in the images.</figcaption></figure>

the urge to spatially interact with them". High similarity to the real-world scenario means each video avatar's placement in the AR experience is very similar to that in a real-world conversation where the remote users are physically co-located in the space. We ask the participants to find the video avatars' optimal placement to balance the three objectives of optimizing the three corresponding metrics.

## 3.5 Procedure

First, we introduced the basic concept of AR teleconferencing to the participants. Second, after they got its idea, we introduced the small-group conversation scenario in our study (as described in Sec. 3.2) verbally using real-life examples and told them their role as the local user. Afterward, we presented the participants with the task of finding the optimal placement for the remote users' video avatars. We explained the meaning of the three metrics to them using the definition in Sec. 3.4 and gave examples for their better understanding. We ensured they understood each metric and clarified the optimal placement being the result of balancing all three metrics. Afterward, we introduced how they could adjust the video avatars' placement and submit the result in each task using the keyboard, and walked them through a practice task to get familiar with the operation and the experience. After completing the preparatory work, we proceeded to the formal experiment process.

The experiment had 12 tasks in total (3 FoV conditions in each of the 4 scenarios) for each participant. One participant took part in the study at a time on his/her own and completed 3 tasks in all 4 scenarios with the pre-recorded RUs' video avatars (as illustrated in Sec. 3.3). The video avatars waved hello to the participant and each other to create a sense of a group meeting, but there was no actual verbal conversation or discussion during the study. The sequence of the 3 FoV conditions (corresponding to the 3 tasks in each scenario) was counterbalanced hierarchically
using a balanced Latin square, with each FoV condition presented equally 6 times in each position of the experimental sequence. In each task, the participant adjusted the video avatar placement given an FoV condition and submitted the optimal placement after the adjustment. The controller script saved the (Radius, Radian) parameter pair. The participant then proceeded to the next condition until finishing all 3 conditions in a scenario and proceeded to the tasks in the next scenario. The participant experienced every scenario exactly once and completed the tasks in all scenarios in turn. The sequence of the 4 scenarios was also partially counterbalanced. After finishing all 12 tasks, each participant engaged in a short semi-structured interview, which lasted about five minutes to explore potential insights. In the interview, we mainly discussed how they felt during the experiment, what they thought was the major reason for the described experience, and suggestions for improving the proposed system.

## 3.6 Results

We collected 18 × 3 × 4 = 216 sets of (Radius, Radian) data in total (Radian in the 1-RU scenario has no meaning). Fig. 3 shows the distribution and statistical results. The results in each condition help determine the quantitative proxemics for placing teleconferencing video avatars on OST HMDs with corresponding FoVs. The comparison of the results from the three conditions can help initially explore the FoV's effects on the video avatar placement in this small range. To analyze the results, following the analysis in similar works [9], [31], we first ran Shapiro-Wilk Normality Tests on the collected data to check their normality. With the non-normal distribution issues found, we conducted non-parametric Friedman Test and Post-hoc Nemenyi Test on the data in each scenario to see if there are significant differences in the Radian and Radius values under the three FoV conditions. We also conducted Pearson Correlation Tests on the normally distributed samples and Spearman Correlation Tests on the non-normally distributed samples to see if the Radian and Radius values significantly correlate with the FoV values. We will elaborate on the specific results under the confidence interval of *p* < .05 below.

**Optimal placement.** For the optimal placement, we distill the average Radian and Radius in each scenario as the parameter pair to determine the optimal placement. Note that the Radian is measured in degrees, and the Radius is measured in meters. We list the data for the **30-** and **40-FoV** conditions in the supplemental file (Sec. 2.1), and report the data in the **50-FoV** condition below, which will be used in the subsequent evaluation in Sec. 4. For the **50-FoV** condition, the optimal (Radian, Radius) is (1, 1.09) for the 1-RU scenario, (40.20, 1.09) for the 2-RU scenario, (66.63, 1.17) for the 3-RU scenario, and (80.42, 1.50) for the 4-RU scenario.

**Initial analysis of the FoV's effect.** We find no significant correlation between (Radian, Radius) and FoV. We list the detailed statistical results of the correlation test in Tab. 1 in the supplemental file. We report the data below for the significant tests among the three conditions (N = 18 in each scenario).

**Radian.** The Friedman Test shows a significant effect of the three FoV conditions in all three scenarios (Chi-square

statistic = 14.33, *p* < .001 in the 2-RU scenario, Chi-square statistic = 16.78, *p* < .001 in the 3-RU scenario, and Chi-square statistic = 11.44, *p* = .003 in the 4-RU scenario). We further conducted Post-hoc Nemenyi Tests for pairwise comparisons and summarized the results in Fig. 3 (b) with annotations of the level of significance. The results show that Radians are not significantly different between the **40-FoV** (*M* = 39.1, *SD* = 11.7 in the 2-RU scenario, *M* = 68.1, *SD* = 16.8 in the 3-RU scenario, and *M* = 75.2, *SD* = 23.3 in the 4-RU scenario) and **50-FoV** (*M* = 40.2, *SD* = 10.4 in the 2-RU scenario, *M* = 66.6, *SD* = 15.0 in the 3-RU scenario, and *M* = 80.4, *SD* = 23.1 in the 4-RU scenario) conditions. However, Radians are significantly smaller in the **30-FoV** condition (*M* = 33.8, *SD* = 10.4 in the 2-RU scenario, *M* = 59.6, *SD* = 18.3 in the 3-RU scenario, and *M* = 73.0, *SD* = 23.6 in the 4-RU scenario) than in the **50-FoV** condition in all three scenarios. Radians in the **30-FoV** condition are also significantly smaller than those in the **40-FoV** condition in the 2-RU and 3-RU scenarios.

**Radius.** The Friedman Test shows a significant effect of the three FoV conditions in the 2-RU scenario (Chi-square statistic = 7.00, *p* = .030) but shows no significant effect in the other three scenarios (Chi-square statistic = 4.78, *p* = .092 in the 1-RU scenario, Chi-square statistic = 2.33, *p* = .311 in the 3-RU scenario, and Chi-square statistic = 0.11, *p* = .946 in the 4-RU scenario). We summarized and annotated the Post-hoc Nemenyi Test results in the 2-RU scenario in Fig. 3 (c). The results show that in the 2-RU scenario, there is a significant difference in Radius between **30-FoV** (*M* = 1.3, *SD* = 0.3) and **40-FoV** (*M* = 1.2, *SD* = 0.4) conditions. But there are no significant differences between the **30-FoV** and **50-FoV** (*M* = 1.1, *SD* = 0.2) conditions and **40-FoV** and **50-FoV** conditions.

## 3.7 Study Conclusion and Discussion

From the pilot study, we obtain the video avatars' optimal placement in four scenarios on Microsoft HoloLens 2. We distill the results for the subsequent evaluation (Sec. 4) using the same HMD and for other OST HMDs with FoVs corresponding to the study conditions. We also initially analyzed the potential effect of FoV on the optimal placement, which informs the subsequent further study (Sec. 5) to quantitatively examine the effect with a larger range of FoV. Next, we discuss some of the observations in detail.

**FoV can partially affect the optimal placement of the remote users, especially horizontally.** The significantly smaller Radian in the **30-FoV** condition than those in the **40-FoV** and **50-FoV** conditions tells that local users tend to let the remote users gather closer to each other to make sure they stay in the area where he/she can be aware of their existence without rotating their heads too often. However, the local user would not simply place the remote users' video avatars tightly next to each other to fit the FoV window because "it would feel like I am interviewing them and they are closer to each other than I am." and "they would feel not very present when stuck tightly to each other". For Radius, some participants reflect that "I cannot see the whole body of the remote peers, so I have to let them stand further away from me during the whole
<figure><img src="image_5.png" /><figcaption>Fig. 3. The statistical results for the data of the pilot study. (a) shows the distribution of all the placement data in all four scenarios. The solid point in dark blue at (0, 0) denotes the local user. (b) shows the comparison of the Radian data in 2-RU, 3-RU, and 4-RU scenarios since the participant stands directly face-to-face with the remote peer with a radian = 0 in the 1-RU scenario. (c) shows the comparison of the Radius data in all four scenarios but with outliers hidden since they contribute little to the overall trend and some of them are too far away, as seen in (a).</figcaption></figure>

process to see at least the upper body, but not too far to feel unnatural for the group conversation". We think the main reason for the insignificant effect of *FoV* on *Radius* in most scenarios is that the *FoV* conditions are all too small to observe the remote user's whole body without vertical head rotations. Users have to put the video avatars to the upper limit of the distance for natural conversations in all three *FoV* conditions for a better visual experience while preserving an acceptable spatial relation.

**Insignificant correlation.** We think it is because the *FoV* values in the three conditions are all too small to show the difference, and the number of *FoV* conditions is too limited to reveal the correlation. We can observe changes in *Means* between different *FoV* conditions in the data report and in Fig. 3 (b) and (c) for *Radian* and *Radius*, respectively. Though not significantly, there are tendencies that *Radian* would increase and *Radius* would decrease when *FoV* becomes larger. These indications are part of the motivation for our Study 3 in Sec. 5 to include more and larger *FoV* conditions to find out the correlation.

# 4 IMPLEMENTATION AND EVALUATION

As elaborated previously in this paper, using video avatars helps make HMD-based AR teleconferencing easy to set up and has the potential to achieve a balance between fidelity and co-presence. Therefore, we first implement a proof-of-concept prototype system to demonstrate the use of video avatars. Then we use the prototype as the platform to conduct an evaluation to verify that using video avatars can present the aforementioned balanced experience.

## 4.1 Proof-of-Concept Implementation

Our prototype has three modes (as illustrated in Fig. 4), namely *Avatar mode*, *Video Grid mode*, and *Video Avatar mode*, where remote users are represented as avatars, videos on AR virtual screens (the direct transplant of video conferencing), and video avatars, respectively. For each user, our system setup includes only an AR HMD (HoloLens 2 in our implementation) to render the remote users' agents and a PC with a webcam to capture and process the real-time image of the local user.

In the *Avatar mode*, the PC runs a 3D human pose estimation script using the video captured by the webcam, and transmits the estimated joint position to other users' HMDs to get the corresponding avatar moving. We use the solution from *MediaPipe*<fnref n="3" /> to implement the pose estimation. We build TCP connections for data transmission. Referring to the aforementioned avatar-mediated teleconference applications such as Microsoft Mesh, we only enable the avatar's hand movement, with legs staying in an Idle animation state. We use the positions of the user's wrist joints and apply Inverse Kinematics to present the avatar's arm motion. For the head motion, we make the avatar look at the local user during the whole experience. The avatar and the animation clips in our implementation are from Mixamo<fnref n="4" />.

In the *Video Grid mode*, the PC directly transmits the video frames of the local user to other users' HMDs through TCP networking. The videos are rendered on a plane as a virtual screen placed in front of the local user at the head level.

In the *Video Avatar mode*, the PC runs a background removal script utilizing the image segmentation solution also from *MediaPipe* to separate the human from the background in the video captured by the webcam in real time and transmits the matted frames to other users' HMDs through TCP networking as video avatars. The bandwidth requirement in our prototype is only around 300Kb/s in the Video Grid and Video Avatar modes, and lower than 10Kb/s in the *Avatar mode* since we only transmit the joint positions.

We manually adjust virtual light sources to match the physical lighting condition. We also set a virtual floor with the same texture as the physical floor in the room. This setup makes the lighting and shadowing effect of virtual content similar to that of physical objects in the AR environment. This process can be further automated following relative research [66].

## 4.2 Participants

We recruited 20 graduate students (15 males and 5 females; average age: 23.3 (SD = 1.2)) from the local campus as the

<footnote n="3">3. https://mediapipe.dev</footnote>

<footnote n="4">4. https://www.mixamo.com</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 4. An illustration of the (a) Avatar, (b) Video grid, and (c) Video avatar modes in our prototype system. They also correspond to the conditions with the same names in the evaluation. The Video condition is not shown here since it is implemented with commercial software instead of our prototype system and is commonly seen in everyday life.</figcaption></figure>

participants. Most of them had a moderate level of AR/VR experience: 8 had no prior AR/VR experience, 11 had experienced several times, and 1 had used HMDs extensively.

## 4.3 Study Scenario

The scenario in the evaluation is still the small-group conversation introduced in the pilot study in Sec. 3.2. We let the participants play a debate-like conversational game called "who is the spy" with two remote users. It is utilized in previous studies [67] as the task for evaluating group conversation applications. In this game, all users get a word of their own, with one of them being different (the spy). Users describe their words in a round and take a vote on who is the suspected spy. The spy wins if not voted out, and others win if they correctly spot the spy. One round of the game normally ends with either side winning in 2-4 minutes.

## 4.4 Experiment Setup

We use HoloLens 2 to present the AR experience using our prototype system and a commercial application to present the conventional video conferencing experience. We use the default FoV size (about 50°, as mentioned previously) of the HMD in the evaluation. We have two confederates to play the role of two remote users in the game and the participant joins as the local user, forming a three-person group conversation for each participant. We set the remote users' video avatars at the optimal position obtained from our pilot study. We pre-calibrate the distance between the remote users and their webcam(s) to make sure their video avatars observed by the participant are life-size.

## 4.5 Study Conditions

We set four conditions, namely **Video**, **Avatar**, **Video Grid**, and **Video Avatar**. **Video** is the conventional video conferencing condition. Users complete the task on the PC with remote users' videos on the screen. The **Avatar**, **Video Grid**, and **Video Avatar** conditions correspond to the three modes of our prototype by name, as illustrated in Sec. 4.1. We set

the **Avatar** condition to represent the effect of most prevalent commercial apps such as Microsoft Mesh and Meta Horizon Workrooms. They use simplified cartoon avatars instead of life-like avatars to avoid being uncanny due to the lack of detailed real appearance and animation. In this evaluation, the conditions are within the scope of being mobile and easy to set up. The **Avatar** and **Video Grid** conditions represent the effect of existing HMD-based solutions. Complicated solutions such as high-quality full-body volumetric capturing and reconstruction are thus not considered. Since the effect of conventional video conferencing is well-known, we show the three AR conditions in Fig. 4. Note that FoVs in these three conditions are the same since the only independent variable in the evaluation is agent representation.

## 4.6 Metrics

By referring to similar studies [11], [14], we identify three metrics, namely "Fidelity/Realism", "Co-Presence", and "Preference". The "Fidelity/Realism" is defined as how photorealistic the remote user's representation is. We also describe it to the participants as "to what extent you think they are real people or can recognize them". "Co-Presence" is defined and described the same as in the pilot study in Sec. 3.4. "Preference" is an overall subjective rating of the experience. We present the three metrics to the participants using a seven-point Likert scale ranging from 1 to 7. 1 and 7 represent the lowest and highest "Fidelity/Realism", "Co-Presence", and "Preference", respectively.

## 4.7 Procedure

We first introduced the participants the task and made sure they were familiar with the rules of the game. Then we explained that they would use either a PC or an AR HMD to do the task in different conditions. Afterward, we introduced the definition of each metric and clarified them by giving examples and descriptions. After giving instructions on how to use the AR HMD, we started the experiment.

In the experiment, the participants completed four tasks, one for each condition, in which the representation of remote users (the two confederates) changed accordingly. We employed a partial counterbalancing approach utilizing the Latin square design that ensured each condition was presented equally 5 times in each position of the experimental sequence. This allowed us to achieve a reasonable level of counterbalancing as much as possible given the available sample size. We reminded the participants of the metrics' definitions and the meanings of the scores (as in Sec. 4.6) and asked them to rate their experience by selecting scores under the title of each metric on the seven-point Likert scale introduced earlier after each task. They could change the score of any task at any time. After finishing all four tasks, they engaged in a five-minute semi-structured interview similar to that in the pilot study to explore the potential insights and suggestions. The experiment lasted around 20 minutes for one participant with 4 rounds of games in different conditions and an interview. The relatively short experiment duration, sufficient rest time between experiments, and the fun of the game ensured a high level of engagement of the two confederates during the experiment.
<figure><img src="image_7.png" /><figcaption>Fig. 5. The statistical results for the data of the user study in the Evaluation.</figcaption></figure>

<table><caption>TABLE 1
Post-hoc Nemenyi Test results for Fidelity/Realism in the Evaluation. (*p < .05, **p < .01, ***p < .001)</caption><thead><tr><th>Fidelity/Realism</th><th>Video</th><th>Avatar</th><th>Video Grid</th><th>Video Avatar</th></tr></thead><tbody><tr><td>Video</td><td>/</td><td></td><td></td><td></td></tr><tr><td>Avatar</td><td>*</td><td>/</td><td></td><td></td></tr><tr><td>Video Grid</td><td>.900</td><td>**</td><td>/</td><td></td></tr><tr><td>Video Avatar</td><td>.900</td><td>*</td><td>.900</td><td>/</td></tr></tbody></table>

We make the following hypotheses:

* **H1:** The Fidelity of the **Avatar** condition will be significantly lower than the video-based conditions.

* **H2:** The Co-Presence of **Avatar** and **Video Avatar** conditions will be significantly higher than **Video** and **Video Grid**.

* **H3:** Compared to the traditional **Video** condition, users will prefer the AR-HMD-based conditions more and prefer **Video Avatar** the most.

## 4.8 Results

To analyze the study results, we ran Shapiro-Wilk normality tests on the collected data and found non-normally distributed samples. Therefore, we conducted non-parametric Friedman Test and Post-hoc Nemenyi Test to compare the subjective ratings of all the metrics. We will elaborate on the specific results under the confidence interval of *p* < .05 below. The statistical results for the data are shown in Fig. 5.

*Fidelity/Realism.* The Friedman Test shows a significant effect of the four conditions (*Chi-square statistic* = 18.66, *p* < .001). We further conducted the Post-hoc Nemenyi Test for pairwise comparisons and summarized the results in Tab. 1. The fidelity of **Avatar** (*M* = 3.7, *SD* = 1.2) is significantly lower than **Video** (*M* = 5.0, *SD* = 1.6), **Video Grid** (*M* = 5.4, *SD* = 1.1), and **Video Avatar** (*M* = 5.1, *SD* = 1.0). There is no significant difference between each two of the three video-based representations (i.e., **Video**, **Video Grid**, and **Video Avatar**).

*Co-presence.* The Friedman Test shows a significant effect of the four conditions (*Chi-square statistic* = 26.93, *p* < .001). We further conducted the Post-hoc Nemenyi Test for pairwise comparisons and summarized the results in Tab.

<table><caption>TABLE 2
Post-hoc Nemenyi Test results for Co-Presence in the Evaluation. (*p* < .05, **p* < .01, ***p* < .001)</caption><thead><tr><th>Co-Presence</th><th>Video</th><th>Avatar</th><th>Video Grid</th><th>Video Avatar</th></tr></thead><tbody><tr><td>Video</td><td>/</td><td></td><td></td><td></td></tr><tr><td>Avatar</td><td>***</td><td>/</td><td></td><td></td></tr><tr><td>Video Grid</td><td>.384</td><td>.106</td><td>/</td><td></td></tr><tr><td>Video Avatar</td><td>***</td><td>.900</td><td>*</td><td>/</td></tr></tbody></table>

<table><caption>TABLE 3
Post-hoc Nemenyi Test results for Preference in the Evaluation. (*p* < .05, **p* < .01, ***p* < .001)</caption><thead><tr><th>Preference</th><th>Video</th><th>Avatar</th><th>Video Grid</th><th>Video Avatar</th></tr></thead><tbody><tr><td>Video</td><td>/</td><td></td><td></td><td></td></tr><tr><td>Avatar</td><td>.900</td><td>/</td><td></td><td></td></tr><tr><td>Video Grid</td><td>.122</td><td>.256</td><td>/</td><td></td></tr><tr><td>Video Avatar</td><td>*</td><td>.122</td><td>.900</td><td>/</td></tr></tbody></table>

2. We can see that the co-presence score of **Avatar** (*M* = 5.3, *SD* = 1.4) is significantly higher than that of **Video** (*M* = 2.9, *SD* = 1.5), but is not significantly different from that of **Video Grid** (*M* = 4.2, *SD* = 1.3). **Video Avatar** (*M* = 5.2, *SD* = 1.3) can achieve a significantly higher level of co-presence than both **Video** and **Video Grid**, reaching a high level with no significant difference with **Avatar**. There is no significant difference between **Video Grid** and **Video** regarding co-presence.

*Preference.* The Friedman Test shows a significant effect of the four conditions (*Chi-square statistic* = 11.89, *p* = .007). We further conducted the Post-hoc Nemenyi Test for pairwise comparisons and summarized the results in Tab. 3. Compared with **Video** (*M* = 4.2, *SD* = 1.2), only **Video Avatar** (*M* = 5.5, *SD* = 1.0) is significantly more preferred. Both **Avatar** (*M* = 4.5, *SD* = 1.5) and **Video Grid** (*M* = 5.3, *SD* = 1.0) are not significantly different from it. There are also no significant differences in preference between the rest pairs of conditions.

## 4.9 Discussion

**H1 is supported.** **Avatar** is perceived to have a significantly lower fidelity than the other conditions. Users feel they are less real because “I cannot see the remote user’s facial expression, which is an important clue to tell if he is making up words.” and “I cannot tell when the remote user describes his word using gestures and body language”. This is in line with related research in other scenarios [12], [14] and could happen to all widely used avatar-based solutions. However, if the 3D modeling and animation of avatars could be photorealistic enough to cross the uncanny valley in easy-to-set-up configurations in the future, re-evaluation would be necessary. Apple demonstrated its solution on the Vision Pro HMD with life-like scanned upper-body-only avatars⁵. But the real effect is unclear since it is not publicly available yet. Several reviews reflect that, unlike high-end solutions such as Project Starline [1] from Google, it is still uncanny.

**H2 is mainly supported.** Due to the life-size user portrait “standing” on the floor in **Video Avatar** condition, users perceive remote peers significantly more present than both
**Video and Video Grid** conditions. With the good 3D effect of **Avatar**, it feels more present in the physical space than **Video**. However, the disadvantage in fidelity harms the sense of co-presence to be not significantly better than **Video Grid**. In addition, the direct transplant on the AR HMD (i.e., **Video Grid**) cannot bring an improved spatial experience for video conferencing, with no significant difference with **Video**.

**H3 is partially supported.** Compared to **Video, Video Avatar** is significantly more preferred, but **Avatar and Video Grid** are not. We think it is due to the avatar's drawback of low fidelity and the similar experience between the direct transplant and conventional video conferencing. It indicates that fidelity might be more important than co-presence in the conversational scenario. It can also be seen from the insignificant difference in preference between **Video Grid** and **Video Avatar**, with **Video Grid** having a significantly lower co-presence. **Video Avatar** is the most preferred. Though not significant, its mean score is higher than **Video Grid**.

In conclusion, the results verify that the video avatar representation can balance fidelity and co-presence, being the better choice for HMD-based small-group AR teleconferencing compared with the existing avatar-mediated approach and the direct AR transplant of video conferencing. Moreover, it can truly upgrade the experience of current video conferencing, being a feasible solution for AR HMDs with its simple setup.

# 5 FURTHER STUDY ON THE EFFECT OF FOV

In our pilot study, we initially explored the effect of FoV on the video avatar's optimal placement. We conclude the insignificant results are due to the limited number and range of the FoV conditions. Therefore, to further examine whether such effects exist, we conduct this follow-up study using a VR-simulated AR environment following existing practice [39], [40]. With OST AR HMDs suffering from limited FoVs, VR simulation can provide a similar experience to Video See-Through (VST) AR HMDs with larger FoVs. We also adopt a within-subjects design in this study and set FoV and aspect ratio as independent variables.

For the study condition, other than the **30-, 40-, and 50-FoV** conditions (for comparison with the pilot study to see if there is a difference in the optimal placement between using OST AR HMD and the VR-simulated AR), we add **70-, 90-, and 110-FoV** conditions (the range of FoVs of most existing AR and VR HMDs) in this study. Moreover, we involve the aspect ratio factor in this study. We consider the aspect ratios of the **16:9** and **3:2**. Most of the experimental setup is the same as the pilot study, except for using a VR HMD. To include large FoV conditions, we use HTC Vive Pro to present the experience, with an approximate 110° diagonal FoV. In order to better simulate the setup in the pilot study, we take a 360° picture of the same study room at the same spot where the participants stood in the pilot study and make it the spherical background of the VR environment. We keep all video avatars the same as those in the pilot study, and manipulate the lighting to be more similar to the real environment. Fig. 1 (the HMDs are post-added) and Fig. 2 show screenshots from this further VR study

and the pilot study, respectively, and could demonstrate the comparison. We recruited the same group of participants in the pilot study to ensure the design was within-subjects. The study scenarios, metrics, and procedures were all the same as the counterparts in the pilot study. We also employed a partial counterbalancing approach utilizing the Latin square design, ensuring each FoV condition was presented equally 3 times in each position of the experimental sequence. There were 2 or 3 days between this study and the pilot study for the participants, depending on their schedules. When joining the study, all participants clearly remembered the background, objections, and procedures of the previous pilot study in AR, which are still the same in this further VR study. They were also still familiar with the control, and most of them could start the experiment directly with no need for practice.

## 5.1 Results

The initial comparison between data collected from AR and VR environments. To initially examine whether the simulated "AR teleconference scenario" can actually represent the true AR experience, we first compared the *Radian* and *Radius* data with the same FoVs (30°, 40°, and 50°) and aspect ratio (3 : 2) from our experiments in VR simulation and AR environment to see if there are significant differences.

<figure><img src="image_8.png" /><figcaption>Fig. 6. The scatter plots showing the original placement data distribution and the corresponding polynomial fitting results for *Radian* in 2-, 3-, and 4-RU scenarios.</figcaption></figure>
For the result analysis, we examined the normality of the data using Shapiro-Wilk Normality Tests and conducted pairwise T-Tests on normally distributed samples and non-parametric Wilcoxon Signed-Rank Tests on non-normally distributed samples for each scenario, adding up to 21 pairs of data to test in total. We also ran TOST on these pairs to initially test the equivalence between these two groups of data. We set the TOST bound to 20 for *Radian* and 0.5 for *Radius*. The comparisons are visualized in Fig. 2 in the supplemental file, and the significance test results and the initial equivalence test results are shown in Tab. 2 in the supplemental file. Other than two exceptions, which will be illustrated later, we found significant equivalence in the rest 19 pairs of data and found that they were not significantly different from each other within pairs. The two exceptions both occur on *Radian* in the 3-RU Scenario. The *Radian* in 30-FoV condition in AR (*M* = 59.6, *SD* = 18.3) is significantly larger than that in VR (*M* = 46.7, *SD* = 13.3), *t-statistic* = 2.36, *p* = .024, *TOSTp* = .106. The *Radian* in 40-FoV condition in AR (*M* = 68.1, *SD* = 16.8) is also significantly larger than that in VR (*M* = 49.2, *SD* = 14.8), *t-statistic* = 3.50, *p* = .001, *TOSTp* = .418. We discarded the VR data from these two exceptions and combined other data that is not significantly different in VR and AR studies for the subsequent correlation and regression analyses. We discuss the results and the two exceptions in Sec. 5.2. For other statistical results showing the significant equivalence and no significant difference in data between AR and VR environments, please refer to Tab. 2 in the supplemental file.

**The comparison of the aspect ratio.** We also conducted pairwise T-Tests and non-parametric Wilcoxon Signed-Rank Tests on normally and non-normally distributed samples, respectively. The comparison of the data and the statistical results are shown in Fig. 1 and Tab. 3 in the supplemental file. We can see that the 16:9 and 3:2 aspect ratios do not have a significant effect on the placement of remote users in video-mirror-based AR teleconference applications in most conditions.

**Correlation test.** To examine whether there are correlations between the remote user placement (formulated by the *Radian* and *Radius* values) and the *FoV*, we first ran normality tests to determine the normality of the data and then conducted Pearson Correlation Test and Spearman Correlation Test on the normally distributed samples and the non-normally distributed samples, respectively. We summarized the corresponding statistical results in Tab. 4. From the correlation test results, we can see that both *Radian* and *Radius* are significantly correlated with *FoV*. The correlation between *Radian* and *FoV* is positive while the correlation between *Radius* and *FoV* is negative.

**Model fitting.** With the significant correlation results, we can now quantitatively explore the mapping from *FoV* to *Radian* and *Radius*. We regressed the data in each scenario using Polynomial Fitting with *FoV* as the independent variable. The interview conducted after the participants finished all tasks indicates that their adjustment of the *Radian* and *Radius* was consistent and monotonic. We thus choose the highest order for each fitting function that results in a monotonic curve when *FoV* = [0, 180]. We present the distribution of the data and the fitting results for *Radian* and *Radius* in Figs. 6,7,8,9,10, respectively. We choose *FoV*

<table><caption>**TABLE 4**
Correlation Test results between the *Radian* and *Radius* values and the *FoV* in the VR Study. (*p* < .05, **p* < .01, ***p* < .001)</caption><thead><tr><th></th><th></th><th>1-RU</th><th>2-RU</th><th>3-RU</th><th>4-RU</th></tr></thead><tbody><tr><td rowspan="2">Radian</td><td>Statistic</td><td>/</td><td>Pearson</td><td>Pearson</td><td>Spearman</td></tr><tr><td><i>P</i>-Value</td><td>/</td><td>.69</td><td>.36</td><td>.46</td></tr><tr><td rowspan="2">Radius</td><td>Statistic</td><td>Spearman</td><td>Spearman</td><td>Spearman</td><td>Spearman</td></tr><tr><td><i>P</i>-Value</td><td>-.36</td><td>-.32</td><td>-.29</td><td>-.24</td></tr></tbody></table>

<table><caption>**TABLE 5**
Coefficients and *F*-Test results of linear regressed models in the VR Study. (*p* < .05, **p* < .01, ***p* < .001)</caption><thead><tr><th></th><th></th><th>1-RU</th><th>2-RU</th><th>3-RU</th><th>4-RU</th></tr></thead><tbody><tr><td rowspan="3">Radian</td><td>Coefficients</td><td>/</td><td>/</td><td>0.22,<br>52.87</td><td>0.37,<br>61.79</td></tr><tr><td><i>F</i></td><td>/</td><td>/</td><td><i>F</i>(1, 196)<br>= 29.38</td><td><i>F</i>(1, 268)<br>= 73.40</td></tr><tr><td><i>P</i>-Value</td><td>/</td><td>/</td><td>***</td><td>***</td></tr><tr><td rowspan="3">Radius</td><td>Coefficients</td><td>-0.0045,<br>1.34</td><td>-0.0042,<br>1.33</td><td>-0.0045,<br>1.47</td><td>-0.0043,<br>1.63</td></tr><tr><td><i>F</i>(1, 268)</td><td>39.84</td><td>29.62</td><td>27.94</td><td>15.63</td></tr><tr><td><i>P</i>-Value</td><td>***</td><td>***</td><td>***</td><td>***</td></tr></tbody></table>

as the only independent variable and set the *number of remote users* as a scenario parameter instead of another independent variable because our application targets the small-group conversation use case and shapes the group as a circle. With the increase in participant numbers, the group formation could change to layouts with multiple rows or layers and may vary regarding the functional area and scene semantics. Therefore, a model regressed from the independent variable containing only small participant numbers would be less descriptive in scenarios involving large numbers of participants, in which case separately building new models is necessary. From the regression result, we can see that in order to meet the monotone constraints described earlier, except the *Radian* model in the 2-RU scenario was regressed as a quadratic function (as shown in Eq. 1, *F*(2, 267) = 124.5, *p* < .001), the rest sets of data were regressed linearly. They are formulated as $\lambda_0 FoV + \lambda_1$ with $\lambda_0$ and $\lambda_1$ denoting the first and second values in the coefficients respectively and *FoV* denoting the *FoV* value. We list the values of the coefficients and *F*-Test results in Tab. 5. Note that the difference in the *df* of the *Fstatistic* in the 3-RU scenario for *Radian* is caused by the removal of exceptional data as illustrated earlier.

$$ \text{Radian}_{2\text{RU}} = -0.0012\text{FoV}^2 + 0.45\text{FoV} + 20.20. \quad (1) $$

## 5.2 Discussion

**Choice of aspect ratio.** The two conditions in our study are not significantly different from each other. However, some participants reflect that “When the FoV decreases vertically, I would let the remote users stand a little further away from me to see more parts of each user’s body than rotate my head vertically to observe”. “Vertically rotating my head to see more body parts of a person feels impolite. It is like I am checking out him/her from head to toe while
<figure><img src="image_9.png" /><figcaption>Fig. 7. The scatter plots showing the placement data distribution and the corresponding polynomial fitting results for *Radius* in the 1-RU scenario.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 8. The scatter plots showing the placement data distribution and the corresponding polynomial fitting results for *Radius* in the 2-RU scenario.</figcaption></figure>

he/she is standing at a rather close distance during the conversation". "Wearing an HMD makes it tiring to rotate my head vertically". These reflections, along with the relatively smaller mean *Radius* values for the 3:2 aspect ratio in most of the conditions, imply that for video-mirror-based AR teleconference applications, when the FoV is limited by hardware, a lower horizontal-vertical aspect ratio might be a better choice. We can further explore lower aspect ratios such as 4:3, 4:5, 2:3, and 9:16 to determine the optimal aspect ratio for video-mirror-based AR teleconference applications.

**Validity of VR simulation.** It is common in previous research [38], [39], [40] to adopt VR simulations to explore AR problems. We initially verify its validity in our small group teleconferencing scenarios by finding no significant difference and significant equivalence between most of the data from our experiment in VR simulation and AR environment. It makes sense to explore our video-mirror-based AR teleconferencing application in bigger-FoV conditions in the VR-simulated AR environment when AR HMDs currently available only provide rather limited FoVs. Further specific tests to compare studying in VR simulation and directly on AR devices could be useful regarding the empirical bound values with larger data collections.

**The results are applicable to both VST and OST AR HMDs.** Our VR-simulated AR environment provides an experience similar to the VST AR HMD in terms of the HMD's size, weight, wearing comfort, and imaging quality. Together with the experiment in the AR environment using an OST HMD (i,e, Microsoft HoloLens 2), our study covers these two major types of devices from the perspective of overall user experience. Therefore, we initially reason that our results are applicable to both situations. Specific further quantitative studies would be useful to study the difference between these two types of AR HMDs or even among various products.

<figure><img src="image_11.png" /><figcaption>Fig. 9. The scatter plots showing the placement data distribution and the corresponding polynomial fitting results for *Radius* in the 3-RU scenario.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 10. The scatter plots showing the placement data distribution and the corresponding polynomial fitting results for *Radius* in the 4-RU scenario.</figcaption></figure>

**The two exceptional cases.** The *Radians* in 30- and 40-FoV conditions in the 3-RU scenario are significantly larger in AR than in VR. This can be explained using the reflection from some participants: "I am more willing to rotate my head when wearing the AR HMD because it is much lighter," and "Rotating the head in AR feels easier than in VR because there is no motion sickness issue". Note that the "AR HMD" and "AR" in the quotations both refer to the OST AR HMD used in our pilot study. We think the reasons why the exceptions occur only in the 30- and 40-FoV conditions in the 3-RU scenario are that 1) very small FoVs lead to significantly more head rotations during the study, which makes the difference between AR and VR significant as well. 2) Participants want to adjust the video avatars' placement while observing every one of them in 1-, 2-, and 3-RU scenarios, while they tend to only refer to two or three video avatars when doing the task in the 4-RU scenario. This makes 3-RU the scenario requiring the most head rotations since observing the whole RU group in 1- and 2-RU scenarios and the partial RU group in the 4-RU scenario is easier with targets distributed in relatively small angle ranges. We drop the VR data in these two exceptional scenarios in the model regressing process.

**The effectiveness of the regressed model.** When the HMD's FoV can cover the full FoV of human eyes with FoV = 180, the values predicted by our models are in line with those in real scenarios. *Radian* approaches the inside angles of regular polygons with vertices being the participants in the group conversation (60 for a triangle, 90 for a rectangle, and 120 for a pentagon, corresponding to the 2-, 3-, and 4-RU scenarios). *Radius* in each of the four scenarios with different numbers of users also reaches the value that can form a layout that preserves the common distance for friends or strangers (around 1m [28], [29]) between each participant.
This similarity between the predicted layout and the common real-world layout also proves the effectiveness of our regressed models and the overall high level of realism and co-presence provided by the video-based avatars.

# 6 CONCLUSION, LIMITATIONS, AND FUTURE WORK

In conclusion, we have explored the use of life-size 2D video avatars to represent remote users in small-group AR teleconferencing on the HMD platform. We have conducted a pilot study first to find the optimal placement of video avatars for teleconferencing on HoloLens 2, one of the most commonly used AR HMDs. We have also found initial implications of the effect of FoV on the video avatars' optimal placement in the pilot study. To demonstrate the use of video avatars in a real-time AR teleconferencing system, we have developed a proof-of-concept prototype and conducted an evaluation using the prototype and the optimal placement obtained from the pilot study. We found that using video avatars achieves a balance between fidelity and co-presence in HMD-based AR teleconferencing, and outperforms conventional videoconferencing. To make the video avatars' optimal placement results more generalizable and applicable to future AR HMDs with bigger FoVs, we have conducted a further study in a VR-simulated AR environment and fitted placement models as quantitative references for the teleconferencing applications on both current-available and future AR HMDs. Our findings through the series of user studies along with the proof-of-concept prototype could potentially benefit the development of widely accessible and immersive HMD-based AR telepresence applications. Next, we discuss the limitations of our work and the corresponding potential future work.

**Generalization.** Our current studies involve limited participant number and diversity. We are interested in conducting larger-scale studies with a greater number of participants that could cover the condition orders for complete counterbalancing and a more diverse population in gender, age, race, and appearance to make the results more generalizable. We compared agent representations on a single device with only one FoV size. We can investigate the video avatar representation across a wider range of devices and FoV conditions to obtain more general results. We used keyboards for participants to indirectly adjust the video avatar placement. We are interested in exploring whether and how such indirect control affects the optimal placement compared to direct manipulations. We can also quantitatively explore the potential effect of user engagement on the perception of agent representations. Moreover, we initially investigated the validity of VR simulation for studying AR applications. Further tests in more scenarios with larger data scales would benefit the study on the equivalence of AR and VR-simulated experiments.

**Constrained point of view.** This is the intrinsic limitation of the 2D video representation. We currently rotate the billboard vertically to make it always orient to the user. As a result, if the local user moves around, he/she still can only see the remote users from the frontal view. Moreover, though we ensure the video avatars are life-size, users of different heights still have different points of view vertically.

To tackle this problem, we will explore incorporating neural rendering techniques [68], [69], [70] to synthesize and show the correct view according to the user's head position in real-time.

**Semantic interaction with physical environments.** Our prototype system currently focuses only on the standing pose and makes sure the video avatars' feet are on the floor. In the future, we will explore more poses, such as sitting, and align video avatars with objects that are semantically affordable for the pose. Additionally, when a remote user moves around in the remote space, we need to adapt the position and content of his/her video avatar regarding the semantics of the physical space as existing approaches do with virtual avatars [9], [10], [71], [72]. We can also adjust the lighting and shadowing effects in the video to adapt to the lighting conditions in the local environment [73]. In addition, the influence of presenting lighting and shadowing effects that match the physical environment for video avatars could be further studied.

**Incorporating more features.** We take the first step to investigate the use of video avatars in our current studies utilizing a rather simple proof-of-concept prototype. Adding more features such as visualizing shared gaze, user attention, and gestures [8] could further resemble a stronger system. We can further adapt such visualization techniques to our video avatar-based prototype. We can also add more detailed animations, such as lip movements and facial animations, for virtual avatars and incorporate spatial audio features into the prototype system to make the experience more immersive.

**More integrated setup.** We use a webcam put statically about 2 meters away at approximately the same height as the remote user's eyes to capture the whole body portrait. We can explore more integrated configurations such as attaching a 360° camera to the HMD to capture the user and apply image distortion correction techniques [74] to restore the video avatar.

**Large scale scenarios.** In this work, we focus on the small-group teleconferencing scenario involving up to 5 participants (counting all the remote and local users). We can further explore scenarios with tens and even hundreds of people to apply the application for use cases such as hybrid classrooms and virtual events.

# ACKNOWLEDGMENTS

We thank the anonymous reviewers for their constructive comments and the interview and user study participants for their time. This work was supported by grants from the National Natural Science Foundation of China (No. 62192781 and 62172326), the City University of Hong Kong (No. 9678338, 9667260), the Innovation and Technology Commission (No. ITS/106/22), and Project of China Knowledge Centre for Engineering Science and Technology.

# REFERENCES

[1] J. Lawrence, D. B. Goldman, S. Achar, G. M. Blascovich, J. G. Desloge, T. Fortes, E. M. Gomez, S. Häberling, H. Hoppe, A. Huibers, C. Knaus, B. Kuschak, R. Martin-Brualla, H. Nover, A. I. Russell, S. M. Seitz, and K. Tong, "Project starline: A high-fidelity telepresence system," *ACM Transactions on Graphics (Proc. SIGGRAPH Asia)*, vol. 40(6), 2021.
[2] S. Orts-Escolano, C. Rhemann, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson, S. Khamis, M. Dou, V. Tankovich, C. Loop, Q. Cai, P. A. Chou, S. Mennicken, J. Valentin, V. Pradeep, S. Wang, S. B. Kang, P. Kohli, Y. Lutchyn, C. Keskin, and S. Izadi, "Holoportation: Virtual 3d teleportation in real-time," in *Proceedings of the 29th Annual Symposium on User Interface Software and Technology*, ser. UIST '16. New York, NY, USA: Association for Computing Machinery, 2016, p. 741–754. [Online]. Available: https://doi.org/10.1145/2984511.2984517

[3] Y. Zhang, J. Yang, Z. Liu, R. Wang, G. Chen, X. Tong, and B. Guo, "Virtualcube: An immersive 3d video communication system," *IEEE Transactions on Visualization and Computer Graphics*, vol. 28, no. 5, pp. 2146–2156, 2022.

[4] Microsoft, "Remote assist," https://dynamics.microsoft.com/en-us/mixed-reality/remote-assist.

[5] I. Magic Leap, "Assist app," https://www.magicleap.com/on-device/solutions/assist.

[6] Microsoft, "Microsoft mesh," https://www.microsoft.com/mesh.

[7] Meta, "Horizon workrooms," https://www.oculus.com/workrooms.

[8] T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W. Lindeman, B. H. Thomas, and M. Billinghurst, "Mini-me: An adaptive avatar for mixed reality remote collaboration," in *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*, ser. CHI '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 1–13. [Online]. Available: https://doi.org/10.1145/3173574.3173620

[9] L. Yoon, D. Yang, J. Kim, C. Chung, and S.-H. Lee, "Placement retargeting of virtual avatars to dissimilar indoor environments," *IEEE Transactions on Visualization and Computer Graphics*, vol. 28, no. 3, pp. 1619–1633, 2022.

[10] X. Wang, H. Ye, C. Sandor, W. Zhang, and H. Fu, "Predict-and-drive: Avatar motion adaption in room-scale augmented reality telepresence with heterogeneous spaces," *IEEE Transactions on Visualization and Computer Graphics*, vol. 28, no. 11, pp. 3705–3714, 2022.

[11] S. Junuzovic, K. Inkpen, J. Tang, M. Sedlins, and K. Fisher, "To see or not to see: A study comparing four-way avatar, video, and audio conferencing for work," in *Proceedings of the 2012 ACM International Conference on Supporting Group Work*, ser. GROUP '12. New York, NY, USA: Association for Computing Machinery, 2012, p. 31–34. [Online]. Available: https://doi.org/10.1145/2389176.2389181

[12] Y. Pan and A. Steed, "A comparison of avatar-, video-, and robot-mediated interaction on users' trust in expertise," *Frontiers in Robotics and AI*, vol. 3, p. 12, 2016.

[13] F. De Simone, J. Li, H. G. Debarba, A. E. Ali, S. N. Gunkel, and P. Cesar, "Watching videos together in social virtual reality: An experimental study on user's qoe," in *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, 2019, pp. 890–891.

[14] K. Yu, G. Gorbachev, U. Eck, F. Pankratz, N. Navab, and D. Roth, "Avatars for teleconsultation: Effects of avatar embodiment techniques on user perception in 3d asymmetric telepresence," *IEEE Transactions on Visualization and Computer Graphics*, vol. 27, no. 11, pp. 4129–4139, 2021.

[15] R. Vertegaal, "The gaze groupware system: Mediating joint attention in multiparty communication and collaboration," in *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, ser. CHI '99. New York, NY, USA: Association for Computing Machinery, 1999, p. 294–301. [Online]. Available: https://doi.org/10.1145/302979.303065

[16] M. Billinghurst and H. Kato, "Out and about—real world teleferencing," *BT technology journal*, vol. 18, no. 1, pp. 80–82, 2000.

[17] K. Otsuka, "Mmspace: Kinetically-augmented telepresence for small group-to-group conversations," in *2016 IEEE Virtual Reality (VR)*, 2016, pp. 19–28.

[18] Z. He, K. Wang, B. Y. Feng, R. Du, and K. Perlin, "Gazechat: Enhancing virtual conferences with gaze-aware 3d photos," in *The 34th Annual ACM Symposium on User Interface Software and Technology*, ser. UIST '21. New York, NY, USA: Association for Computing Machinery, 2021, p. 769–782. [Online]. Available: https://doi.org/10.1145/3472749.3474785

[19] S.-Y. Lee, I.-J. Kim, S. C. Ahn, M.-T. Lim, and H.-G. Kim, "Toward immersive telecommunication: 3d video avatar with physical interaction," in *Proceedings of the 2005 International Conference on Augmented Tele-Existence*, ser. ICAT '05. New York, NY, USA: Association for Computing Machinery, 2005, p. 56–61. [Online]. Available: https://doi.org/10.1145/1152399.1152411

[20] T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. Wilson, "Room2room: Enabling life-size telepresence in a projected augmented reality environment," in *Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing*, ser. CSCW '16. New York, NY, USA: Association for Computing Machinery, 2016, p. 1716–1725. [Online]. Available: https://doi.org/10.1145/2818048.2819965

[21] A. Maimone, X. Yang, N. Dierk, A. State, M. Dou, and H. Fuchs, "General-purpose telepresence with head-worn optical see-through displays and projector-based lighting," in *2013 IEEE Virtual Reality (VR)*, 2013, pp. 23–26.

[22] J. A. Insley, D. J. Sandin, and T. A. DeFanti, "Using video to create avators in virtual reality," in *ACM SIGGRAPH 97 Visual Proceedings: The art and interdisciplinary programs of SIGGRAPH'97*, 1997, p. 128.

[23] C. W. Borst, N. G. Lipari, and J. W. Woodworth, "Teacher-guided educational vr: Assessment of live and prerecorded teachers guiding virtual field trips," in *2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, 2018, pp. 467–474.

[24] X. Cui, D. Khan, Z. He, and Z. Cheng, "Fusing surveillance videos and three-dimensional scene: A mixed reality system," *Computer Animation and Virtual Worlds*, p. e2129, 2022.

[25] A. Kendon, *Conducting interaction: Patterns of behavior in focused encounters*. CUP Archive, 1990, vol. 7.

[26] S. K. Pathi, A. Kristoffersson, A. Kiselev, and A. Loutfi, "Formations for social interaction in simulation using virtual agents and mobile robotic telepresence systems," *Multimodal Technologies and Interaction*, vol. 3, no. 4, p. 69, 2019.

[27] P. Marshall, Y. Rogers, and N. Pantidi, "Using f-formations to analyse spatial patterns of interaction in physical environments," in *Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work*, ser. CSCW '11. New York, NY, USA: Association for Computing Machinery, 2011, p. 445–454. [Online]. Available: https://doi.org/10.1145/1958824.1958893

[28] E. T. Hall, R. L. Birdwhistell, B. Bock, P. Bohannan, A. R. Diebold Jr, M. Durbin, M. S. Edmonson, J. Fischer, D. Hymes, S. T. Kimball et al., "Proxemics [and comments and replies]," *Current anthropology*, vol. 9, no. 2/3, pp. 83–108, 1968.

[29] H. Hecht, R. Welsch, J. Viehoff, and M. R. Longo, "The shape of personal space," *Acta psychologica*, vol. 193, pp. 113–122, 2019.

[30] A. Huang, P. Knierim, F. Chiossi, L. L. Chuang, and R. Welsch, "Proxemics for human-agent interaction in augmented reality," in *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*, ser. CHI '22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491102.3517593

[31] Z.-M. Ye, J.-L. Chen, M. Wang, and Y.-L. Yang, "Paval: Position-aware virtual agent locomotion for assisted virtual reality navigation," in *2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*, 2021, pp. 239–247.

[32] M. Czerwinski, D. S. Tan, and G. G. Robertson, "Women take a wider view," in *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, ser. CHI '02. New York, NY, USA: Association for Computing Machinery, 2002, p. 195–202. [Online]. Available: https://doi.org/10.1145/503376.503412

[33] N. Kishishita, K. Kiyokawa, J. Orlosky, T. Mashita, H. Takemura, and E. Kruijff, "Analysing the effects of a wide field of view augmented reality display on search performance in divided attention tasks," in *2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*, 2014, pp. 177–186.

[34] E. D. Ragan, D. A. Bowman, R. Kopper, C. Stinson, S. Scrbo, and R. P. McMahan, "Effects of field of view and visual complexity on virtual reality training effectiveness for a visual scanning task," *IEEE Transactions on Visualization and Computer Graphics*, vol. 21, no. 7, pp. 794–807, 2015.

[35] J. Kim, A. Madhusudan, B. Watson, B. Boudaoud, R. Tarrazo, and J. Spjut, "Display size and targeting performance: Small hurts, large may help," in *SIGGRAPH Asia 2022 Conference Papers*, ser. SA '22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3550469.3555396

[36] M. Lee, G. Bruder, T. Höllerer, and G. Welch, "Effects of unaugmented periphery and vibrotactile feedback on proxemics with virtual humans in ar," *IEEE Transactions on Visualization and Computer Graphics*, vol. 24, no. 4, pp. 1525–1534, 2018.
[37] I. Wang, J. Smith, and J. Ruiz, "Exploring virtual agents for augmented reality," in *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems*, ser. CHI '19. New York, NY, USA: Association for Computing Machinery, 2019, p. 1–12. [Online]. Available: https://doi.org/10.1145/3290605.3300511

[38] D. Ren, T. Goldschwendt, Y. Chang, and T. Höllerer, "Evaluating wide-field-of-view augmented reality with mixed reality simulation," in *2016 IEEE Virtual Reality (VR)*, 2016, pp. 93–102.

[39] D. Medeiros, M. McGill, A. Ng, R. McDermid, N. Pantidi, J. Williamson, and S. Brewster, "From shielding to avoidance: Passenger augmented reality and the layout of virtual displays for productivity in shared transit," *IEEE Transactions on Visualization and Computer Graphics*, vol. 28, no. 11, pp. 3640–3650, 2022.

[40] J. a. M. Evangelista Belo, M. N. Lystbæk, A. M. Feit, K. Pfeuffer, P. Kân, A. Oulasvirta, and K. Grønbæk, "Auit – the adaptive user interfaces toolkit for designing xr applications," in *Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology*, ser. UIST '22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3526113.3545651

[41] S. Prince, A. Cheok, F. Farbiz, T. Williamson, N. Johnson, M. Billinghurst, and H. Kato, "3d live: real time captured content for mixed reality," in *Proceedings. International Symposium on Mixed and Augmented Reality*, 2002, pp. 7–317.

[42] A. Jones, M. Lang, G. Fyffe, X. Yu, J. Busch, I. McDowall, M. Bolas, and P. Debevec, "Achieving eye contact in a one-to-many 3d video teleconferencing system," *ACM Trans. Graph.*, vol. 28, no. 3, jul 2009. [Online]. Available: https://doi.org/10.1145/1531326.1531370

[43] S. Beck, A. Kunert, A. Kulik, and B. Froehlich, "Immersive group-to-group telepresence," *IEEE Transactions on Visualization and Computer Graphics*, vol. 19, no. 4, pp. 616–625, 2013.

[44] G. Gamelin, A. Chellali, S. Cheikh, A. Ricca, C. Dumas, and S. Otmane, "Point-cloud avatars to improve spatial communication in immersive collaborative virtual environments," *Personal and Ubiquitous Computing*, vol. 25, pp. 467–484, 2021.

[45] S. Yura, T. Usaka, and K. Sakamura, "Video avatar: embedded video for collaborative virtual environment," in *Proceedings IEEE International Conference on Multimedia Computing and Systems*, vol. 2, 1999, pp. 433–438 vol.2.

[46] Y. Wang, D. M. Krum, E. M. Coelho, and D. A. Bowman, "Contextualized videos: Combining videos with environment models to support situational understanding," *IEEE Transactions on Visualization and Computer Graphics*, vol. 13, no. 6, pp. 1568–1575, 2007.

[47] M. J. Prins, S. N. B. Gunkel, H. M. Stokking, and O. A. Niamut, "Togethervr: A framework for photorealistic shared media experiences in 360-degree vr," *SMPTE Motion Imaging Journal*, vol. 127, no. 7, pp. 39–44, 2018.

[48] M. Billinghurst and H. Kato, "Real world teleconferencing," in *CHI '99 Extended Abstracts on Human Factors in Computing Systems*, ser. CHI EA '99. New York, NY, USA: Association for Computing Machinery, 1999, p. 194–195. [Online]. Available: https://doi.org/10.1145/632716.632838

[49] C. Neustaedter, S. Singhal, R. Pan, Y. Heshmat, A. Forghani, and J. Tang, "From being there to watching: Shared and dedicated telepresence robot usage at academic conferences," *ACM Trans. Comput.-Hum. Interact.*, vol. 25, no. 6, dec 2018. [Online]. Available: https://doi.org/10.1145/3243213

[50] Y. Furuya and K. Takashio, "Telepresence robot blended with a real landscape and its impact on user experiences," in *2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)*, 2020, pp. 406–411.

[51] J. H. Chuah, A. Robb, C. White, A. Wendling, S. Lampotang, R. Kopper, and B. Lok, "Increasing agent physicality to raise social presence and elicit realistic behavior," in *2012 IEEE Virtual Reality Workshops (VRW)*. IEEE, 2012, pp. 19–22.

[52] D. Jo, K.-H. Kim, and G. J. Kim, "Spacetime: adaptive control of the teleported avatar for improved ar tele-conference experience," *Computer Animation and Virtual Worlds*, vol. 26, no. 3-4, pp. 259–269, 2015.

[53] Y. Lang, W. Liang, and L.-F. Yu, "Virtual agent positioning driven by scene semantics in mixed reality," in *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, 2019, pp. 767–775.

[54] D. Lindlbauer, A. M. Feit, and O. Hilliges, "Context-aware online adaptation of mixed reality interfaces," in *Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology*, ser. UIST '19. New York, NY, USA: Association

for Computing Machinery, 2019, p. 147–160. [Online]. Available: https://doi.org/10.1145/3332165.3347945

[55] J. N. Bailenson, J. Blascovich, A. C. Beall, and J. M. Loomis, "Inter-personal distance in immersive virtual environments," *Personality and social psychology bulletin*, vol. 29, no. 7, pp. 819–833, 2003.

[56] J. E. Swan, A. Jones, E. Kolstad, M. A. Livingston, and H. S. Smallman, "Egocentric depth judgments in optical, see-through augmented reality," *IEEE Transactions on Visualization and Computer Graphics*, vol. 13, no. 3, pp. 429–442, 2007.

[57] J. W. Woodworth, N. G. Lipari, and C. W. Borst, "Evaluating teacher avatar appearances in educational vr," in *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, 2019, pp. 1235–1236.

[58] D. Gonzalez-Morin, E. Gonzalez-Sosa, P. Perez-Garcia, and A. Villegas, "Bringing real body as self- avatar into mixed reality: A gamified volcano experience," in *2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)*, 2022, pp. 794–795.

[59] D. G. Morin, E. Gonzalez-Sosa, P. Perez, and A. Villegas, "Full body video-based self-avatars for mixed reality: from e2e system to user study," *arXiv preprint arXiv:2208.12639*, 2022.

[60] G. A. Lee, J. Chen, M. Billinghurst, and R. Lindeman, "Enhancing immersive cinematic experience with augmented virtuality," in *2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct)*, 2016, pp. 115–116.

[61] S. Jung, G. Bruder, P. J. Wisniewski, C. Sandor, and C. E. Hughes, "Over my hand: Using a personalized hand in vr to improve object size estimation, body ownership, and presence," in *Proceedings of the 2018 ACM Symposium on Spatial User Interaction*, ser. SUI '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 60–68. [Online]. Available: https://doi.org/10.1145/3267782.3267920

[62] K. Caine, "Local standards for sample size at CHI," in *Proceedings of the 2016 CHI conference on human factors in computing systems*, 2016, pp. 981–992.

[63] D. Kim, J.-e. Shin, J. Lee, and W. Woo, "Adjusting relative translation gains according to space size in redirected walking for mixed reality mutual space generation," in *2021 IEEE Virtual Reality and 3D User Interfaces (VR)*, 2021, pp. 653–660.

[64] S. Lin, L. Yang, I. Saleemi, and S. Sengupta, "Robust high-resolution video matting with temporal guidance," in *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, 2022, pp. 238–247.

[65] K. L. Nowak and F. Biocca, "The effect of the agency and anthropomorphism on users' sense of telepresence, copresence, and social presence in virtual environments," *Presence: Teleoperators and Virtual Environments*, vol. 12, no. 5, pp. 481–494, 10 2003. [Online]. Available: https://doi.org/10.1162/105474603322761289

[66] T. Rhee, L. Petikam, B. Allen, and A. Chalmers, "Mr360: Mixed reality rendering for 360° panoramic videos," *IEEE Transactions on Visualization and Computer Graphics*, vol. 23, no. 4, pp. 1379–1388, 2017.

[67] Z. He, R. Du, and K. Perlin, "Lookatchat: Visualizing gaze awareness for remote small-group conversations," *arXiv preprint arXiv:2107.06265*, 2021.

[68] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "Nerf: Representing scenes as neural radiance fields for view synthesis," *Commun. ACM*, vol. 65, no. 1, p. 99–106, dec 2021. [Online]. Available: https://doi.org/10.1145/3503250

[69] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, "D-nerf: Neural radiance fields for dynamic scenes," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2021, pp. 10 318–10 327.

[70] G. Gafni, J. Thies, M. Zollhofer, and M. Niessner, "Dynamic neural radiance fields for monocular 4d facial avatar reconstruction," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2021, pp. 8649–8658.

[71] Y. Kim, H. Park, S. Bang, and S.-H. Lee, "Retargeting human-object interaction to virtual avatars," *IEEE Transactions on Visualization and Computer Graphics*, vol. 22, no. 11, pp. 2405–2412, 2016.

[72] T. Tahara, T. Seno, G. Narita, and T. Ishikawa, "Retargetable ar: Context-aware augmented reality in indoor scenes based on 3d scene graph," in *2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)*, 2020, pp. 249–255.
[73] G. Song, T.-J. Cham, J. Cai, and J. Zheng, "Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence," in 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2022, pp. 729-738.

[74] M. Elgharib, M. Mendiratta, J. Thies, M. Nießner, H.-P. Seidel, A. Tewari, V. Golyanik, and C. Theobalt, "Egocentric videoconferencing," ACM Transactions on Graphics, vol. 39, no. 6, Dec 2020.

<figure><img src="image_13.png" /></figure>

**Xuanyu Wang** received a bachelor's degree in computer science and technology from Xi'an Jiaotong University in 2018. He is currently working towards a joint PhD degree at the School of Computer Science and Technology, Xi'an Jiaotong University and the School of Creative Media, City University of Hong Kong. His research interests lie in the intersection between Human-Computer Interaction and Computer Graphics.

<figure><img src="image_14.png" /></figure>

mobile computing.

**Weizhan Zhang** received the BS degree in electronics engineering from Zhejiang University, China, in 1999, and the PhD degree in computer science from Xi'an Jiaotong University, China, in 2010. He was also a visiting scholar in the Department of Computer Science and Engineering at Pennsylvania State University, in 2015. Now he is a professor at the School of Computer Science and Technology, Xi'an Jiaotong University. His research interests include multimedia networking, edge intelligence, and

<figure><img src="image_15.png" /></figure>

**Christian Sandor** received the doctorate degree in computer science from the Munich University of Technology, Germany, in 2005 under the supervision of Prof. Gudrun Klinker and Prof. Steven Feiner. He is a Professor at Université Paris-Saclay and leader of Team ARAI at CNRS (Centre National de la Recherche Scientifique). He is a member of the IEEE. He serves as an editorial board member for IEEE Transactions on Visualization and Computer Graphics and as a steering committee member for ACM Symposium on Spatial User Interaction. He has been program chair for numerous conferences, including IEEE ISMAR, SIGGRAPH Asia XR, and SIGGRAPH Asia Symposium On Mobile Graphics And Interactive Applications.

<figure><img src="image_16.png" /></figure>

**Hongbo Fu** received a BS degree in information sciences in 2002 from Peking University, China and a PhD degree in computer science from the Hong Kong University of Science and Technology in 2007. He is a Professor at the Division of Arts and Machine Creativity, Hong Kong University of Science and Technology. His primary research interests fall in the fields of computer graphics and human-computer interaction. He has served as an Associate Editor of The Visual Computer, Computers & Graphics, and Computer Graphics Forum.