<figure><img src="image_1.png" /></figure>

HAL
open science

Metaverse Meets Consumer Electronics

Haiwei Dong, Yang Liu

► To cite this version:

Haiwei Dong, Yang Liu. Metaverse Meets Consumer Electronics. IEEE consumer electronics magazine, In press, 10.1109/MCE.2022.3229180. hal-03978027

HAL Id: hal-03978027

https://hal.science/hal-03978027

Submitted on 8 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Metaverse Meets
# Consumer Electronics

Haiwei Dong

Huawei Technologies Canada

Yang Liu

Magic Leap Canada

■ **THE** first year of the Metaverse Era is considered 2021 mainly because the gaming company Roblox was listed on March 10, 2021. Different from the traditional games, in the Roblox metaverse, the game players can construct their own games to let others play together.

In October 2021, Facebook's parent company was rebranded Meta Platforms which further boosted metaverse. Mark Zuckerberg announced Meta's vision of metaverse called Horizon in which users can throw parties, buy/sell houses and cars, open concerts, customize the clothes of users' avatars, and design different environments for home and working offices. He believes the aforementioned immersive interaction would completely change human life in the future.

Following Meta, Microsoft announced its hybrid collaboration platform Mesh. Nvidia announced Omniverse which aims to realize highly productive remote work through metaverse. Currently, the most immediate emerging products in metaverse are considered to be gaming and cooperative working.

The dominant companies in metaverse have different advantages in jumping into the metaverse area: Epic's Unreal Engine renders a high-fidelity 3D virtual environment; Meta's Quest VR (virtual reality) headsets provide users with a fully-immersive environment to interact; Microsoft's HoloLens MR (mixed reality)

headsets enable the users to overlay the virtual objects onto the real environment and interact with real-world through gesture vocabulary, speech recognition, etc.

## The Evolution of the Metaverse

Nowadays, different technical giants are separately promoting their own metaverses resulting in “islands” of decentralized, multi-centered metaverse systems. Although users join different metaverses respectively, their virtual assets and customer data are not shared between the metaverses. In order to build a unified fully-connected metaverse ecosystem, the standards of the metaverse as well as corresponding communication protocols are highly required as shown in Figure 1. In this regard, NVIDIA Omniverse is one of the pioneers which integrates digital models across multiple metaverse systems through its open-source file format and framework Universal Scene Description (USD). Furthermore, it provides the capability to build and render different 3D scenes and assets for different metaverse systems, and deploy them to a variety of eXtended reality devices remotely.

Additionally, Khronos Group builds an open standard named OpenXR for developing applications running on any major eXtended reality devices and platforms through a unifying application programming interface. According to Deloitte's report [1], it is expected that the distributed metaverse systems will gradually share data and establish unified standards across platforms, thus achieving full integration in the

Digital Object Identifier 10.1109/MCE.2022.Doi Number

Date of publication 00 xxxx 0000; date of current version 00 xxxx 0000
<figure><img src="image_3.png" /><figcaption>**Figure 1.** Interconnected metaverse systems with seamless experiences for users.</figcaption></figure>

mature stage.

## Metaverse-CE Taxonomy

The metaverse requires consumer electronics (CE) to measure users' interaction information and meanwhile create an immersive environment with multimodality. Additionally, the consumer electronics take the users in the metaverse to experience a new virtual life with other peers as shown in Figure 2. There are a few key technology pillars that support the metaverse in the virtual world [2] as the following:

* **Sensing, Actuating:** To create a fully immersive metaverse environment, various kinds of sensory measurements are made, such as binocular stereo scenes computed by camera transformations. Furthermore, different kinds of actuator actions are generated, such as interaction forces calculated by a physics engine.

* **Representation, Assets:** High-fidelity virtual scenes and customized avatars are rendered and displayed in 3D. In addition, digital assets authenticated by NFT (non-fungible token), a digital collectible as an example, are also included.

* **Artificial Intelligence:** It enables the metaverse to be self-cognitive, such as interactively talking with users by natural language processing, effectively engaging with users by making inferences from users' behaviors, etc.

* **Blockchain, Cybersecurity:** The virtual metaverse requires privacy protection, encryption, and authentication to protect users from cyber-attacks. In addition, blockchain as a decentralized ledger would keep all critical transactions safely recorded.

To enable the users immersively interact in the metaverse, consumer electronics use many different ways to strengthen the sense of reality, including:

* **Vision, Sound, Haptics, Smell, Taste:** The users look at a binocular 3D high-resolution displays by an eXtended reality headset or a smartphone [3]. Broader field of view (130° horizontally by 100° vertically or even 200° by 130°), higher resolution (4K or even 8K), and higher refresh rate (90 fps or even 120 fps) provide a natural view to the users. Spatial audio enriches the 3D surround sound experience of sonic changes with the movement of the head. Haptic vest and haptic gloves enable the users to feel the stiffness and even texture of a virtual object in the metaverse. The electrical generator or simulator of smell and taste could further enhance the ultimate immersion experience.

* **Voice, Gesture:** Multi-channel microphone array makes the users' voice input clear enough, followed by speech recognition. Users' gesture vocabularies are recognized by a set of RGB-D cameras.

Between the metaverse and users, the Web 3.0 network [4] transmits the datagrams between the virtual and real world with ultra-low end-to-end latency (ideally less than 7 milliseconds), including:

* **Downlink Traffic:** Mainly the rendered virtual environment frames from the metaverse cloud to the consumer electronic end host. As the rendered frame is too big to fit in one packet for transmission, it is divided into a number of small packets resulting in the traffic pattern as a micro-burst of packets [5].

* **Uplink Traffic:** Typically including the pose mea-
<figure><img src="image_4.png" /><figcaption>Figure 2. The key components and information flow between a metaverse and consumer electronic devices.</figcaption></figure>

surement of the user's head, signals from the con-
trollers, and synchronization from the consumer
electronic end host to the metaverse cloud, etc.
The traffic pattern is a much more frequent packet
transmission with much smaller packet sizes.

Outlook: When Metaverse Comes to Life

Metaverse brings a future where the virtual and
real worlds are intertwined. With an immersive user
experience, it digitizes the real experience in the meta-
verse. Users leverage metaverse to not only make their
work more efficient and intuitive (e.g., by using Meta
Horizon workrooms), but also enrich their social life
and entertainment. All these immersive experiences
greatly rely on the collaborative virtual environment
provided by consumer electronics, realizing the high
quality of experience (QoE) for users.

On the other side, the metaverse also emphasizes
the concept of *realizing the digital experience*: Virtual
objects would have a great impact on users' lives in
the real world. For instance, through augmented reality
glasses, the metaverse could overlay vivid navigation
guidance on the real road by taking measurements
via consumer electronics (such as LiDAR sensors or
cameras). More interestingly, virtual assets such as
cryptocurrency and NFT create real economic value,
namely meta-economy.

■ REFERENCES

1. "Metaverse report - Future is here," Deloitte China, Tech. Rep., 2022.

2. A. El Saddik, “Digital twins: The convergence of multimedia technologies,” *IEEE Multimedia*, vol. 25, no. 2, pp. 97–92, 2018.

3. Z. Long, H. Dong, and A. El Saddik, “Interacting with New York city data by HoloLens through remote rendering,” *IEEE Consumer Electronics Magazine*, vol. 11, no. 5, pp. 64–72, 2022.

4. “Opportunities in the metaverse,” J.P.Morgan, Tech. Rep., 2022.

5. H. Dong and J. Lee, "The metaverse from a multimedia communications perspective," *IEEE Multimedia*, vol. 29, no. 4, pp. 123–127, 2022.

**Haiwei Dong** is a Principal Researcher with Huawei Technologies Canada, Ottawa, Canada. He also serves as Associate Editor of IEEE Consumer Electronics Magazine, ACM Transactions on Multimedia Computing, Communications, and Applications; and Department Editor of IEEE Multimedia Magazine. He is a Senior Member of the IEEE. Contact him at haiwei.dong@ieee.org.

**Yang Liu** is a Computer Vision Researcher with Magic Leap Inc., Canada. His research interests include computer vision and multimedia. Contact him at yaliu@magicleap.com.